Occasional freezing of the processor

[Paresh Shah]

Nifi Version: 1.3
#nodes: 4
Using embedded zookeeper.

I have a custom processor which underneath runs a Jetty Server and is used for 
receiving external push notifications. The processor is scheduled to run every 
1 sec and will pull the data from the queue where the Jetty layer queues it.

Problem:
We are seeing that occasionally the processor appears to be not running i.e not 
processing any data, though there is data to be processed. The processor is in 
the running state. The Jetty server internally is still receiving new requests 
and queuing them. There are no OutOfMemory errors or Exceptions. The processor 
just freezes.

The Processor’s in the rest of the pipeline continue to run. Also the CPU 
utilization is only 5-10%.

Temporary Workaround:
We have had to stop and start this processor to have the processor start 
processing again.

Thanks
Paresh

Re: [EXT] Re: URL configuration for the remote process group in Nifi 1.3

[Paresh Shah]

Mark,

Wanted to make sure you did see my response. Looking forward to yours.

Thanks
Paresh

On 6/4/18, 10:24 AM, "Paresh Shah"  wrote:

Mark

Want some more clarity. Let me see if I understand this. Just to be clear 
we are using RPG purely for load balancing on the same cluster.

Step 1: When it initially connects to Node1, it would fetch all the cluster 
details i.e it would know all the nodes that exist in my cluster which are
Node1
Node2
Node3

Question: Where is this information persisted and does it resolve this 
every time there are flow files sent to the RPG( remote process group ). 

Step 2: Now when Node1 goes down it would try to establish communication 
with one of the following nodes which it had retrieved and stored initially or 
as a background task.
Node2
Node3

Question: Does this update the persisted information in Step1. Is there any 
way to update the actual URL for the RPG. Basically we do not want every 
incoming flow file on the RPG to end up selecting the target node. 

Thanks
Paresh

On 6/3/18, 9:09 AM, "Mark Payne"  wrote:

Paresh,

When NiFi establishes a connection to the remote instance, it will 
request information from the remote instance about all nodes in the cluster. It 
then persists this information in case nifi is restarted. So whichever node you 
use in your URL is only important for the initial connection. Additionally, 
NiFi will periodically reach out to the remote nifi instances to determine 
which nodes are in the cluster, in case nodes are added to or removed from the 
cluster.

Does that all make sense?

Thanks
-Mark

Sent from my iPhone

> On Jun 3, 2018, at 11:15 AM, Paresh Shah  
wrote:
> 
> I have a cluster with 3 nodes. We are using RPG for load balancing
> 
> Node1 ( primary and cluster coordinator ).
> Node2
> Node3
> 
> When configuring the RPG is use Node1 as the target URL. My question 
is what happens to this RPG when the Node1 goes down or is offline. At this 
point how does the RPG keep functioning, since we cannot update the URL once 
its created.
> 
> Thanks
> Paresh
> 

Re: [EXT] Re: URL configuration for the remote process group in Nifi 1.3

[Paresh Shah]

Mark

Want some more clarity. Let me see if I understand this. Just to be clear we 
are using RPG purely for load balancing on the same cluster.

Step 1: When it initially connects to Node1, it would fetch all the cluster 
details i.e it would know all the nodes that exist in my cluster which are
Node1
Node2
Node3

Question: Where is this information persisted and does it resolve this every 
time there are flow files sent to the RPG( remote process group ). 

Step 2: Now when Node1 goes down it would try to establish communication with 
one of the following nodes which it had retrieved and stored initially or as a 
background task.
Node2
Node3

Question: Does this update the persisted information in Step1. Is there any way 
to update the actual URL for the RPG. Basically we do not want every incoming 
flow file on the RPG to end up selecting the target node. 

Thanks
Paresh

On 6/3/18, 9:09 AM, "Mark Payne"  wrote:

Paresh,

When NiFi establishes a connection to the remote instance, it will request 
information from the remote instance about all nodes in the cluster. It then 
persists this information in case nifi is restarted. So whichever node you use 
in your URL is only important for the initial connection. Additionally, NiFi 
will periodically reach out to the remote nifi instances to determine which 
nodes are in the cluster, in case nodes are added to or removed from the 
cluster.

Does that all make sense?

Thanks
-Mark

Sent from my iPhone

> On Jun 3, 2018, at 11:15 AM, Paresh Shah  wrote:
> 
> I have a cluster with 3 nodes. We are using RPG for load balancing
> 
> Node1 ( primary and cluster coordinator ).
> Node2
> Node3
> 
> When configuring the RPG is use Node1 as the target URL. My question is 
what happens to this RPG when the Node1 goes down or is offline. At this point 
how does the RPG keep functioning, since we cannot update the URL once its 
created.
> 
> Thanks
> Paresh
> 

URL configuration for the remote process group in Nifi 1.3

[Paresh Shah]

I have a cluster with 3 nodes. We are using RPG for load balancing

Node1 ( primary and cluster coordinator ).
Node2
Node3

When configuring the RPG is use Node1 as the target URL. My question is what 
happens to this RPG when the Node1 goes down or is offline. At this point how 
does the RPG keep functioning, since we cannot update the URL once its created.

Thanks
Paresh

Re: EndpointConnectionPool is not reusable.

[Paresh Shah]

Joe,

For our use case we want to be able to do the following.

1. In certain paths we want to be able to send the same FlowFiles to all
nodes of a remote cluster ( broadcast use case )
2. In another use case send it to one of the nodes of the remote cluster (
regular RPG use case ).

What we are working on is a general purpose DistributeProcessor that works
of a application defined strategy. This strategy in-turn would determine
which nodes need to be sent the data to.

We can set up a meeting to discuss this in more detail.

Paresh

On 3/16/16, 7:34 AM, "Joe Percivall" <joeperciv...@yahoo.com.INVALID>
wrote:

>Hello Paresh,
>
>Just to be sure I understand your use-case correctly, you want to
>transfer FlowFiles from one NiFi instance to a NiFi cluster using site to
>site and you want to have the FlowFiles go to a certain node in the
>cluster based on an application specific strategy.
>
>I have not heard any plans to make the routing extensible but that's
>primarily because I don't think any one has asked for it before.
>
>Forgive me if this is a dumb question but, why not group similar nodes
>together into smaller clusters instead of treating them all the same?
>
>Sorry no one had gotten back to you earlier,
>
>Joe
>
>- - - - - -
>Joseph Percivall
>linkedin.com/in/Percivall
>e: joeperciv...@yahoo.com
>
>
>
>
>
>On Friday, March 11, 2016 10:19 AM, Paresh Shah
><paresh.s...@lifelock.com> wrote:
>Have not seen a response from the Nifi team on this. Can someone from the
>team, please share their views on this.
>
>Thanks
>Paresh
>
>
>From: Paresh Shah
><paresh.s...@lifelock.com<mailto:paresh.s...@lifelock.com>>
>Date: Tuesday, March 8, 2016 at 11:58 AM
>To: "dev@nifi.apache.org<mailto:dev@nifi.apache.org>"
><dev@nifi.apache.org<mailto:dev@nifi.apache.org>>
>Subject: EndpointConnectionPool is not reusable.
>
>We have a use case where we would like to perform some routing based on
>application defined strategy. In order to be able to do this we need to
>have a mechanism to pass in the selcted nodes based on the app strategy.
>
>Ideally we would like to pass this strategy into the
>EndpointConnectionPool. Currently we see that the EndpointConnectionPool
>is not implemented as an interface and thus SocketClient explicitly
>contains an instance of this class. Also the logic to perform the load
>balancing is buried inside the EndpointConnectionPool.
>
>Are there any plans in the near term to make these components extensible.
>
>Thanks
>Paresh
>
>
>The information contained in this transmission may contain privileged and
>confidential information. It is intended only for the use of the
>person(s) named above. If you are not the intended recipient, you are
>hereby notified that any review, dissemination, distribution or
>duplication of this communication is strictly prohibited. If you are not
>the intended recipient, please contact the sender by reply email and
>destroy all copies of the original message.
>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: EndpointConnectionPool is not reusable.

[Paresh Shah]

Have not seen a response from the Nifi team on this. Can someone from the team, 
please share their views on this.

Thanks
Paresh

From: Paresh Shah <paresh.s...@lifelock.com<mailto:paresh.s...@lifelock.com>>
Date: Tuesday, March 8, 2016 at 11:58 AM
To: "dev@nifi.apache.org<mailto:dev@nifi.apache.org>" 
<dev@nifi.apache.org<mailto:dev@nifi.apache.org>>
Subject: EndpointConnectionPool is not reusable.

We have a use case where we would like to perform some routing based on 
application defined strategy. In order to be able to do this we need to have a 
mechanism to pass in the selcted nodes based on the app strategy.

Ideally we would like to pass this strategy into the EndpointConnectionPool. 
Currently we see that the EndpointConnectionPool is not implemented as an 
interface and thus SocketClient explicitly contains an instance of this class. 
Also the logic to perform the load balancing is buried inside the 
EndpointConnectionPool.

Are there any plans in the near term to make these components extensible.

Thanks
Paresh


The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

EndpointConnectionPool is not reusable.

[Paresh Shah]

We have a use case where we would like to perform some routing based on 
application defined strategy. In order to be able to do this we need to have a 
mechanism to pass in the selcted nodes based on the app strategy.

Ideally we would like to pass this strategy into the EndpointConnectionPool. 
Currently we see that the EndpointConnectionPool is not implemented as an 
interface and thus SocketClient explicitly contains an instance of this class. 
Also the logic to perform the load balancing is buried inside the 
EndpointConnectionPool.

Are there any plans in the near term to make these components extensible.

Thanks
Paresh


The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Lost Trace of FowFiles only with attributes

[Paresh Shah]

In the cluster mode we have seeing the following two issues.

  1.  In one scenario we see that the flowFile( only containing attributes ) is 
sent over to the remote cluster and after the first processor following the 
InputPort in the pipeline the data provnance shows no trace of that flow file. 
Where can we locate this flow file in the repository?

2. In another scenario we see that the data provenance on the receiving cluster 
is not showing any updates since 4/5 hours even though the sending cluster 
indicates  that it has successfully sent data to it. So we are not sure where 
the data is dropped.

We have checked that the clocks on all the cluster nodes are in sync.

Paresh



The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Smart Load Balancing behavior replicating data to different cluster nodes.

[Paresh Shah]

The problem is not quite what we thought.

What we see is that we generate a bunch of flowFiles on the sender
processor and then commit. But all these get delivered to the same remote
node. What we are expecting is they should be split depending on the no of
nodes and the loads on each node.


Paresh

On 2/4/16, 10:53 AM, "Paresh Shah" <paresh.s...@lifelock.com> wrote:

>We have the pipeline as follows.
>
>Sender Side
>  P1..->Pn -> RPG ( connected to ForwardInputPort )
>
>Receiver Side
>ForwardInputPort -> ProcessGroup
>
>Inside the processGroup
>   InputPort-> P1 Š-> Pn
>
>We see that we sent 10 flow files to the remote cluster. There we see the
>same set of flow files being sent to different nodes of the cluster. We
>are confirming this by looking at the DataProvnance and the fileNames in
>there. They are the same ones for two nodes of the cluster.
>
>This behaviour seems to incorrect.
>
>Paresh
>
>
>The information contained in this transmission may contain privileged and
>confidential information. It is intended only for the use of the
>person(s) named above. If you are not the intended recipient, you are
>hereby notified that any review, dissemination, distribution or
>duplication of this communication is strictly prohibited. If you are not
>the intended recipient, please contact the sender by reply email and
>destroy all copies of the original message.
>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Unable to communicate between two Nifi clusters.

[Paresh Shah]

We have multiple cluster setup.

Cluster 1:
NCM : 10.233.0.183:8080
Nodes: 10.233.0.156, 10.233.0.180

Cluster 2:
NCM : i-e9c63569.dev.aws.lifelock.ad
Nodes : i-e9c63569.dev.aws.lifelock.ad, 
i-e9c63569.dev.aws.lifelock.ad

We were able to successfully connect the RPG from Cluster 1 to the remote input 
port. But are unable to send the data to the remote cluster. This means

We are able to log in to the cluster(s) and see that the nodes are connected.
Here’s snippets of the logs from the cluster that we are trying to send the 
data from.


2016-02-03 04:30:58,565 INFO [NiFi Web Server-236] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET 
http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller (source ip: 
10.233.0.156)

2016-02-03 04:31:06,093 INFO [NiFi Web Server-236] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET 
http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller/process-groups/ec38494d-b925-4afb-a0c0-2a02861ec03e/status
 (source ip: 172.16.2.33)

2016-02-03 04:31:06,210 INFO [NiFi Web Server-2695] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET 
http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller/status (source 
ip: 172.16.2.33)

2016-02-03 04:31:06,217 INFO [NiFi Web Server-1273] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET 
http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller/revision (source 
ip: 172.16.2.33)

2016-02-03 04:31:07,407 INFO [NiFi Web Server-2694] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET 
http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller (source ip: 
10.233.0.180)

2016-02-03 04:31:07,433 INFO [NiFi Web Server-2695] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET 
http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller (source ip: 
10.233.0.156)


The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Unable to communicate between two Nifi clusters.

[Paresh Shah]

More updates. Now the sending cluster reports that the nodes are
disconnected.

...
2016-02-03 04:41:09,176 WARN [NiFi Web Server-313]
o.a.n.c.manager.impl.WebClusterManager All nodes failed to process URI
http://10.233.0.183:8080/nifi-api/controller/process-groups/c8236793-3ea2-4
e03-a721-f63ada78c1cd/remote-process-groups/1a8cb3c7-8705-4e11-ac03-df3adf8
df113. As a result, no node will be disconnected from cluster
2016-02-03 04:41:09,177 INFO [NiFi Web Server-313]
o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request:
org.apache.nifi.cluster.manager.exception.NoConnectedNodesException: All
nodes were disconnected as a result of applying request PUT
http://10.233.0.183:8080/nifi-api/controller/process-groups/c8236793-3ea2-4
e03-a721-f63ada78c1cd/remote-process-groups/1a8cb3c7-8705-4e11-ac03-df3adf8
df113. Returning Internal Server Error response.
2016-02-03 04:41:09,178 INFO [NiFi Web Server-348]
o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request:
org.apache.nifi.cluster.manager.exception.NoConnectedNodesException.
Returning Internal Server Error response.
2016-02-03 04:41:09,179 INFO [NiFi Web Server-365]
o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request:
org.apache.nifi.cluster.manager.exception.NoConnectedNodesException.
Returning Internal Server Error response.
2016-02-03 04:41:09,180 INFO [NiFi Web Server-264]
o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request:
org.apache.nifi.cluster.manager.exception.NoConnectedNodesException.
Returning Internal Server Error response.
2016-02-03 04:41:09,180 INFO [NiFi Web Server-346]
o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request:
org.apache.nifi.cluster.manager.exception.NoConnectedNodesException.
Returning Internal Server Error response.
2016-02-03 04:41:09,180 INFO [NiFi Web Server-367]
o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request:
org.apache.nifi.cluster.manager.exception.NoConnectedNodesException:
Cannot apply GET request to
http://10.233.0.183:8080/nifi-api/controller/process-groups/c8236793-3ea2-4
e03-a721-f63ada78c1cd/remote-process-groups/1a8cb3c7-8705-4e11-ac03-df3adf8
df113 because there are currently no connected Nodes. Returning Internal
Server Error response.
2016-02-03 04:41:10,066 INFO [NiFi Web Server-368]
o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request:
org.apache.nifi.cluster.manager.exception.NoConnectedNodesException:
Cannot apply GET request to
http://10.233.0.183:8080/nifi-api/controller/config because there are
currently no connected Nodes. Returning Internal Server Error response.
2016-02-03 04:41:10,681 INFO [Process NCM Request-6]
o.a.n.c.p.impl.SocketProtocolListener Received request
55b07d8c-a5ce-4a4f-bb59-059789153fcd from 10.233.0.180
2016-02-03 04:41:10,683 INFO [Process NCM Request-6]
o.a.n.c.p.impl.SocketProtocolListener Finished processing request
55b07d8c-a5ce-4a4f-bb59-059789153fcd (type=BULLETINS, length=1877 bytes)
in 61 millis



On 2/2/16, 8:40 PM, "Paresh Shah" <paresh.s...@lifelock.com> wrote:

>We have multiple cluster setup.
>
>Cluster 1:
>NCM : 10.233.0.183:8080
>Nodes: 10.233.0.156, 10.233.0.180
>
>Cluster 2:
>NCM : i-e9c63569.dev.aws.lifelock.ad
>Nodes :
>i-e9c63569.dev.aws.lifelock.ad<http://i-e9c63569.dev.aws.lifelock.ad>,
>i-e9c63569.dev.aws.lifelock.ad
>
>We were able to successfully connect the RPG from Cluster 1 to the remote
>input port. But are unable to send the data to the remote cluster. This
>means
>
>We are able to log in to the cluster(s) and see that the nodes are
>connected.
>Here¹s snippets of the logs from the cluster that we are trying to send
>the data from.
>
>
>2016-02-03 04:30:58,565 INFO [NiFi Web Server-236]
>org.apache.nifi.web.filter.RequestLogger Attempting request for
>(anonymous) GET
>http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller (source
>ip: 10.233.0.156)
>
>2016-02-03 04:31:06,093 INFO [NiFi Web Server-236]
>org.apache.nifi.web.filter.RequestLogger Attempting request for
>(anonymous) GET
>http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller/process-gro
>ups/ec38494d-b925-4afb-a0c0-2a02861ec03e/status (source ip: 172.16.2.33)
>
>2016-02-03 04:31:06,210 INFO [NiFi Web Server-2695]
>org.apache.nifi.web.filter.RequestLogger Attempting request for
>(anonymous) GET
>http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller/status
>(source ip: 172.16.2.33)
>
>2016-02-03 04:31:06,217 INFO [NiFi Web Server-1273]
>org.apache.nifi.web.filter.RequestLogger Attempting request for
>(anonymous) GET
>http://i-e9c63569.dev.aws.lifelock.ad:8080/nifi-api/controller/revision
>(source ip: 172.16.2.33)
>
>2016-02-03 04:31:07,407 INFO [NiFi Web Server-2694]
>org.apache.nifi.web.filter.RequestLogger Attempting request

Getting error when a previously disconnected node is being brought up and trying to connect back to the NCM

[Paresh Shah]

Here’s the scenario.

1. Deploy Nifi Cluster :
- NCM
- Pirmary
- Node.

2. Verified that everything is OK.

3. I then disconnected the slave Node and deployed some pipelines on the 
cluster.

4. Now I tried to bring up the slave node. It gives the following error.


2016-02-01 19:41:05,393 WARN [main] org.apache.nifi.web.server.JettyServer 
Failed to start web server... shutting down.

java.lang.Exception: Unable to load flow due to: java.io.IOException: 
org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster 
because local flow is different than cluster flow.

at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:784) 
~[nifi-jetty-0.4.1.jar:0.4.1]

at org.apache.nifi.NiFi.(NiFi.java:137) [nifi-runtime-0.4.1.jar:0.4.1]

at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.4.1.jar:0.4.1]

Caused by: java.io.IOException: org.apache.nifi.cluster.ConnectionException: 
Failed to connect node to cluster because local flow is different than cluster 
flow.

at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:453)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:775) 
~[nifi-jetty-0.4.1.jar:0.4.1]

... 2 common frames omitted

Caused by: org.apache.nifi.cluster.ConnectionException: Failed to connect node 
to cluster because local flow is different than cluster flow.

at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:732)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:433)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

... 3 common frames omitted

Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
configuration is not inheritable by the flow controller because of flow 
differences: Found difference in Templates:

Local Fingerprint:   
fc0a802d-0915-3b71-8694-52b8d7b2c556melissa_rawalert_daq_dataavailabilityNO_CONNECTIONSNO_FUNNELSNO_

Cluster Fingerprint: 
42809382-bb1f-3785-b3c3-7d8b875c3e88melissa_rawalert_reqest_dataavailability_daqNO_CONNECTIONSNO_FUN

at 
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:223)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1203) 
~[nifi-framework-core-0.4.1.jar:0.4.1]

at 
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:72)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:608)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:709)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

... 4 common frames omitted


What I see is that the flow.tar file on the slave is 0 bytes. This does not 
look right. Since the file on the NCM is about 20K bytes.


thanks

Paresh



The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Getting error when a previously disconnected node is being brought up and trying to connect back to the NCM

[Paresh Shah]

But our scenario is as follows.

1. We will deploy a cluster with NCM and Primary.
2. We deploy out pipelines.
3. Add slave nodes to the cluster.

In this scenairo we cannot be wiping way the flow.* files since we will
loose the pipelines, right?

Paresh

On 2/1/16, 11:48 AM, "Joe Witt" <joe.w...@gmail.com> wrote:

>Paresh,
>
>In the log it says "Caused by:
>org.apache.nifi.controller.UninheritableFlowException: Proposed
>configuration is not inheritable by the flow controller because of
>flow differences: Found difference in Templates:".  So on your node
>blow away both the flow.xml.gz, flow.tar, and the templates directory.
>In an upcoming release we will relax the template matching
>requirement.  That was a mistake.
>
>Thanks
>Joe
>
>On Mon, Feb 1, 2016 at 2:45 PM, Paresh Shah <paresh.s...@lifelock.com>
>wrote:
>> Here¹s the scenario.
>>
>> 1. Deploy Nifi Cluster :
>> - NCM
>> - Pirmary
>> - Node.
>>
>> 2. Verified that everything is OK.
>>
>> 3. I then disconnected the slave Node and deployed some pipelines on
>>the cluster.
>>
>> 4. Now I tried to bring up the slave node. It gives the following error.
>>
>>
>> 2016-02-01 19:41:05,393 WARN [main]
>>org.apache.nifi.web.server.JettyServer Failed to start web server...
>>shutting down.
>>
>> java.lang.Exception: Unable to load flow due to: java.io.IOException:
>>org.apache.nifi.cluster.ConnectionException: Failed to connect node to
>>cluster because local flow is different than cluster flow.
>>
>> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:784)
>>~[nifi-jetty-0.4.1.jar:0.4.1]
>>
>> at org.apache.nifi.NiFi.(NiFi.java:137)
>>[nifi-runtime-0.4.1.jar:0.4.1]
>>
>> at org.apache.nifi.NiFi.main(NiFi.java:227)
>>[nifi-runtime-0.4.1.jar:0.4.1]
>>
>> Caused by: java.io.IOException:
>>org.apache.nifi.cluster.ConnectionException: Failed to connect node to
>>cluster because local flow is different than cluster flow.
>>
>> at
>>org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.j
>>ava:453) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:775)
>>~[nifi-jetty-0.4.1.jar:0.4.1]
>>
>> ... 2 common frames omitted
>>
>> Caused by: org.apache.nifi.cluster.ConnectionException: Failed to
>>connect node to cluster because local flow is different than cluster
>>flow.
>>
>> at
>>org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse
>>(StandardFlowService.java:732) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at
>>org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.j
>>ava:433) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> ... 3 common frames omitted
>>
>> Caused by: org.apache.nifi.controller.UninheritableFlowException:
>>Proposed configuration is not inheritable by the flow controller because
>>of flow differences: Found difference in Templates:
>>
>> Local Fingerprint:
>>fc0a802d-0915-3b71-8694-52b8d7b2c556melissa_rawalert_daq_dataavailability
>>NO_CONNECTIONSNO_FUNNELSNO_
>>
>> Cluster Fingerprint:
>>42809382-bb1f-3785-b3c3-7d8b875c3e88melissa_rawalert_reqest_dataavailabil
>>ity_daqNO_CONNECTIONSNO_FUN
>>
>> at
>>org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSync
>>hronizer.java:223) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at
>>org.apache.nifi.controller.FlowController.synchronize(FlowController.java
>>:1203) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at
>>org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(Standard
>>XMLFlowConfigurationDAO.java:72) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at
>>org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlow
>>Service.java:608) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at
>>org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse
>>(StandardFlowService.java:709) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> ... 4 common frames omitted
>>
>>
>> What I see is that the flow.tar file on the slave is 0 bytes. This does
>>not look right. Since the file on the NCM is about 20K bytes.
>>
>>
>> thanks
>>
>> Paresh
>>
>>
>> 
>> The information contained in this transmission may contain privileged
>>and confidential information. It is intended only for the use of the
>>pe

Re: frequency of communication between two different wifi clusters

[Paresh Shah]

Version is 0.4.1. Instances are not secured.

Paresh

On 1/29/16, 2:02 PM, "Matt Gilman" <matt.c.gil...@gmail.com> wrote:

>Paresh,
>
>The RPG should communicate with the target instance everything 30 seconds
>in the background to discover available ports. A couple of possibilities
>
>1) Which version of NiFi are you running? In 0.4.1 we addressed an issue
>that caused the RPGs to not refresh as often as it should [1]. You should
>be able to confirm this is the case by manually refreshing your browser.
>
>2) Are your instances secured? If so, the target instances need to have
>the
>accounts for the referencing NiFi approved and access to specific ports
>need to be granted. Once this is completed, you should only see the
>available ports you have access to.
>
>Matt
>
>[1]
>https://issues.apache.org/jira/browse/NIFI-1309?focusedCommentId=15064696;
>page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comm
>ent-15064696
>
>On Fri, Jan 29, 2016 at 3:05 PM, Paresh Shah <paresh.s...@lifelock.com>
>wrote:
>
>> We are in the process of automating the pipeline deloyment across
>>multiple
>> clusters. In our scenario there are two different clusters with their
>>own
>> pipelines. And the ports are open on both the machines.
>>
>> When I¹m creating the RPG on the sending cluster it failed saying that
>>the
>> destination is not reachable. Then after 3/4 minutes I was able to
>> successfully hook up the RPG. This 3/4 minutes is it configurable
>>somewhere.
>>
>> Paresh
>> 
>> The information contained in this transmission may contain privileged
>>and
>> confidential information. It is intended only for the use of the
>>person(s)
>> named above. If you are not the intended recipient, you are hereby
>>notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all
>>copies
>> of the original message.
>> 
>>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Unable to bring up cluster when the flow's go out of sync

[Paresh Shah]

Thanks.

On 1/29/16, 2:07 PM, "Mark Payne"  wrote:

>Paresh,
>In the top-right-hand corner of the NiFi UI, there is a Cluster
>Management icon. I believe it is the next-to-last icon there.If you click
>on that icon, you will be given a dialog that shows all nodes in the
>cluster, and you can click the "Disconnect" iconto remove a node from the
>cluster entirely.
>Generally, though, the easiest way to bring that node back into the
>cluster is to remove the flow.xml.gz file from the conf/ directoryand
>restart the node. NiFi does not do this automatically because there are
>cases when the user wants to intentionally disconnecta node from the
>cluster for diagnostic purposes, etc. and will have modified the flow.
>However, if you want to just bring it back upto speed with the rest of
>the nodes in your cluster, it's safe to just remove the file and restart.
>Thanks-Mark
>
>> From: paresh.s...@lifelock.com
>> To: dev@nifi.apache.org
>> Subject: Unable to bring up cluster when the flow's go out of sync
>> Date: Fri, 29 Jan 2016 21:48:41 +
>>
>> I have the cluster in a state with the message on the problem nodeŠ
>>
>>
>> Caused by: org.apache.nifi.controller.UninheritableFlowException:
>>Proposed configuration is not inheritable by the flow controller because
>>of flow differences: Found difference in Flows:
>>
>> Local Fingerprint:
>>170e8ec-5b16-4564-aa31-11ea8bfeda3ehttp://10.233.0.183:8080/nifi8a003652-
>>9b47-43d3-b067-e57c5c4d6da8RUNNING1false1f4c0f48-030f-4ba2-8dcd-464d31ad5
>>d11352c0157-f3a2-32b4-a9e2-0139a56958a3ec38494d-b925-4
>>
>> Cluster Fingerprint:
>>170e8ec-5b16-4564-aa31-11ea8bfeda3ehttp://10.233.0.183:8080/nifi8a003652-
>>9b47-43d3-b067-e57c5c4d6da8STOPPED1false1f4c0f48-030f-4ba2-8dcd-464d31ad5
>>d11352c0157-f3a2-32b4-a9e2-0139a56958a3ec38494d-b925-4
>>
>> at
>>org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSync
>>hronizer.java:216) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at
>>org.apache.nifi.controller.FlowController.synchronize(FlowController.java
>>:1203) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at
>>org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(Standard
>>XMLFlowConfigurationDAO.java:72) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>> at
>>org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlow
>>Service.java:608) ~[nifi-framework-core-0.4.1.jar:0.4.1]
>>
>>
>>
>> But now I¹m not able to stop the processors due to this. How can I get
>>out of this situation I.e remove the problem node from the cluster.
>>
>>
>> Paresh
>>
>> 
>> The information contained in this transmission may contain privileged
>>and confidential information. It is intended only for the use of the
>>person(s) named above. If you are not the intended recipient, you are
>>hereby notified that any review, dissemination, distribution or
>>duplication of this communication is strictly prohibited. If you are not
>>the intended recipient, please contact the sender by reply email and
>>destroy all copies of the original message.
>> 
>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Unable to bring up cluster when the flow's go out of sync

[Paresh Shah]

I have the cluster in a state with the message on the problem node…


Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
configuration is not inheritable by the flow controller because of flow 
differences: Found difference in Flows:

Local Fingerprint:   
170e8ec-5b16-4564-aa31-11ea8bfeda3ehttp://10.233.0.183:8080/nifi8a003652-9b47-43d3-b067-e57c5c4d6da8RUNNING1false1f4c0f48-030f-4ba2-8dcd-464d31ad5d11352c0157-f3a2-32b4-a9e2-0139a56958a3ec38494d-b925-4

Cluster Fingerprint: 
170e8ec-5b16-4564-aa31-11ea8bfeda3ehttp://10.233.0.183:8080/nifi8a003652-9b47-43d3-b067-e57c5c4d6da8STOPPED1false1f4c0f48-030f-4ba2-8dcd-464d31ad5d11352c0157-f3a2-32b4-a9e2-0139a56958a3ec38494d-b925-4

at 
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:216)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1203) 
~[nifi-framework-core-0.4.1.jar:0.4.1]

at 
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:72)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]

at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:608)
 ~[nifi-framework-core-0.4.1.jar:0.4.1]



But now I’m not able to stop the processors due to this. How can I get out of 
this situation I.e remove the problem node from the cluster.


Paresh


The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

frequency of communication between two different wifi clusters

[Paresh Shah]

We are in the process of automating the pipeline deloyment across multiple 
clusters. In our scenario there are two different clusters with their own 
pipelines. And the ports are open on both the machines.

When I’m creating the RPG on the sending cluster it failed saying that the 
destination is not reachable. Then after 3/4 minutes I was able to successfully 
hook up the RPG. This 3/4 minutes is it configurable somewhere.

Paresh

The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Unable to update the targetUri for the RemoteProcessorGroup once created.

[Paresh Shah]

Matt,

Any ideas when will this fix be available. Since we are kind of blocked
without having this ability.

Thanks
Paresh

On 1/25/16, 5:22 PM, "Matt Gilman" <matt.c.gil...@gmail.com> wrote:

>Paresh,
>
>Currently, the definition of a Remote Process Group (inputs/outputs, etc)
>is driven by the instance running at the target URI. They are modeled both
>visually and structurally after local Process Groups. They allow one NiFi
>instance to communicate with another as though it was a local Process
>Group. In trying to keep as consistent as possible with local Process
>Groups changing the target URI was never implemented. It's certainly a
>valid use case as we already offer changing the target URI for various
>*Http Processors as well as changing available relationships for various
>RouteOn* Processors. I've created a JIRA [1] for adding this capability.
>
>Your comments about deployment into different environments is something
>that we will be addressing with the variable registry [2]. This will allow
>users to define variables which could be referenced in various locations
>throughout the application.
>
>Matt
>
>[1] https://issues.apache.org/jira/browse/NIFI-1440
>[2] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>
>On Mon, Jan 25, 2016 at 7:51 PM, Paresh Shah <paresh.s...@lifelock.com>
>wrote:
>
>> We are not able to update the ³targetUri² for a RPG once its
>>instantiated.
>> Is there any reason why this is the case. We need to be able to set it
>>at
>> deployment time depending upon the different env ( dev, staging Š)
>>
>> Another question was, why is the RPG not following the standard
>>processor
>> framework I.e why is it being treated as a special case.
>>
>> Any insights will be appreciated.
>>
>> Paresh
>>
>> 
>> The information contained in this transmission may contain privileged
>>and
>> confidential information. It is intended only for the use of the
>>person(s)
>> named above. If you are not the intended recipient, you are hereby
>>notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all
>>copies
>> of the original message.
>> 
>>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Unable to update the targetUri for the RemoteProcessorGroup once created.

[Paresh Shah]

We are not able to update the “targetUri” for a RPG once its instantiated. Is 
there any reason why this is the case. We need to be able to set it at 
deployment time depending upon the different env ( dev, staging …)

Another question was, why is the RPG not following the standard processor 
framework I.e why is it being treated as a special case.

Any insights will be appreciated.

Paresh


The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Unable to create Input Port using rest-api

[Paresh Shah]

Got it. I see it is hidden behind some other processors. So what about the
validation errors?

Is it safe to rely only the HTTP status code of 201.

On 1/22/16, 11:25 AM, "Matt Gilman" <matt.c.gil...@gmail.com> wrote:

>Paresh,
>
>So the Response in your original email was from the curl command?
>
>If so, can you trying searching in the UI for the value "TestCurlPort" or
>look in the birdseye/overview on the right to see if maybe the component
>is
>just not in the current viewport?
>
>Matt
>
>On Fri, Jan 22, 2016 at 2:20 PM, Paresh Shah <paresh.s...@lifelock.com>
>wrote:
>
>> Again.
>>
>> curl -X POST -H "Content-Type: application/x-www-form-urlencoded;
>> charset=UTF-8" http://lcess-groups/root/input-ports --data
>>
>>"version=58=2c4a82b5-1bec-4db6-af0c-d50f5facefcb=TestCurlPo
>>rt
>> =30=30"
>>
>>
>>
>> On 1/22/16, 11:16 AM, "Paresh Shah" <paresh.s...@lifelock.com> wrote:
>>
>> >Typo. The command is
>> >
>> >curl -X POST -H "Content-Type: application/x-www-form-urlencoded;
>> >charset=UTF-8"
>>
>>>http://localhost:9080/nifi-api/controller/process-groups/root/input-port
>>>s
>> >--data
>>
>>>"version=58=2c4a82b5-1bec-4db6-af0c-d50f5facefcb=TestCurlP
>>>or
>> >t
>> >"
>> >
>> >
>> >UI shows the yellow icon with the message what I see in the rest api
>> >response.
>> >
>> >On 1/22/16, 11:12 AM, "Matt Gilman" <matt.c.gil...@gmail.com> wrote:
>> >
>> >>Paresh,
>> >>
>> >>The URI in your curl command appears corrupt. I'm guessing that was an
>> >>issue with copying and pasting into the email.
>> >>
>> >>Just to clarify...
>> >>
>> >>1) The Response in your email is the response from the curl command?
>> >>2) What error message do you see in the UI?
>> >>
>> >>Matt
>> >>
>> >>
>> >>On Fri, Jan 22, 2016 at 2:03 PM, Paresh Shah
>><paresh.s...@lifelock.com>
>> >>wrote:
>> >>
>> >>> I have not been able to create a InputPort at the root level using
>>the
>> >>> following.
>> >>>
>> >>>
>> >>> curl -X POST -H "Content-Type: application/x-www-form-urlencoded;
>> >>> charset=UTF-8" http://lcess-groups/root/input-ports --data
>> >>>
>>
>>>>>"version=58=2c4a82b5-1bec-4db6-af0c-d50f5facefcb=TestCur
>>>>>lP
>> >>>o
>> >>>rt=30=30²
>> >>>
>> >>>
>> >>> Response:
>> >>>
>> >>>
>> >>>
>>
>>>>>{"revision":{"clientId":"2c4a82b5-1bec-4db6-af0c-d50f5facefcb","versio
>>>>>n"
>> >>>:
>> >>>59},"inputPort":{"id":"300ed55b-2c10-4786-8b6c-79c45165b522","uri":"
>> >>>
>> >>>
>> http://localhost:9080/nifi-api/controller/process-groups/d68f3eef-301a-4
>> >>>6
>>
>>>>>db-b88c-6c015b55ebf9/input-ports/300ed55b-2c10-4786-8b6c-79c45165b522"
>>>>>,"
>> >>>p
>>
>>>>>osition":{"x":30.0,"y":30.0},"parentGroupId":"d68f3eef-301a-46db-b88c-
>>>>>6c
>> >>>0
>>
>>>>>15b55ebf9","name":"TestCurlPort","state":"STOPPED","type":"INPUT_PORT"
>>>>>,"
>> >>>t
>>
>>>>>ransmitting":false,"concurrentlySchedulableTaskCount":1,"userAccessCon
>>>>>tr
>> >>>o
>> >>>l":[],"groupAccessControl":[],"validationErrors":["'Port
>> >>> 'TestCurlPort'' is invalid because Output connection for port
>> >>> 'TestCurlPort' is not defined."]}}
>> >>>
>> >>>
>> >>> I do not see the port on the UI. When we do the same using
>>drag-drop we
>> >>> are seeing the error message but the port does get instantiated.
>> >>>
>> >>>
>> >>> Paresh
>> >>>
>> >>> 
>> >>> The information contained in this transmission may contain
>>privileged
>> >>>and
>> >>> confidential information. It is intended only for the use of the
>> >>>person(s)
>> >>> named above. If you are not the intended recipient, you are hereby
>> >>>notified
>> >>> that any review, dissemination, distribution or duplication of this
>> >>> communication is strictly prohibited. If you are not the intended
>> >>> recipient, please contact the sender by reply email and destroy all
>> >>>copies
>> >>> of the original message.
>> >>> 
>> >>>
>> >
>>
>> 
>>  The information contained in this transmission may contain privileged
>>and
>> confidential information. It is intended only for the use of the
>>person(s)
>> named above. If you are not the intended recipient, you are hereby
>>notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all
>>copies
>> of the original message.
>> 
>>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Unable to create Input Port using rest-api

[Paresh Shah]

Typo. The command is

curl -X POST -H "Content-Type: application/x-www-form-urlencoded;
charset=UTF-8"
http://localhost:9080/nifi-api/controller/process-groups/root/input-ports
--data
"version=58=2c4a82b5-1bec-4db6-af0c-d50f5facefcb=TestCurlPort
"


UI shows the yellow icon with the message what I see in the rest api
response.

On 1/22/16, 11:12 AM, "Matt Gilman" <matt.c.gil...@gmail.com> wrote:

>Paresh,
>
>The URI in your curl command appears corrupt. I'm guessing that was an
>issue with copying and pasting into the email.
>
>Just to clarify...
>
>1) The Response in your email is the response from the curl command?
>2) What error message do you see in the UI?
>
>Matt
>
>
>On Fri, Jan 22, 2016 at 2:03 PM, Paresh Shah <paresh.s...@lifelock.com>
>wrote:
>
>> I have not been able to create a InputPort at the root level using the
>> following.
>>
>>
>> curl -X POST -H "Content-Type: application/x-www-form-urlencoded;
>> charset=UTF-8" http://lcess-groups/root/input-ports --data
>>
>>"version=58=2c4a82b5-1bec-4db6-af0c-d50f5facefcb=TestCurlPo
>>rt=30=30²
>>
>>
>> Response:
>>
>>
>>
>>{"revision":{"clientId":"2c4a82b5-1bec-4db6-af0c-d50f5facefcb","version":
>>59},"inputPort":{"id":"300ed55b-2c10-4786-8b6c-79c45165b522","uri":"
>>
>>http://localhost:9080/nifi-api/controller/process-groups/d68f3eef-301a-46
>>db-b88c-6c015b55ebf9/input-ports/300ed55b-2c10-4786-8b6c-79c45165b522","p
>>osition":{"x":30.0,"y":30.0},"parentGroupId":"d68f3eef-301a-46db-b88c-6c0
>>15b55ebf9","name":"TestCurlPort","state":"STOPPED","type":"INPUT_PORT","t
>>ransmitting":false,"concurrentlySchedulableTaskCount":1,"userAccessContro
>>l":[],"groupAccessControl":[],"validationErrors":["'Port
>> 'TestCurlPort'' is invalid because Output connection for port
>> 'TestCurlPort' is not defined."]}}
>>
>>
>> I do not see the port on the UI. When we do the same using drag-drop we
>> are seeing the error message but the port does get instantiated.
>>
>>
>> Paresh
>>
>> 
>> The information contained in this transmission may contain privileged
>>and
>> confidential information. It is intended only for the use of the
>>person(s)
>> named above. If you are not the intended recipient, you are hereby
>>notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all
>>copies
>> of the original message.
>> 
>>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Unable to create Input Port using rest-api

[Paresh Shah]

I have not been able to create a InputPort at the root level using the 
following.


curl -X POST -H "Content-Type: application/x-www-form-urlencoded; 
charset=UTF-8" http://lcess-groups/root/input-ports --data 
"version=58=2c4a82b5-1bec-4db6-af0c-d50f5facefcb=TestCurlPort=30=30”


Response:

{"revision":{"clientId":"2c4a82b5-1bec-4db6-af0c-d50f5facefcb","version":59},"inputPort":{"id":"300ed55b-2c10-4786-8b6c-79c45165b522","uri":"http://localhost:9080/nifi-api/controller/process-groups/d68f3eef-301a-46db-b88c-6c015b55ebf9/input-ports/300ed55b-2c10-4786-8b6c-79c45165b522","position":{"x":30.0,"y":30.0},"parentGroupId":"d68f3eef-301a-46db-b88c-6c015b55ebf9","name":"TestCurlPort","state":"STOPPED","type":"INPUT_PORT","transmitting":false,"concurrentlySchedulableTaskCount":1,"userAccessControl":[],"groupAccessControl":[],"validationErrors":[;'Port
 'TestCurlPort'' is invalid because Output connection for port 'TestCurlPort' 
is not defined."]}}


I do not see the port on the UI. When we do the same using drag-drop we are 
seeing the error message but the port does get instantiated.


Paresh


The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Unable to pick different input ports in the RemoteProcessorGroup

[Paresh Shah]

To clarify I do see the ports in the RPG but am not able to turn them
“On”. That option is only available for the first in the list.

Paresh

On 1/21/16, 9:55 AM, "Paresh Shah" <paresh.s...@lifelock.com> wrote:

>I¹m not sure where I would see the ports being called remote group ports.
>Using the UI I picked the Input Port on to the canvas at the root level
>and am able to see one of them successfully in the RemoteProcessorGroup.
>Also I do not see any place where I can set the authorization level.
>
>
>
>On 1/21/16, 9:50 AM, "Joe Witt" <joe.w...@gmail.com> wrote:
>
>>Paresh,
>>
>>Screenshots through the apache mailing lists require pretty specific
>>delivery mechanism which honestly I don't know.  So...best to find a
>>way to link to screenshots or to put them in a JIRA.
>>
>>When you create InputPorts at the root group level they become
>>available as ports that another authorized system can see.  If they
>>are not at the root group level or if they are not authorized to be
>>seen by the other system they will not show up in the list.
>>
>>Can you confirm that these are remote group ports?
>>
>>Thanks
>>Joe
>>
>>On Thu, Jan 21, 2016 at 12:48 PM, Paresh Shah <paresh.s...@lifelock.com>
>>wrote:
>>> I have created different set of InputPorts for the different pipelines.
>>>But
>>> when I see the list of InputPorts in the RPG I do not see option to
>>>pick
>>> other ports except the first one in the list.
>>>
>>> I have attached the screeshots
>>>
>>>
>>> 
>>> The information contained in this transmission may contain privileged
>>>and
>>> confidential information. It is intended only for the use of the
>>>person(s)
>>> named above. If you are not the intended recipient, you are hereby
>>>notified
>>> that any review, dissemination, distribution or duplication of this
>>> communication is strictly prohibited. If you are not the intended
>>>recipient,
>>> please contact the sender by reply email and destroy all copies of the
>>> original message.
>>> 
>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Unable to pick different input ports in the RemoteProcessorGroup

[Paresh Shah]

I¹m not sure where I would see the ports being called remote group ports.
Using the UI I picked the Input Port on to the canvas at the root level
and am able to see one of them successfully in the RemoteProcessorGroup.
Also I do not see any place where I can set the authorization level.



On 1/21/16, 9:50 AM, "Joe Witt" <joe.w...@gmail.com> wrote:

>Paresh,
>
>Screenshots through the apache mailing lists require pretty specific
>delivery mechanism which honestly I don't know.  So...best to find a
>way to link to screenshots or to put them in a JIRA.
>
>When you create InputPorts at the root group level they become
>available as ports that another authorized system can see.  If they
>are not at the root group level or if they are not authorized to be
>seen by the other system they will not show up in the list.
>
>Can you confirm that these are remote group ports?
>
>Thanks
>Joe
>
>On Thu, Jan 21, 2016 at 12:48 PM, Paresh Shah <paresh.s...@lifelock.com>
>wrote:
>> I have created different set of InputPorts for the different pipelines.
>>But
>> when I see the list of InputPorts in the RPG I do not see option to pick
>> other ports except the first one in the list.
>>
>> I have attached the screeshots
>>
>>
>> 
>> The information contained in this transmission may contain privileged
>>and
>> confidential information. It is intended only for the use of the
>>person(s)
>> named above. If you are not the intended recipient, you are hereby
>>notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>>recipient,
>> please contact the sender by reply email and destroy all copies of the
>> original message.
>> 


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Unable to pick different input ports in the RemoteProcessorGroup

[Paresh Shah]

I did not see the option to pick the port when dragging the RPG. This is
no longer a problem.


On 1/21/16, 10:10 AM, "Paresh Shah" <paresh.s...@lifelock.com> wrote:

>To clarify I do see the ports in the RPG but am not able to turn them
>“On”. That option is only available for the first in the list.
>
>Paresh
>
>On 1/21/16, 9:55 AM, "Paresh Shah" <paresh.s...@lifelock.com> wrote:
>
>>I¹m not sure where I would see the ports being called remote group ports.
>>Using the UI I picked the Input Port on to the canvas at the root level
>>and am able to see one of them successfully in the RemoteProcessorGroup.
>>Also I do not see any place where I can set the authorization level.
>>
>>
>>
>>On 1/21/16, 9:50 AM, "Joe Witt" <joe.w...@gmail.com> wrote:
>>
>>>Paresh,
>>>
>>>Screenshots through the apache mailing lists require pretty specific
>>>delivery mechanism which honestly I don't know.  So...best to find a
>>>way to link to screenshots or to put them in a JIRA.
>>>
>>>When you create InputPorts at the root group level they become
>>>available as ports that another authorized system can see.  If they
>>>are not at the root group level or if they are not authorized to be
>>>seen by the other system they will not show up in the list.
>>>
>>>Can you confirm that these are remote group ports?
>>>
>>>Thanks
>>>Joe
>>>
>>>On Thu, Jan 21, 2016 at 12:48 PM, Paresh Shah <paresh.s...@lifelock.com>
>>>wrote:
>>>> I have created different set of InputPorts for the different
>>>>pipelines.
>>>>But
>>>> when I see the list of InputPorts in the RPG I do not see option to
>>>>pick
>>>> other ports except the first one in the list.
>>>>
>>>> I have attached the screeshots
>>>>
>>>>
>>>> 
>>>> The information contained in this transmission may contain privileged
>>>>and
>>>> confidential information. It is intended only for the use of the
>>>>person(s)
>>>> named above. If you are not the intended recipient, you are hereby
>>>>notified
>>>> that any review, dissemination, distribution or duplication of this
>>>> communication is strictly prohibited. If you are not the intended
>>>>recipient,
>>>> please contact the sender by reply email and destroy all copies of the
>>>> original message.
>>>> 
>>
>
>
> The information contained in this transmission may contain privileged
>and confidential information. It is intended only for the use of the
>person(s) named above. If you are not the intended recipient, you are
>hereby notified that any review, dissemination, distribution or
>duplication of this communication is strictly prohibited. If you are not
>the intended recipient, please contact the sender by reply email and
>destroy all copies of the original message.
>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Why the Input Ports are not allowed to be inside the ProcessorGroups

[Paresh Shah]

In our use case to keep the UI managable we create different ProcessorGroups 
for the different pipelines. But when using the RemoteProcessorGroups and 
InputPorts we are forced to move the pipelines out of the processor groups and 
into the root canvas. This helps us tear down and do the deployment cleanly.

If someone can give reasons for the above and also can it be changed so that we 
can work with RPG/InputPorts/ProcessorGroups.

Thanks
Paresh

The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: How to configure/start multiple Input ports.

[Paresh Shah]

Another question.

To use the Remote Processor Group, on the sender side should we do the 
following.

  1.  Create Output Port.
  2.  Create Remote Processor Group using the OutputPort created.

Or

1. Create the target pipeline with the input port
2. Create Remote Processor Group with the input port -> This gives the 
error that it has timed out reading from the port. But we need to be sending 
the data to this port rather then read from it.

Thanks
Paresh

From: Paresh Shah <paresh.s...@lifelock.com<mailto:paresh.s...@lifelock.com>>
Date: Wednesday, January 20, 2016 at 4:01 PM
To: "dev@nifi.apache.org<mailto:dev@nifi.apache.org>" 
<dev@nifi.apache.org<mailto:dev@nifi.apache.org>>
Subject: Re: How to configure/start multiple Input ports.

Still don’t see the images.

Also if the Input Ports are at the root canvas, we are not able to connect it 
to the ProcessorGroup. We have all our pipelines within different processor 
groups.

Paresh

From: Matthew Clarke 
<matt.clarke@gmail.com<mailto:matt.clarke@gmail.com>>
Reply-To: "dev@nifi.apache.org<mailto:dev@nifi.apache.org>" 
<dev@nifi.apache.org<mailto:dev@nifi.apache.org>>
Date: Wednesday, January 20, 2016 at 2:37 PM
To: "dev@nifi.apache.org<mailto:dev@nifi.apache.org>" 
<dev@nifi.apache.org<mailto:dev@nifi.apache.org>>
Subject: Re: How to configure/start multiple Input ports.

Three images attached:

On Wed, Jan 20, 2016 at 4:31 PM, Paresh Shah 
<paresh.s...@lifelock.com<mailto:paresh.s...@lifelock.com>> wrote:
I cannot see the pictures. Can you please resend it with attachements.

Thanks
Paresh

From: Matthew Clarke 
<matt.clarke@gmail.com<mailto:matt.clarke@gmail.com><mailto:matt.clarke@gmail.com<mailto:matt.clarke@gmail.com>>>
Reply-To: 
"dev@nifi.apache.org<mailto:dev@nifi.apache.org><mailto:dev@nifi.apache.org<mailto:dev@nifi.apache.org>>"
 
<dev@nifi.apache.org<mailto:dev@nifi.apache.org><mailto:dev@nifi.apache.org<mailto:dev@nifi.apache.org>>>
Date: Wednesday, January 20, 2016 at 12:55 PM
To: 
"dev@nifi.apache.org<mailto:dev@nifi.apache.org><mailto:dev@nifi.apache.org<mailto:dev@nifi.apache.org>>"
 
<dev@nifi.apache.org<mailto:dev@nifi.apache.org><mailto:dev@nifi.apache.org<mailto:dev@nifi.apache.org>>>
Subject: Re: How to configure/start multiple Input ports.

Paresh,
   All Site-to-Site data comes across a single port connection.  When 
adding a RPG to the source NiFi, the URL you provide will be the URL for the 
NCM of your internal NiFi cluster.  You will add a a different RPG for each 
cluster you are connecting to. so in your case the external facing NiFi cluster 
will have 2 RPGs  on the graph with 1 configured form the URL of the NCM on 
internal NiFi cluster 1 and the other for the NCM of internal NiFi cluster 2.  
Once a connection is established to an NCM, the NCM will communicate to the RPG 
the URLs and site-to-site ports for each of the clusters connected Nodes as 
well as the current load on those nodes.  The NiFi with the RPG will then do a 
smart load-balanced delivery of data to those nodes. On the target cluster you 
will add input ports at the root level of the graph (They cannot be nested 
inside of any process groups) .  You can add as many uniquely named input ports 
as you would like.  These input ports will be exposed to the RPG on the sending 
system.  When you draw a connection to the RPG, you will be provided witha 
pull-down selection menu of all available ports. Selecting one will complete 
this connection.  So all data between NiFi systems via Site-to-Site will use 
the same site-to-site port but as you can see it enters the flow on the 
destination system via input ports.

[Inline image 1]

So on your externally visible NiFi you would have two RPGs setup like below:
[Inline image 3]

The receiving NiFi cluster will have one or more  uniquely named input ports:

[Inline image 4]

As you can see from the above two screenshots the sending system's connection 
to the RPG shows that the connection as being connected to the input port 
"Node-Input" on the receiving cluster.


Thanks,
Matt



On Wed, Jan 20, 2016 at 2:59 PM, Paresh Shah 
<paresh.s...@lifelock.com<mailto:paresh.s...@lifelock.com><mailto:paresh.s...@lifelock.com<mailto:paresh.s...@lifelock.com>>>
 wrote:

We are trying to see how to run our pipelines in a clustered env using 
site-to-site. Our scenario is something like the following.

  1.  3 different clusters.
  2.  One of the clusters is externally visible and will be the primary cluster.
  3.  All the data receivers will run on the primary node.

Each of the pipelines has a RemoteProcessorGroup( RPG ) that would in –turn 
forward the data to their corresponding Input port.

Loo

How to configure/start multiple Input ports.

[Paresh Shah]

We are trying to see how to run our pipelines in a clustered env using 
site-to-site. Our scenario is something like the following.

  1.  3 different clusters.
  2.  One of the clusters is externally visible and will be the primary cluster.
  3.  All the data receivers will run on the primary node.

Each of the pipelines has a RemoteProcessorGroup( RPG ) that would in –turn 
forward the data to their corresponding Input port.

Looking at the way the site-to-site configuration indicates that we are only 
able to specify one port that would be started on a particular node. So the 
question is how can we pin a particular RPG to it corresponding InputPort.

Any insights would be greatly appreciated.

Thanks
Paresh

The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: How to configure/start multiple Input ports.

[Paresh Shah]

I cannot see the pictures. Can you please resend it with attachements.

Thanks
Paresh

From: Matthew Clarke 
<matt.clarke@gmail.com<mailto:matt.clarke@gmail.com>>
Reply-To: "dev@nifi.apache.org<mailto:dev@nifi.apache.org>" 
<dev@nifi.apache.org<mailto:dev@nifi.apache.org>>
Date: Wednesday, January 20, 2016 at 12:55 PM
To: "dev@nifi.apache.org<mailto:dev@nifi.apache.org>" 
<dev@nifi.apache.org<mailto:dev@nifi.apache.org>>
Subject: Re: How to configure/start multiple Input ports.

Paresh,
   All Site-to-Site data comes across a single port connection.  When 
adding a RPG to the source NiFi, the URL you provide will be the URL for the 
NCM of your internal NiFi cluster.  You will add a a different RPG for each 
cluster you are connecting to. so in your case the external facing NiFi cluster 
will have 2 RPGs  on the graph with 1 configured form the URL of the NCM on 
internal NiFi cluster 1 and the other for the NCM of internal NiFi cluster 2.  
Once a connection is established to an NCM, the NCM will communicate to the RPG 
the URLs and site-to-site ports for each of the clusters connected Nodes as 
well as the current load on those nodes.  The NiFi with the RPG will then do a 
smart load-balanced delivery of data to those nodes. On the target cluster you 
will add input ports at the root level of the graph (They cannot be nested 
inside of any process groups) .  You can add as many uniquely named input ports 
as you would like.  These input ports will be exposed to the RPG on the sending 
system.  When you draw a connection to the RPG, you will be provided witha 
pull-down selection menu of all available ports. Selecting one will complete 
this connection.  So all data between NiFi systems via Site-to-Site will use 
the same site-to-site port but as you can see it enters the flow on the 
destination system via input ports.

[Inline image 1]

So on your externally visible NiFi you would have two RPGs setup like below:
[Inline image 3]

The receiving NiFi cluster will have one or more  uniquely named input ports:

[Inline image 4]

As you can see from the above two screenshots the sending system's connection 
to the RPG shows that the connection as being connected to the input port 
"Node-Input" on the receiving cluster.


Thanks,
Matt



On Wed, Jan 20, 2016 at 2:59 PM, Paresh Shah 
<paresh.s...@lifelock.com<mailto:paresh.s...@lifelock.com>> wrote:

We are trying to see how to run our pipelines in a clustered env using 
site-to-site. Our scenario is something like the following.

  1.  3 different clusters.
  2.  One of the clusters is externally visible and will be the primary cluster.
  3.  All the data receivers will run on the primary node.

Each of the pipelines has a RemoteProcessorGroup( RPG ) that would in –turn 
forward the data to their corresponding Input port.

Looking at the way the site-to-site configuration indicates that we are only 
able to specify one port that would be started on a particular node. So the 
question is how can we pin a particular RPG to it corresponding InputPort.

Any insights would be greatly appreciated.

Thanks
Paresh

The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.



The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Handling of moving the contents of a large file through NIFI.

[Paresh Shah]

We have the following use case:
On a scheduled basis, reading of a large no of records from an external system 
and moving the records through the NIFI pipeline.

What we see is that the flowFiles are not moved to the relationship until the 
session is committed. And once the session is committed we are not able to 
transfer anything else on that session.

We see that in GetFileTransfer where the entire file contents are moved using 
the “importFrom” api on the session. But since we need to handle the individual 
records in the pipeline it does not work for our use case.

Is there a different mechanism to do what we want. Any insights will be 
appreciated.

Thanks
Paresh

The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Re: Handling of moving the contents of a large file through NIFI.

[Paresh Shah]

The file is quite large. So reading all of it into the flow file is not an
option. We wanting to be processing the individual records of the file,
that is why are creating ³n² flow files.

Paresh

On 12/13/15, 11:11 AM, "Joe Witt" <joe.w...@gmail.com> wrote:

>Paresh
>
>Is it feasible to read the large object I to the flow and then split as
>needed?  It sounds like you are reading from the orig file in place and
>making flowfiles from it.
>
>Perhaps you can share a screenshot of the flow?
>
>Thanks
>Joe
>On Dec 13, 2015 1:49 PM, "Paresh Shah" <paresh.s...@lifelock.com> wrote:
>
>> We have the following use case:
>> On a scheduled basis, reading of a large no of records from an external
>> system and moving the records through the NIFI pipeline.
>>
>> What we see is that the flowFiles are not moved to the relationship
>>until
>> the session is committed. And once the session is committed we are not
>>able
>> to transfer anything else on that session.
>>
>> We see that in GetFileTransfer where the entire file contents are moved
>> using the ³importFrom² api on the session. But since we need to handle
>>the
>> individual records in the pipeline it does not work for our use case.
>>
>> Is there a different mechanism to do what we want. Any insights will be
>> appreciated.
>>
>> Thanks
>> Paresh
>> 
>> The information contained in this transmission may contain privileged
>>and
>> confidential information. It is intended only for the use of the
>>person(s)
>> named above. If you are not the intended recipient, you are hereby
>>notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all
>>copies
>> of the original message.
>> 
>>


 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

Questions about the ordering of the FlowFile.

[Paresh Shah]

Here’s my use case.
We have a application protocol between the start and end processors in a data 
flow, that expect the flow files to arrive in the order they are generated. For 
e.g

Start Record Flowfile

End Record Flowfile.

The first processor does the following.

  1.  Generates and transfers the StartRecord flow file.
  2.  Generates data records and transfers them.
  3.  Generates and transfers the EndRecord flow file

The last processor in the data flow does the following.

  1. Looks for the StartRecord flow file and does its thing.
  2. Looks for the DataRecord flow file and does its thing.
  3.  Looks for the EndRecord flow file and updates and cleanups up the 
target state.

The first processor is doing multiple transfers on the session object before 
calling commit.

We see that they are being received in random order. As a result we are not 
able to execute the app protocol. We have tried the FirstInFirstOutPrioritizer 
and OldestFlowFilePrioritizer.

We would appreciate any insights into this we can get as it seems to be a 
blocking issue for us.

Thanks
Paresh

The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.