Re: Rehosting flow files from cluster nodes to primary node only

2020-11-13 Thread James McMahon
Thank you both Andrew and Pierre. I think I’ll try both approaches on our integration system to see how each behaves in our typical conditions for throughput. Cheers, Jim On Fri, Nov 13, 2020 at 2:50 PM Pierre Villard wrote: > Hi, > > You can configure your relationship before the ExecuteScript

Re: Rehosting flow files from cluster nodes to primary node only

2020-11-13 Thread Pierre Villard
Hi, You can configure your relationship before the ExecuteScript processor to use the load balance strategy to "single node" so that all the data gets to one node (not necessarily the primary node so you'd still keep the processor configured to be executed on all nodes). Pierre Le ven. 13 nov. 2

Re: Rehosting flow files from cluster nodes to primary node only

2020-11-13 Thread Andrew Grande
I think a better design is for every node to write to this network share with some form of a partition (node) id in the filename. In a 5 node cluster you will have 5 parts. Next, either query over this directory directly with your engine of choice. Reducing all traffic to 1 node will be a bottlen

Rehosting flow files from cluster nodes to primary node only

2020-11-13 Thread James McMahon
My flow files are distributed across my cluster nodes. I am using an ExecuteScript processor running a python script to write custom log messages to a log file I maintain. It appears that i am losing records when all the nodes are attempting to write to the one log file that exists on a common netw