Hi Mark, One more thing I have noticed about the putHDFS that only two files has been written and rest of them are in complete. Please refer image. https://i.imgur.com/hu26hQG.png
Thank you, Mark On 2019/12/04 07:30:03, nayan sharma <nayansharm...@gmail.com> wrote: > Hi Mark, > Thanks for your valuable suggestion. It worked a lot. Now I can understand, > there is no point in load balancing between FetchSFTP and CompressContent. > > After making all the changes it worked but some of the flow files are stuck > between CompressContent and putHDFS https://i.imgur.com/oSYkYuA.png > > And 2nd thing is that 10 FlowFiles between ListSFTP and FetchSFTP is there > for long time > https://i.imgur.com/Q44VDW6.png > > Please suggested where I can start debugging these two issues. > > Meanwhile we are migrating to 1.10.0. This time we are doing through HDF and > it has NIFI 1.9.0 as latest version. We are planing to replace the library > and content of 1.9.0 with 1.10.0. Can we go ahead with this approach or is > there are other way. > > Currently 1.9.2 is an independent cluster. > > > > On 2019/12/03 14:30:43, Mark Payne <marka...@hotmail.com> wrote: > > Nayan, > > > > Looking at the screenshot, I can see two different connections there that > > are load balanced. One of them holds the nearly 100 GB of data. > > > > There are a handful of bugs related to load-balanced connections in 1.9.2 > > that were addressed in 1.10.0. If you're relying on load-balanced > > connections to spread data across the cluster (and this particular flow > > clearly is), then I would strongly encourage you to upgrade to 1.10.0 > > because at least one of these bugs does cause the flow to appear to stop > > flowing. > > > > That being said, there are two other things that you may want to consider: > > > > 1. You're trying to load balance 100 GB of data spread across 6 files. So > > each file is nearly 20 GB of data. It may take a little while to push that > > from Node A to Node B. If the data is queued up, waiting to go to another > > node, or is on the way to another node, it will not be shown in the > > FlowFile listing. That will only show FlowFiles that are queued up to be > > processed on the node that it currently lives on. > > > > 2. You should not be using a load balanced connection between FetchSFTP and > > CompressContent. The way that these processors are designed, the listing > > should be performed, and then the connection between ListSFTP and FetchSFTP > > should be load balanced. Once that has happened, the listing has been > > federated across the cluster, so whichever node receives the listing for > > File A should be responsible for fetching and processing it. Since the > > listing has already been spread across the cluster, there is no benefit to > > fetching the data, and then re-spreading it across the cluster. This will > > be very expensive with little to no benefit. Similarly, you don't want to > > load balance between CompressContent and PutHDFS. Simply load balance the > > listing itself (which is very cheap because the FlowFiles have no content) > > and the data will automatically be balanced across the cluster. > > > > Thanks > > -Mark > > > > > > > On Dec 3, 2019, at 9:18 AM, nayan sharma <nayansharm...@gmail.com> wrote: > > > > > > Hi, > > > Thanks for your reply. > > > Please find the attachment. Flow files has been for last 7 days. And > > > while listing flow files it says The queue has no Flow Files. > > > Let me know your thoughts. > > > > > > Thanks & Regards, > > > Nayan Sharma > > > +91-8095382952 > > > > > > <https://www.linkedin.com/in/nayan-sharma> > > > <http://stackoverflow.com/users/3687426/nayan-sharma?tab=profile> > > > > > > On Tue, Dec 3, 2019 at 7:34 PM Bryan Bende <bbe...@gmail.com > > > <mailto:bbe...@gmail.com>> wrote: > > > Hello, > > > > > > It would be helpful if you could upload a screenshot of your flow > > > somewhere and send a link. > > > > > > Thanks, > > > > > > Bryan > > > > > > On Tue, Dec 3, 2019 at 6:06 AM nayan sharma <nayansharm...@gmail.com > > > <mailto:nayansharm...@gmail.com>> wrote: > > > > > > > > Hi, > > > > I am using 2 nodes cluster. > > > > nodes config Heap(max) 48gb & 64 core machine > > > > Processor flow > > > > ListSFTP--->FetchSFTP(all nodes with 10 threads)--->CompressContent(all > > > > nodes,10 threads)-->PutHDFS > > > > > > > > Queues shows it has 96gb in queue but when I do listing it shows no > > > > flow files. > > > > > > > > Everything seems stuck, nothing is moving. > > > > > > > > I was wondering and curious also even if with such heavy machines, > > > > What I am doing wrong or with which config parameter. > > > > > > > > I couldn't find out solution for by myself so I reached here. Any help > > > > or suggestion will be much highly appreciated. > > > > > > > > Thanks, > > > > Nayan > > > <Screenshot 2019-12-03 at 7.44.25 PM.png> > > > > >
