Re: RouteText / Replcace Text to remove first line of file

2017-03-06 Thread Mark Payne
Joe, In terms of updating a processor to better handle this, we could update RouteText to do so. The idea being that if a large percentage of the time (say > 80% of the time) the FlowFile routed to one of the relationships is a single, contiguous subset of the data in the original FlowFile, the

[ANNOUNCE] CVE-2017-5635 and CVE-2017-5636

2017-03-06 Thread Andy LoPresto
Apache NiFi PMC would like to announce the discovery and resolution of CVE-2017-5635 and CVE-2017-5636. These issues have been resolved and new versions of the Apache NiFi project were released in accordance with the Apache Release Process. Fixed in Apache NiFi 0.7.2 and 1.1.2 CVE-2107-5635: A

Re: RouteText / Replcace Text to remove first line of file

2017-03-06 Thread Joe Witt
Juan, I think RouteText is the right answer. It would indeed need to check all lines to determine whether the condition is satisfied and to remove the first line it will need to write out all the remaining lines. If a majority of the input files do not have this problematic header I'd use RouteO

Re: RouteText / Replcace Text to remove first line of file

2017-03-06 Thread Lee Laim
Juan, If you're in the linux environment, you can use the Execute Stream command (ESC) processor to run "head -n 1" the contents of the incoming large flowfile to quickly extract the first line. ESC has an option to put the output of the command directly into a new attribute, and pass the "orig

RouteText / Replcace Text to remove first line of file

2017-03-06 Thread Juan Sequeiros
Good afternoon all,, I am trying to remove the first line of a file if it has a certain word in it "ERROR" I know it will exist only in the first line ( I can not fix the reason why it gets put there ) These files are big and lots of them. and I can not find a "fast" fix to pop the first line of

Re: Merge join processor for nifi

2017-03-06 Thread Joe Witt
Purbon The MergeContent processor supports merging data using a correlation key. The processor does not support SQL style inner/outer joins as it is not joining data based on per record/tuple type logic but rather at a per flowfile level. We can correlated based on any flowfile attribute or comb

Merge join processor for nifi

2017-03-06 Thread Pere Urbón Bayes
HI, my name is Pere and I am kinda new to nifi, working now on integrating this awesome project as a data pipeline tool for a project I am working on. In this project I aim to get a collection of FlowFiles what share some kind of unique key, then might be a join operation involved. I am wonderin

Re: Visual Indicator for "Can't run because there are no threads"?

2017-03-06 Thread Andy LoPresto
I created NIFI-3558 [1] to capture these scenarios. I added this specific example, but if anyone has more, please contribute them on the ticket. [1] https://issues.apache.org/jira/browse/NIFI-3558 Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F

Re: Visual Indicator for "Can't run because there are no threads"?

2017-03-06 Thread Andy LoPresto
Peter, There have been intermittent discussions around a “system status/configuration traffic light tool” which would be a visual indicator in the UI that addresses common problems that are easily attributed to a specific configuration value or environment scenario not matching best practices.

RE: Visual Indicator for "Can't run because there are no threads"?

2017-03-06 Thread Peter Wicks (pwicks)
Joe, In my case I had not seen the issue until I added 7 new QueryDatabaseProcessor's. All seven of them kicked off against the same SQL database on restart and took 10 to 15 minutes to come back. During that time my default 10 threads was running with only 3 to spare, which were being shared

NiFi 1.1.1 AWS EC2 Secure Cluster Zookeeper Connection Loss Error

2017-03-06 Thread Ryan H
Hi All, I am running into another issue setting up a secure NiFi cluster across 2 EC2 instances in AWS. Shortly after starting up the two nodes, the nifi-app.log is completely spammed with the following error message(s): 2017-03-06 13:48:06,029 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFram

Re: How to specify priority attributes for seperate flowfiles?

2017-03-06 Thread prabhu Mahendran
Bryan & Andre, PriorityAttributePrioritizer ,FIFO strategy worked if flow has no loop processing. But i have configured loop processing in my workflow by using this reference. https://gist.github.com/ijokarumawak/01c4fd2d9291d3e74ec424a581659ca8#file-loop_sample-xml For example if i process tw