Hi Mike, Depending on the specifics of the inputs (source, format, correlating attributes, etc.) it sounds like you have a few good options to achieve this.
1. MergeContent [1] — This processor is very old (read: stable & widely-used, not legacy) and is used to merge content from multiple incoming flowfiles into batches according to a variety of strategies. There are many threads on these lists and articles/Stack Overflow/HCC posts about it, as it is very versatile. 2. Wait/Notify [2] — New in NiFi 1.2.0 (not yet released but available currently on the master branch), Joe Gresock and Koji Kawamura have contributed some very useful processors which allow you to wait for an arbitrary number of “signals” to invoke further processors. See the PRs [3][4][5] and excellent explanation [6]. 3. ExecuteScript — If neither of those options work for you, although I suspect they will, you can write custom code in any of 6 languages (Ruby, Python, Lua, Javascript, Groovy, or Java) and run it with the ExecuteScript or InvokeScriptedProcessor to perform arbitrary operations without having to build a complete NAR and custom processors. Of course, you can still do that if you desire. Hope that helps. Please let us know if you have any other questions. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html <https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html> [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java> [3] https://github.com/apache/nifi/pull/1329 <https://github.com/apache/nifi/pull/1329> [4] https://github.com/apache/nifi/pull/1420 <https://github.com/apache/nifi/pull/1420> [5] https://github.com/apache/nifi/pull/1490 <https://github.com/apache/nifi/pull/1490> [6] https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67 <https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Mar 15, 2017, at 6:38 PM, Michael John LEWIS <michaelle...@dbs.com> wrote: > > Hi, > > I’ve just started taking a look at Nifi as it seems perfect for a project I’m > working on right now. > > The one thing I’m trying to solve is this – I have a process (say a batch job > or similar) that I want to invoke, > only once all the inputs have arrived ( a simple dependency graph ). > > i.e. I have 3 files and I don’t want to run a process after each file > arrival, but once all 3 files have arrived. > > Is this easily achievable in Nifi ? Or would I have to think about custom > processor? > > Thanks in advance, > > Mike > > > CONFIDENTIAL NOTE: > > The information contained in this email is intended only for the use of the > individual or entity named above and may contain information that is > privileged, confidential and exempt from disclosure under applicable law. If > the reader of this message is not the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you have received this message in > error, please immediately notify the sender and delete the mail. Thank you.
signature.asc
Description: Message signed with OpenPGP using GPGMail
