Is it possible to count records in a MongoDb collection using the GetMongo processor?
Hi everyone, Is there a way to use the GetMongo processor to count the number of records in a collection that has a certain value for a field in the collection? For example, how would I count the number of documents in the Names collection where firstName = Steve? Thank you, Steven K. Byers Perspecta Inc. Contractor Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC) Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC 1681 Nelson Street, Fort Detrick, MD 21702 (443) 538-7575 | (410) 872-4923 Email: steven.k.byers@mail.mil smime.p7s Description: S/MIME cryptographic signature
RE: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0
Yes, I agree. Unfortunately, our schedule may not allow for that at this time so we will probably have to put off migrating for now. Thanks to you and Bryan for your replies. Thank you, Steven K. Byers DXC Technology - Contractor Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC) Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC 1681 Nelson Street, Fort Detrick, MD 21702 (443) 538-7575 | (410) 872-4923 Email: steven.k.byers@mail.mil -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Tuesday, December 26, 2017 1:56 PM To: dev@nifi.apache.org Subject: Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0 Steven To Bryan's point if it will be best if you simply copy the code/base classes over to your Nar rather than extending from existing concrete implemented processors. Thanks On Tue, Dec 26, 2017 at 1:50 PM, Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US) <steven.k.byers@mail.mil> wrote: > Bryan, > > We have extended a few processors from standard processors. For > example, one of the processors we are extending is: > org.apache.nifi.processors.standard.JmsConsumer > > In another processor we are importing > org.apache.nifi.processors.standard.util.JdbcCommon. > > I tried making the dependency on standard processors optional > (optional>true) which excludes the standard processors but then > NiFi won't start because those custom processors that depend on it cannot be > instantiated. > > > > Thank you, > > Steven K. Byers > DXC Technology - Contractor > Software Developer - Joint Medical Logistics Functional Development > Center (JMLFDC) Defense Health Agency (DHA)/ Health Information > Technology (HIT) Directorate/ Solution Delivery Division > (SDD)/Clinical Support Branch/JMLFDC > 1681 Nelson Street, Fort Detrick, MD 21702 > (443) 538-7575 | (410) 872-4923 > Email: steven.k.byers@mail.mil > > > > -Original Message- > From: Bryan Bende [mailto:bbe...@gmail.com] > Sent: Tuesday, December 26, 2017 1:34 PM > To: dev@nifi.apache.org > Subject: Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0 > > Can you give some more details about what the dependency is for? > > If it is some utility code that exists in standard processors then we > should be looking to move that to other reusable modules so that you > can depend on it without depending on the processors. > > If it is because you extended a processor from standard processors, > you would probably want to just copy the processor code into your own > NAR and modify/extend it. > > On Tue, Dec 26, 2017 at 1:26 PM Byers, Steven K (Steve) CTR USARMY > MEDCOM JMLFDC (US) <steven.k.byers@mail.mil> wrote: > >> Thanks for the reply, Bryan, >> >> Yes, two of our custom processors have a dependency on the standard >> processors. The dependency cannot be removed or those processors >> will not compile. >> >> Thank you, >> >> Steven K. Byers >> DXC Technology - Contractor >> Software Developer - Joint Medical Logistics Functional Development >> Center >> (JMLFDC) >> Defense Health Agency (DHA)/ Health Information Technology (HIT) >> Directorate/ Solution Delivery Division (SDD)/Clinical Support >> Branch/JMLFDC >> 1681 Nelson Street, Fort Detrick, MD 21702 >> (443) 538-7575 | (410) 872-4923 >> Email: steven.k.byers@mail.mil >> >> >> >> -Original Message- >> From: Bryan Bende [mailto:bbe...@gmail.com] >> Sent: Tuesday, December 26, 2017 11:25 AM >> To: dev@nifi.apache.org >> Subject: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0 >> >> Hello, >> >> This means your custom NAR is bundling the standard processors jar >> and as a result they are getting discovered twice, once from your NAR >> and once from the standard NAR. >> >> You’ll have to look at your maven dependencies for your custom NARs >> and figure out why the dependency on standard processors exists and remove >> it. >> >> Thanks, >> >> Bryan >> >> >> On Tue, Dec 26, 2017 at 11:09 AM Byers, Steven K (Steve) CTR USARMY >> MEDCOM JMLFDC (US) <steven.k.byers@mail.mil> wrote: >> >> > Hi devs, >> > >> > I'm looking into moving from NiFi 1.1.2 to 1.4.0. We have several >> > custom processors and services. Everything is compiling without >> > any problems but when I put the services into the 1.4.0 instance, >> > NiFi shows in the list of processors a
RE: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0
Bryan, We have extended a few processors from standard processors. For example, one of the processors we are extending is: org.apache.nifi.processors.standard.JmsConsumer In another processor we are importing org.apache.nifi.processors.standard.util.JdbcCommon. I tried making the dependency on standard processors optional (optional>true) which excludes the standard processors but then NiFi won't start because those custom processors that depend on it cannot be instantiated. Thank you, Steven K. Byers DXC Technology - Contractor Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC) Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC 1681 Nelson Street, Fort Detrick, MD 21702 (443) 538-7575 | (410) 872-4923 Email: steven.k.byers@mail.mil -Original Message- From: Bryan Bende [mailto:bbe...@gmail.com] Sent: Tuesday, December 26, 2017 1:34 PM To: dev@nifi.apache.org Subject: Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0 Can you give some more details about what the dependency is for? If it is some utility code that exists in standard processors then we should be looking to move that to other reusable modules so that you can depend on it without depending on the processors. If it is because you extended a processor from standard processors, you would probably want to just copy the processor code into your own NAR and modify/extend it. On Tue, Dec 26, 2017 at 1:26 PM Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US) <steven.k.byers@mail.mil> wrote: > Thanks for the reply, Bryan, > > Yes, two of our custom processors have a dependency on the standard > processors. The dependency cannot be removed or those processors will not > compile. > > Thank you, > > Steven K. Byers > DXC Technology - Contractor > Software Developer - Joint Medical Logistics Functional Development Center > (JMLFDC) > Defense Health Agency (DHA)/ Health Information Technology (HIT) > Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC > 1681 Nelson Street, Fort Detrick, MD 21702 > (443) 538-7575 | (410) 872-4923 > Email: steven.k.byers@mail.mil > > > > -Original Message- > From: Bryan Bende [mailto:bbe...@gmail.com] > Sent: Tuesday, December 26, 2017 11:25 AM > To: dev@nifi.apache.org > Subject: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0 > > Hello, > > This means your custom NAR is bundling the standard processors jar and as > a result they are getting discovered twice, once from your NAR and once > from the standard NAR. > > You’ll have to look at your maven dependencies for your custom NARs and > figure out why the dependency on standard processors exists and remove it. > > Thanks, > > Bryan > > > On Tue, Dec 26, 2017 at 11:09 AM Byers, Steven K (Steve) CTR USARMY MEDCOM > JMLFDC (US) <steven.k.byers@mail.mil> wrote: > > > Hi devs, > > > > I'm looking into moving from NiFi 1.1.2 to 1.4.0. We have several > > custom processors and services. Everything is compiling without any > > problems but when I put the services into the 1.4.0 instance, NiFi > > shows in the list of processors a 1.1.2 and 1.4.0 version of all > > processors including the stock NiFi processors. If I just load our > > custom processors that do not require a service, NiFi is fine and the > > processor list looks like it should and the custom processors work > > fine. It seems to be something related to the custom services. Is > > there some documentation or any guidance someone can provide to assist > > with what I am doing? > > > > Thank you, > > > > Steven K. Byers > > DXC Technology - Contractor > > Software Developer - Joint Medical Logistics Functional Development > > Center > > (JMLFDC) > > Defense Health Agency (DHA)/ Health Information Technology (HIT) > > Directorate/ Solution Delivery Division (SDD)/Clinical Support > > Branch/JMLFDC > > 1681 Nelson Street, Fort Detrick, MD 21702 > > (443) 538-7575 | (410) 872-4923 > > Email: steven.k.byers@mail.mil > > > > > > > > -- > Sent from Gmail Mobile > -- Sent from Gmail Mobile smime.p7s Description: S/MIME cryptographic signature
RE: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0
Thanks for the reply, Bryan, Yes, two of our custom processors have a dependency on the standard processors. The dependency cannot be removed or those processors will not compile. Thank you, Steven K. Byers DXC Technology - Contractor Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC) Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC 1681 Nelson Street, Fort Detrick, MD 21702 (443) 538-7575 | (410) 872-4923 Email: steven.k.byers@mail.mil -Original Message- From: Bryan Bende [mailto:bbe...@gmail.com] Sent: Tuesday, December 26, 2017 11:25 AM To: dev@nifi.apache.org Subject: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0 Hello, This means your custom NAR is bundling the standard processors jar and as a result they are getting discovered twice, once from your NAR and once from the standard NAR. You’ll have to look at your maven dependencies for your custom NARs and figure out why the dependency on standard processors exists and remove it. Thanks, Bryan On Tue, Dec 26, 2017 at 11:09 AM Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US) <steven.k.byers@mail.mil> wrote: > Hi devs, > > I'm looking into moving from NiFi 1.1.2 to 1.4.0. We have several > custom processors and services. Everything is compiling without any > problems but when I put the services into the 1.4.0 instance, NiFi > shows in the list of processors a 1.1.2 and 1.4.0 version of all > processors including the stock NiFi processors. If I just load our > custom processors that do not require a service, NiFi is fine and the > processor list looks like it should and the custom processors work > fine. It seems to be something related to the custom services. Is > there some documentation or any guidance someone can provide to assist > with what I am doing? > > Thank you, > > Steven K. Byers > DXC Technology - Contractor > Software Developer - Joint Medical Logistics Functional Development > Center > (JMLFDC) > Defense Health Agency (DHA)/ Health Information Technology (HIT) > Directorate/ Solution Delivery Division (SDD)/Clinical Support > Branch/JMLFDC > 1681 Nelson Street, Fort Detrick, MD 21702 > (443) 538-7575 | (410) 872-4923 > Email: steven.k.byers@mail.mil > > > > -- Sent from Gmail Mobile smime.p7s Description: S/MIME cryptographic signature
Moving from version 1.1.2 to 1.4.0
Hi devs, I'm looking into moving from NiFi 1.1.2 to 1.4.0. We have several custom processors and services. Everything is compiling without any problems but when I put the services into the 1.4.0 instance, NiFi shows in the list of processors a 1.1.2 and 1.4.0 version of all processors including the stock NiFi processors. If I just load our custom processors that do not require a service, NiFi is fine and the processor list looks like it should and the custom processors work fine. It seems to be something related to the custom services. Is there some documentation or any guidance someone can provide to assist with what I am doing? Thank you, Steven K. Byers DXC Technology - Contractor Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC) Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC 1681 Nelson Street, Fort Detrick, MD 21702 (443) 538-7575 | (410) 872-4923 Email: steven.k.byers@mail.mil smime.p7s Description: S/MIME cryptographic signature
RE: [Non-DoD Source] Re: NiFi Workflow deployment
Andy, Thank you for your reply. I have come up with a scheme which should meet our needs. It involves using a key/value approach similar to the idea behind the Variable Registry but is file based. Key/value pairs are stored in JSON files for each template and I have a Python program which matches the keys in the templates and substitutes the value. This allows me to target configurations for each server we will be deploying to. Thank you, Steven K. Byers DXC Technology - Contractor Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC) Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC 1681 Nelson Street, Fort Detrick, MD 21702 (443) 538-7575 | (410) 872-4923 Email: steven.k.byers@mail.mil From: Andy LoPresto [mailto:alopre...@apache.org] Sent: Tuesday, August 15, 2017 3:02 PM To: dev@nifi.apache.org Subject: [Non-DoD Source] Re: NiFi Workflow deployment All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. _ Hi Steven, This is a common problem which we are actively working to improve. Currently, the “Variable Registry” [1] feature is a stop-gap solution in conjunction with publishing and loading templates [2] into the various environments. The new Flow Registry [3] feature will make this much easier and automate the process for you. You can read about the feature roadmap and if you search these mailing lists [4][5] for “SDLC” or “flow registry” you can find many other users discussing their solutions and tips for similar experiences. Hope this helps. [1] Caution-https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry < Caution-https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry > [2] Caution-https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates < Caution-https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates > [3] Caution-https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows < Caution-https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows > [4] Caution-https://lists.apache.org/list.html?dev@nifi.apache.org < Caution-https://lists.apache.org/list.html?dev@nifi.apache.org > [5] Caution-https://lists.apache.org/list.html?us...@nifi.apache.org < Caution-https://lists.apache.org/list.html?us...@nifi.apache.org > Andy LoPresto alopre...@apache.org < Caution-mailto:alopre...@apache.org > alopresto.apa...@gmail.com < Caution-mailto:alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Aug 15, 2017, at 2:50 PM, Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US) <steven.k.byers@mail.mil < Caution-mailto:steven.k.byers@mail.mil > > wrote: Hi. Can someone describe the best way to maintain, configure and deploy workflows to various NiFi instances? The project I work on has three DEV and three TEST instances. Our workflows have host-specific values and flow-specific values so we want to be able to automate configuring the flows and deploying them to the various servers. NiFi out of the box doesn't seem to support doing what we need to do. We are, currently, using NiFi 1.1.2. Any suggestions would be greatly appreciated! Thank you, Steven K. Byers smime.p7s Description: S/MIME cryptographic signature
NiFi Workflow deployment
Hi. Can someone describe the best way to maintain, configure and deploy workflows to various NiFi instances? The project I work on has three DEV and three TEST instances. Our workflows have host-specific values and flow-specific values so we want to be able to automate configuring the flows and deploying them to the various servers. NiFi out of the box doesn't seem to support doing what we need to do. We are, currently, using NiFi 1.1.2. Any suggestions would be greatly appreciated! Thank you, Steven K. Byers smime.p7s Description: S/MIME cryptographic signature
Communication between flows
Is there a mechanism or technique for communicating the results of a flow file to its "sister" flow files? Here is a high-level description of what I am doing: Input to my flow is a JSON array of documents that get split (SplitJson) into individual documents and each document becomes a distinct flow file. Each document (flow file) gets validated against a JSON schema (ValidateJson) then gets updated into a Mongo collection (PutMongoUpdate). At the end of all this, I want to do some post processing but only if all documents processed successfully. When a failure occurs (either in the validation or the Mongo update) is there a way to communicate that to the success branch of the flow process so a decision can be made about whether to proceed to post processing or not. I am using NiFi 1.1.2 Thank you for any guidance you can offer, Steve
