Re: RPG question
We will try your recommendations. thanks RK On Wed, Oct 3, 2018 at 10:47 AM Bryan Bende wrote: > Have you tried with smaller batch settings? > > I think 2000 flow files and 1000MB would be a bit high. Might try > something like 100 flow files and 10MB. > > Also, if you concurrent tasks set to 7 on the RPG side, then you will > likely want to increase the concurrent tasks on Input Port on cluster > 2 where the data is being received. > On Tue, Oct 2, 2018 at 5:47 PM roozbeh karimi > wrote: > > > > The only items that we set are: > > > > Remote Process Groups: > > URLs ... > > Transport Protocol: HTTP > >Communication Timeout: 60 sec > >Yield Duration: 2 sec > > > > Input Ports > > Concurrent Tasks: 7 > > Batch Settings > >Count: 2000 > >Size: 1000MB > >Duration: not set > > > > On Tue, Oct 2, 2018 at 3:47 PM Bryan Bende wrote: > > > > > It looks like there may have been images attached that didn't make it > > > through. > > > > > > I'm not sure why S2S would be slow, it is made to handle very high > > > throughput. > > > > > > What I meant by #4 was... did you perform any specific configuration > of the > > > RPG on cluster 1 or the input port on cluster 2, such as changing batch > > > settings or modifying concurrent tasks, or did you use all default > values? > > > > > > > > > On Tue, Oct 2, 2018 at 2:01 PM roozbeh karimi > > > wrote: > > > > > > > Hello Bryan > > > > 1) The communication is site to site between two separate > > > > instances/clusters of NiFi. > > > > 2) We have tried S2S in both raw and http modes, and neither made a > > > > difference. > > > > 3) The data flows but very slow, and after a while the back pressure > > > > reaches the threshold and we start losing data. > > > > 4) We have set up the RPG input port. > > > > 5) If we replace RPG with a posthttp processor on cluster 1 and a > > > listener > > > > processor on cluster 2 (where rpg nodes were), we don't see any back > > > > pressure buildup on cluster 1 and the data flows between the two > > > instances > > > > very fast. > > > > > > > > > > > > [image: image.png] > > > > [image: image.png] > > > > > > > > On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende > wrote: > > > > > > > >> Hello, > > > >> > > > >> Can you describe the setup a little bit more? > > > >> > > > >> Is this site-to-site between two separate NiFi instances, or is it > > > >> within the same cluster for purposes of redistribution? > > > >> > > > >> If it's separate instances then what type of instances, clustered or > > > >> standalone? > > > >> > > > >> Have you made any changes to the configuration of the RPG, the port > > > >> settings within the RPG, or the Input Port on the remote side? > > > >> > > > >> Is some data being transferred through the RPG and it is just slow, > or > > > >> is no data making it through? > > > >> > > > >> -Bryan > > > >> > > > >> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi < > karimi97...@gmail.com> > > > >> wrote: > > > >> > > > > >> > Hello All, > > > >> > > > > >> > Are there any best practices and guides for setting up RPG? With > my > > > set > > > >> up, > > > >> > I am seeing a large volume of back pressure is being built up and > > > hardly > > > >> > any data is being transmitted from the source NiFi node to the > nodes > > > in > > > >> > RPG. If I replace RPG with PostHttp on the source node and a > listener > > > >> > processor on the destination node, no back pressure is built and > the > > > >> data > > > >> > is transmitted smoothly between the source and destination nodes. > > > >> > > > > >> > I'd appreciate any feedback on to go about tuning my RPG set up. > > > >> > > > > >> > Thanks > > > >> > RK > > > >> > > > > > > > >
Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?
Mike, Processors in particularly are among the toughest at this point. We have very very little headroom on dependency size for the full build size that we upload to ASF infra and mirrors. That and the license review work involved in each... We should really create a way to publish processors on more frequent, irregular intervals where the release work and size/etc.. are far less problematic. We have another discuss thread on that so I'll leave it there for discussion. I do share your view that this processor (among several others outstanding) would be really useful but i am definitely thinking we should keep release pace up. Release more often...release processors separately, etc.. Thanks Joe On Wed, Oct 3, 2018 at 9:30 PM Mike Thomsen wrote: > > I would like to see the Neo4J work that mans2singh is doing get included. > Being able to at least partially support a popular graph database would be > a nice feather in our cap. > > On Wed, Oct 3, 2018 at 5:12 PM Andy LoPresto wrote: > > > I am currently working on a TLS Toolkit refactor (NIFI-5462 & NIFI-5485) > > and HashAttribute updates (NIFI-5582). I believe there are a couple upgrade > > PRs open, and I would really like to see NIFI-5402 (no .tar.gz in the > > build) tackled for this release as well. > > > > > > Andy LoPresto > > alopre...@apache.org > > *alopresto.apa...@gmail.com * > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Oct 3, 2018, at 11:16 AM, Joe Witt wrote: > > > > Jeff - thanks again for volunteering. I just went through the open > > items tagged to 1.8.0 to try and shake some loose, close down ones > > that appear to be done but forgotten, and initiate resolution on one > > that is in a dangling state. > > > > Another very nice release shaping up here. All the work around load > > balancing and node offloading is awesome. > > > > Thanks > > On Wed, Oct 3, 2018 at 2:06 PM Jeff wrote: > > > > > > It looks like we're getting close to a point where we could release NiFi > > 1.8.0The release tracking page for version 1.8.0 [1] shows 3 "in progress" > > and 9 "to do" issues. In addition to what has been tagged with a fix > > version of 1.8.0, it looks like NIFI-5516 and NIFI-5585 are close to > > completion. > > > > Are there other JIRAs that the community considers necessary for the > > release that are close to being resolved, with the goal of getting a > > release candidate out in the next couple of weeks? > > > > I'm happy to perform the release manager duties! > > > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482 > > > > > >
Re: Is it possible to count records in a MongoDb collection using the GetMongo processor?
Right now the only way to do it would be to use RunMongoAggregation to do the aggregation query that counts. On Wed, Oct 3, 2018 at 2:56 PM Byers, Steven K (Steve) CTR USARMY MEDCOM JMLFDC (US) wrote: > > Hi everyone, > > Is there a way to use the GetMongo processor to count the number of > records in > a collection that has a certain value for a field in the collection? > > For example, how would I count the number of documents in the Names > collection > where firstName = Steve? > > Thank you, > > Steven K. Byers > Perspecta Inc. Contractor > Software Developer - Joint Medical Logistics Functional Development Center > (JMLFDC) > Defense Health Agency (DHA)/ Health Information Technology (HIT) > Directorate/ > Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC > 1681 Nelson Street, Fort Detrick, MD 21702 > (443) 538-7575 | (410) 872-4923 > Email: steven.k.byers@mail.mil > >
Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?
I would like to see the Neo4J work that mans2singh is doing get included. Being able to at least partially support a popular graph database would be a nice feather in our cap. On Wed, Oct 3, 2018 at 5:12 PM Andy LoPresto wrote: > I am currently working on a TLS Toolkit refactor (NIFI-5462 & NIFI-5485) > and HashAttribute updates (NIFI-5582). I believe there are a couple upgrade > PRs open, and I would really like to see NIFI-5402 (no .tar.gz in the > build) tackled for this release as well. > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Oct 3, 2018, at 11:16 AM, Joe Witt wrote: > > Jeff - thanks again for volunteering. I just went through the open > items tagged to 1.8.0 to try and shake some loose, close down ones > that appear to be done but forgotten, and initiate resolution on one > that is in a dangling state. > > Another very nice release shaping up here. All the work around load > balancing and node offloading is awesome. > > Thanks > On Wed, Oct 3, 2018 at 2:06 PM Jeff wrote: > > > It looks like we're getting close to a point where we could release NiFi > 1.8.0The release tracking page for version 1.8.0 [1] shows 3 "in progress" > and 9 "to do" issues. In addition to what has been tagged with a fix > version of 1.8.0, it looks like NIFI-5516 and NIFI-5585 are close to > completion. > > Are there other JIRAs that the community considers necessary for the > release that are close to being resolved, with the goal of getting a > release candidate out in the next couple of weeks? > > I'm happy to perform the release manager duties! > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482 > > >
Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?
I am currently working on a TLS Toolkit refactor (NIFI-5462 & NIFI-5485) and HashAttribute updates (NIFI-5582). I believe there are a couple upgrade PRs open, and I would really like to see NIFI-5402 (no .tar.gz in the build) tackled for this release as well. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 3, 2018, at 11:16 AM, Joe Witt wrote: > > Jeff - thanks again for volunteering. I just went through the open > items tagged to 1.8.0 to try and shake some loose, close down ones > that appear to be done but forgotten, and initiate resolution on one > that is in a dangling state. > > Another very nice release shaping up here. All the work around load > balancing and node offloading is awesome. > > Thanks > On Wed, Oct 3, 2018 at 2:06 PM Jeff wrote: >> >> It looks like we're getting close to a point where we could release NiFi >> 1.8.0The release tracking page for version 1.8.0 [1] shows 3 "in progress" >> and 9 "to do" issues. In addition to what has been tagged with a fix >> version of 1.8.0, it looks like NIFI-5516 and NIFI-5585 are close to >> completion. >> >> Are there other JIRAs that the community considers necessary for the >> release that are close to being resolved, with the goal of getting a >> release candidate out in the next couple of weeks? >> >> I'm happy to perform the release manager duties! >> >> [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482 signature.asc Description: Message signed with OpenPGP using GPGMail
RE: [EXT] XKCD use case for NiFi
I don't think NiFi will run on his phone, MiNiFi maybe? -Original Message- From: Joe Gresock [mailto:jgres...@gmail.com] Sent: Wednesday, October 3, 2018 12:08 PM To: dev@nifi.apache.org Subject: [EXT] XKCD use case for NiFi https://xkcd.com/2054/ He needs NiFi!
Is it possible to count records in a MongoDb collection using the GetMongo processor?
Hi everyone, Is there a way to use the GetMongo processor to count the number of records in a collection that has a certain value for a field in the collection? For example, how would I count the number of documents in the Names collection where firstName = Steve? Thank you, Steven K. Byers Perspecta Inc. Contractor Software Developer - Joint Medical Logistics Functional Development Center (JMLFDC) Defense Health Agency (DHA)/ Health Information Technology (HIT) Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC 1681 Nelson Street, Fort Detrick, MD 21702 (443) 538-7575 | (410) 872-4923 Email: steven.k.byers@mail.mil smime.p7s Description: S/MIME cryptographic signature
Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?
Jeff - thanks again for volunteering. I just went through the open items tagged to 1.8.0 to try and shake some loose, close down ones that appear to be done but forgotten, and initiate resolution on one that is in a dangling state. Another very nice release shaping up here. All the work around load balancing and node offloading is awesome. Thanks On Wed, Oct 3, 2018 at 2:06 PM Jeff wrote: > > It looks like we're getting close to a point where we could release NiFi > 1.8.0The release tracking page for version 1.8.0 [1] shows 3 "in progress" > and 9 "to do" issues. In addition to what has been tagged with a fix > version of 1.8.0, it looks like NIFI-5516 and NIFI-5585 are close to > completion. > > Are there other JIRAs that the community considers necessary for the > release that are close to being resolved, with the goal of getting a > release candidate out in the next couple of weeks? > > I'm happy to perform the release manager duties! > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482
Re: XKCD use case for NiFi
nice find. In many ways this other one is a great inspiration for NiFi as well https://xkcd.com/927/ On Wed, Oct 3, 2018 at 2:08 PM Joe Gresock wrote: > > https://xkcd.com/2054/ > > He needs NiFi!
XKCD use case for NiFi
https://xkcd.com/2054/ He needs NiFi!
[DISCUSS] Closing in on a release of NiFi 1.8.0?
It looks like we're getting close to a point where we could release NiFi 1.8.0The release tracking page for version 1.8.0 [1] shows 3 "in progress" and 9 "to do" issues. In addition to what has been tagged with a fix version of 1.8.0, it looks like NIFI-5516 and NIFI-5585 are close to completion. Are there other JIRAs that the community considers necessary for the release that are close to being resolved, with the goal of getting a release candidate out in the next couple of weeks? I'm happy to perform the release manager duties! [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482
Re: Nifi config issues
Ryan, As Joe mentioned, we’ll need to see the full stacktrace, but the problem is likely at the bottom of that error output. The Flow Controller is a critical piece of NiFi’s architecture, and errors with loading it are usually because of a missing or misconfigured value. I’m guessing that your authorizers.xml or login-identity-providers.xml file is misconfigured, and that causes a chain of errors in the bean loading that bubbles up to the FC error you’re seeing. The most specific error (usually at the root of the stacktrace) is the one you want to address. Hope this helps, and we’ll be happy to look at the stacktrace when you share it. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 3, 2018, at 9:19 AM, Joe Witt wrote: > > ryan > > we'll likely need to see the full stack trace and a good bit more > context on the configuration. > > nifi on rel7 is an extremely common deployment scenario so that part > is good/easy and now it is likely a matter of config. > > Thanks > On Wed, Oct 3, 2018 at 12:18 PM McGary, Ryan - US > wrote: >> >> Hello, >> >>My company is in the process of standing up a Test >> environment, and we're installing Nifi on one of our Rhel7 servers. We are >> running into some issues, however. When Nifi is configured for http, it >> runs just fine, but as soon as we change the config to https it starts >> putting out errors. We think we have the Key/truststore configured >> correctly, but after we start the nifi service we continuously get the >> "org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller." >> error. Googling brought up many different responses, but none of them >> seemed to fix our problem. Just wondering if the developers could help, or >> perhaps point me in the right direction? Any help would be much appreciated, >> thanks. >> >> v/r >> >> Ryan McGary signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Nifi config issues
ryan we'll likely need to see the full stack trace and a good bit more context on the configuration. nifi on rel7 is an extremely common deployment scenario so that part is good/easy and now it is likely a matter of config. Thanks On Wed, Oct 3, 2018 at 12:18 PM McGary, Ryan - US wrote: > > Hello, > > My company is in the process of standing up a Test > environment, and we're installing Nifi on one of our Rhel7 servers. We are > running into some issues, however. When Nifi is configured for http, it runs > just fine, but as soon as we change the config to https it starts putting out > errors. We think we have the Key/truststore configured correctly, but after > we start the nifi service we continuously get the > "org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller." > error. Googling brought up many different responses, but none of them seemed > to fix our problem. Just wondering if the developers could help, or perhaps > point me in the right direction? Any help would be much appreciated, thanks. > > v/r > > Ryan McGary
Nifi config issues
Hello, My company is in the process of standing up a Test environment, and we're installing Nifi on one of our Rhel7 servers. We are running into some issues, however. When Nifi is configured for http, it runs just fine, but as soon as we change the config to https it starts putting out errors. We think we have the Key/truststore configured correctly, but after we start the nifi service we continuously get the "org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller." error. Googling brought up many different responses, but none of them seemed to fix our problem. Just wondering if the developers could help, or perhaps point me in the right direction? Any help would be much appreciated, thanks. v/r Ryan McGary
Re: RPG question
Have you tried with smaller batch settings? I think 2000 flow files and 1000MB would be a bit high. Might try something like 100 flow files and 10MB. Also, if you concurrent tasks set to 7 on the RPG side, then you will likely want to increase the concurrent tasks on Input Port on cluster 2 where the data is being received. On Tue, Oct 2, 2018 at 5:47 PM roozbeh karimi wrote: > > The only items that we set are: > > Remote Process Groups: > URLs ... > Transport Protocol: HTTP >Communication Timeout: 60 sec >Yield Duration: 2 sec > > Input Ports > Concurrent Tasks: 7 > Batch Settings >Count: 2000 >Size: 1000MB >Duration: not set > > On Tue, Oct 2, 2018 at 3:47 PM Bryan Bende wrote: > > > It looks like there may have been images attached that didn't make it > > through. > > > > I'm not sure why S2S would be slow, it is made to handle very high > > throughput. > > > > What I meant by #4 was... did you perform any specific configuration of the > > RPG on cluster 1 or the input port on cluster 2, such as changing batch > > settings or modifying concurrent tasks, or did you use all default values? > > > > > > On Tue, Oct 2, 2018 at 2:01 PM roozbeh karimi > > wrote: > > > > > Hello Bryan > > > 1) The communication is site to site between two separate > > > instances/clusters of NiFi. > > > 2) We have tried S2S in both raw and http modes, and neither made a > > > difference. > > > 3) The data flows but very slow, and after a while the back pressure > > > reaches the threshold and we start losing data. > > > 4) We have set up the RPG input port. > > > 5) If we replace RPG with a posthttp processor on cluster 1 and a > > listener > > > processor on cluster 2 (where rpg nodes were), we don't see any back > > > pressure buildup on cluster 1 and the data flows between the two > > instances > > > very fast. > > > > > > > > > [image: image.png] > > > [image: image.png] > > > > > > On Tue, Oct 2, 2018 at 11:55 AM Bryan Bende wrote: > > > > > >> Hello, > > >> > > >> Can you describe the setup a little bit more? > > >> > > >> Is this site-to-site between two separate NiFi instances, or is it > > >> within the same cluster for purposes of redistribution? > > >> > > >> If it's separate instances then what type of instances, clustered or > > >> standalone? > > >> > > >> Have you made any changes to the configuration of the RPG, the port > > >> settings within the RPG, or the Input Port on the remote side? > > >> > > >> Is some data being transferred through the RPG and it is just slow, or > > >> is no data making it through? > > >> > > >> -Bryan > > >> > > >> On Mon, Oct 1, 2018 at 11:53 PM roozbeh karimi > > >> wrote: > > >> > > > >> > Hello All, > > >> > > > >> > Are there any best practices and guides for setting up RPG? With my > > set > > >> up, > > >> > I am seeing a large volume of back pressure is being built up and > > hardly > > >> > any data is being transmitted from the source NiFi node to the nodes > > in > > >> > RPG. If I replace RPG with PostHttp on the source node and a listener > > >> > processor on the destination node, no back pressure is built and the > > >> data > > >> > is transmitted smoothly between the source and destination nodes. > > >> > > > >> > I'd appreciate any feedback on to go about tuning my RPG set up. > > >> > > > >> > Thanks > > >> > RK > > >> > > > > >
RE: Zookeeper - help!
Okay, I have got this working now, albeit with only a single ZK instance (at this stage). The missing piece of the puzzle that wasn’t in the guides from Pierre was that cluster servers’ certificates need to be installed in each server’s keystore, and all the cluster server DNs need to be added as Initial User Identities in authorizers.xml. Thanks again for all the assistance. Sent from Mail for Windows 10 From: Nathan Gough Sent: Wednesday, 3 October 2018 7:27 AM To: dev@nifi.apache.org Subject: Re: Zookeeper - help! I think you are correct on that, I assumed it was a range of some kind but it looks like it's not: http://zookeeper.apache.org/doc/r3.4.3/zookeeperStarted.html#sc_RunningReplicatedZooKeeper On 10/2/18, 5:17 PM, "Phil H" wrote: The second port in the zookeeper server config has been a mystery to me. I thought it was a second port used for elections, not the upper bound in a range. Why is the range so large? Sent from Mail for Windows 10 From: Nathan Gough Sent: Wednesday, 3 October 2018 1:26 AM To: dev@nifi.apache.org Subject: Re: Zookeeper - help! Check your configs on nifi2. I don't believe that NiFi is starting two instances of Zookeeper but the ports configured are unintentionally configured to overlap ie. Ports used twice in different configs where they should be different. It may be that your zookeeper.properties has: clientPort=2180 ... server.1=nifi1.com:2180:3888 server.2=nifi2.com:2180:3888 where it should be: clientPort=2180 ... server.1=nifi1.com:2888:3888 server.2=nifi2.com:2888:3888 noticing that the server.1 and server.2 ranges don't overlap with the client port. Not sure if this helps, but the following is the relevant config that I have for my NiFi cluster nodes that run on the SAME machine where nifi1.com and nifi2.com are configured in /etc/hosts: nifi1/conf zookeeper.properties - clientPort=2180 - server.1=nifi1.com:2888:3888 - server.2=nifi2.com:2888:3888 nifi.properties - nifi.remote.input.host=nifi1.com - nifi.remote.input.socket.port=10440 - nifi.web.http.host=nifi1.com - nifi.web.http.port=9550 - nifi.cluster.node.address=nifi1.com - nifi.cluster.node.protocol.port=11440 nifi1/state/zookeeper /myid (file contents = "1") /state-management.xml (no changes required) /version-2/ nifi2/conf zookeeper.properties - clientPort=2181 - server.1=nifi1.com:2888:3888 - server.2=nifi2.com:2888:3888 nifi.properties - nifi.remote.input.host=nifi2.com - nifi.remote.input.socket.port=10441 - nifi.web.http.host=nifi2.com - nifi.web.http.port=9551 - nifi.cluster.node.address=nifi2.com - nifi.cluster.node.protocol.port=11441 nifi2/state/zookeeper /myid (file contents = "2") /state-management.xml (no changes required) /version-2/ Nathan On 10/2/18, 2:07 AM, "Phil H" wrote: Hi Andy, Thanks for the additional info. I think I saw a link to that while searching but was wary since it was such an old version. I have two VMs (nifi1, and nifi2) both running NiFi with identical configs, and trying to use the inbuilt ZK to cluster them. If I only mention a single machine within the config (eg: if nifi1 doesn’t refer to nifi2, or visa versa) I don’t get any start up errors. Phil From: Andy LoPresto Sent: Tuesday, 2 October 2018 1:00 PM To: dev@nifi.apache.org Subject: Re: Zookeeper - help! Hi Phil, Nathan’s advice is correct but I think he was assuming all other configurations are correct as well. Are you trying to run both NiFi nodes and ZK instances on the same machine? In that case you will have to ensure that the ports in use are different for each service so they don’t conflict. Setting them all to the same value only works if each service is running on an independent physical machine, virtual machine, or container. I find Pierre’s guide [1] to be a helpful step-by-step instruction list as well as a good explanation of how the clustering concepts work in practice. When you get that working, and you’re ready to set up a secure cluster, he has a follow-on guide for that as well [2]. Even as someone who has set up many clustered instances of NiFi, I use his guides regularly to ensure I haven’t forgotten a step. They were originally written for versions 1.0.0 and 1.1.0, but the only thing that has changed is the authorizer configuration for the secure instances (you’ll need to put the Initial Admin Identity and Node Identities in two locations in the authorizers.xml file instead of just once). Hopefully this helps you get a working c
