Apache NiFi 1.8.0 RC1 Release Helper Guide
Hello Apache NiFi community, Please find the associated guidance to help those interested in validating/verifying the release so they can vote. # Download latest KEYS file: https://dist.apache.org/repos/dist/dev/nifi/KEYS # Import keys file: gpg --import KEYS # [optional] Clear out local maven artifact repository # Pull down nifi-1.8.0 source release artifacts for review: wget https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip wget https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.asc wget https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.sha256 wget https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.sha512 # Verify the signature gpg --verify nifi-1.8.0-source-release.zip.asc # Verify the hashes (sha256, sha512) match the source and what was provided in the vote email thread shasum -a 256 nifi-1.8.0-source-release.zip shasum -a 512 nifi-1.8.0-source-release.zip # Unzip nifi-1.8.0-source-release.zip # Verify the build works including release audit tool (RAT) checks cd nifi-1.8.0 mvn clean install -Pcontrib-check,include-grpc # Verify the contents contain a good README, NOTICE, and LICENSE. # Verify the git commit ID is correct # Verify the RC was branched off the correct git commit ID # Look at the resulting convenience binary as found in nifi-assembly/target # Make sure the README, NOTICE, and LICENSE are present and correct # Run the resulting convenience binary and make sure it works as expected # Send a response to the vote thread indicating a +1, 0, -1 based on your findings. Thank you for your time and effort to validate the release!
Re: Scaling source processors in nifi horizontally.
> may have to build some kind of tooling on top of it to monitor/provision new processor for newly added schemas etc. Could you elaborate on this part of your use case? On Wed, Oct 17, 2018 at 2:31 PM ashwin konale wrote: > Hi, > > I am experimenting with nifi for one of our usecases with plans of > extending it to various other data routing, ingestion usecases. Right now I > need to ingest data from mysql binlogs to hdfs/GCS. We have around 250 > different schemas and about 3000 tables to read data from. Volume of the > data flow ranges from 500 - 2000 messages per second in different schemas. > > Right now the problem is mysqlCDC processor can run in only one thread. To > overcome this issue I have two options. > > 1. Use primary node execution, so different processors for each of the > schemas. So eventually all processors which reads from mysql will run in > single node, which will be a bottleneck no matter how big my nifi cluster > is. > > 2. Another approach is to use multiple nifi instances to pull data and have > master nifi cluster for ingestion to various sinks. In this approach I will > have to manage all these small nifi instances, and may have to build some > kind of tooling on top of it to monitor/provision new processor for newly > added schemas etc. > > Is there any better way to achieve my usecase with nifi ? Please advice me > on the architechture. > > Looking forward for suggestion. > > - Ashwin >
Re: Compiling custom processor
That is accurate. Generally the intention isn't really to extend processors across NARs. It is better to create a shared module with a base class that both processors can extend from, or utility code they can both call, but in the case that shared module doesn't exist then the main option is to copy over the code you need. On Wed, Oct 17, 2018 at 2:52 PM John McGinn wrote: > > Thanks Bryan. > > That was a mistake on my part. I thought that I could use part of the NiFi > standard processors in my custom processor. I suppose this also means it's > not possible to extend a standard processor class, say Wait, and override a > method, for instance, onTrigger(), to do different work. Instead, as you > state, I should copy the portions I would need to my custom processor. Is > that accurate? > > In the end, as I was looking through how to modify Wait, I think it is bigger > than the need that I have, so I restarted with a new generated processor > project, and just added the cache service, and appropriate relationships, and > simple logic to control flow through my processor. I'll work on getting a > github account and uploading my simple processor. > > Thanks for the assistance > John McGinn > > > On Tue, 10/16/18, Bryan Bende wrote: > > Subject: Re: Compiling custom processor > To: figgie...@sbcglobal.net > Date: Tuesday, October 16, 2018, 4:45 PM > > You shouldn't really try to > depend on code inside other NARs like > standard processors, you'll want to copy > WaitNotifyProtocol into your > own NAR. > On > Tue, Oct 16, 2018 at 4:40 PM John McGinn > wrote: > > > > The > commons-lang3-3.7.jar file is located in the lib/bootstrap > directory, therefore, I figured it could be considered > provided. I've since change it to normal scope, and it > is included in my NAR and that error goes away. So, this > > then leans on the next error I get, which > is no class found for the WaitNotifyProtocol in > nifi.processors.standard. Is there an easy way to figure out > what NiFi class/object I'm using, and what NAR should be > listed as provided? I tried to add > nifi-standard-processors-nar, but that doesn't exist for > maven, nor does it exist in the lib directory. I looked > inside the nifi-standard-nar NAR file, and the > nifi-standard-processors is there, and inside that is > WaitNotifyProtocol, but yet, my maven won't build if I > have nifi-standard-nar listed, either with type of NAR, or > scope of provided. > > > > > On Tue, 10/16/18, Bryan Bende > wrote: > > > > Subject: > Re: Compiling custom processor > > To: dev@nifi.apache.org, > figgie...@sbcglobal.net > > Date: Tuesday, October 16, 2018, 4:01 > PM > > > > Any reason > why the commons-lang3 > > dependency is > provided and not normal > > compile > scope? > > > > Generally > the only cases where a dependency > > > should be provided are > > when a parent > NAR > > provides it (the API scenario > earlier), or when it > > is a jar > directly in lib which means it is on > > > the classpath of > > everything (like > nifi-api > > or slf4j). > > On Tue, Oct 16, 2018 at 3:47 PM John > McGinn > > > > wrote: > > > > > > I have a > > > directory /home/user/nifi-nars, which is listed in the > > nifi.properties file located under the > nifi-assembly > > directory from the > nifi-master code. That's where I > > > redropped my NAR file at. > > > > > > The error was "due to > uncaught > > Exception: > java.lang.NoClassDefFoundError: > > > org/apache/commons/lang3/StringUtils". I added a > > provided dependency of version 3.4 for > commons-lang3 in my > > processors > pom.xml. That worked initially, but with the > > redropping of the NAR file, the error > occurs. The stacktrace > > points to the > session.get() lambda function trying to use > > StringUtils.isBlank(). (Again, this is a > copy and renamed > > class of the Wait > Processor. Trying to turn it into a Yield, > > as it were, hold unless it's free to > move forward.) > > > > > > > > > On Tue, 10/16/18, Bryan Bende > > > wrote: > > > > > > Subject: > > > Re: Compiling custom processor > > > > To: dev@nifi.apache.org > > > Date: Tuesday, October 16, 2018, > 3:34 > > PM > > > > > > > There > > shouldn't be any issue > > > doing > > > that, generally you just copy the > > > > > > new > > > > NAR into the lib directory and > > > restart. > > > > > > > What > > were the specifics of the > class not found > > > error? > > > > > > On > Tue, Oct 16, 2018 at > > > 3:24 PM > John McGinn > > > wrote: > > > > > > > > > And > I'm > > > > > > back. > > > > > > > > > > So, I > had my > > > processor running, > > no errors, so things were good. > > > > > > > > >
Re: Compiling custom processor
Thanks Bryan. That was a mistake on my part. I thought that I could use part of the NiFi standard processors in my custom processor. I suppose this also means it's not possible to extend a standard processor class, say Wait, and override a method, for instance, onTrigger(), to do different work. Instead, as you state, I should copy the portions I would need to my custom processor. Is that accurate? In the end, as I was looking through how to modify Wait, I think it is bigger than the need that I have, so I restarted with a new generated processor project, and just added the cache service, and appropriate relationships, and simple logic to control flow through my processor. I'll work on getting a github account and uploading my simple processor. Thanks for the assistance John McGinn On Tue, 10/16/18, Bryan Bende wrote: Subject: Re: Compiling custom processor To: figgie...@sbcglobal.net Date: Tuesday, October 16, 2018, 4:45 PM You shouldn't really try to depend on code inside other NARs like standard processors, you'll want to copy WaitNotifyProtocol into your own NAR. On Tue, Oct 16, 2018 at 4:40 PM John McGinn wrote: > > The commons-lang3-3.7.jar file is located in the lib/bootstrap directory, therefore, I figured it could be considered provided. I've since change it to normal scope, and it is included in my NAR and that error goes away. So, this > then leans on the next error I get, which is no class found for the WaitNotifyProtocol in nifi.processors.standard. Is there an easy way to figure out what NiFi class/object I'm using, and what NAR should be listed as provided? I tried to add nifi-standard-processors-nar, but that doesn't exist for maven, nor does it exist in the lib directory. I looked inside the nifi-standard-nar NAR file, and the nifi-standard-processors is there, and inside that is WaitNotifyProtocol, but yet, my maven won't build if I have nifi-standard-nar listed, either with type of NAR, or scope of provided. > > On Tue, 10/16/18, Bryan Bende wrote: > > Subject: Re: Compiling custom processor > To: dev@nifi.apache.org, figgie...@sbcglobal.net > Date: Tuesday, October 16, 2018, 4:01 PM > > Any reason why the commons-lang3 > dependency is provided and not normal > compile scope? > > Generally the only cases where a dependency > should be provided are > when a parent NAR > provides it (the API scenario earlier), or when it > is a jar directly in lib which means it is on > the classpath of > everything (like nifi-api > or slf4j). > On Tue, Oct 16, 2018 at 3:47 PM John McGinn > > wrote: > > > > I have a > directory /home/user/nifi-nars, which is listed in the > nifi.properties file located under the nifi-assembly > directory from the nifi-master code. That's where I > redropped my NAR file at. > > > > The error was "due to uncaught > Exception: java.lang.NoClassDefFoundError: > org/apache/commons/lang3/StringUtils". I added a > provided dependency of version 3.4 for commons-lang3 in my > processors pom.xml. That worked initially, but with the > redropping of the NAR file, the error occurs. The stacktrace > points to the session.get() lambda function trying to use > StringUtils.isBlank(). (Again, this is a copy and renamed > class of the Wait Processor. Trying to turn it into a Yield, > as it were, hold unless it's free to move forward.) > > > > > On Tue, 10/16/18, Bryan Bende > wrote: > > > > Subject: > Re: Compiling custom processor > > To: dev@nifi.apache.org > > Date: Tuesday, October 16, 2018, 3:34 > PM > > > > There > shouldn't be any issue > > doing > that, generally you just copy the > > > new > > NAR into the lib directory and > restart. > > > > What > were the specifics of the class not found > > error? > > > > On Tue, Oct 16, 2018 at > > 3:24 PM John McGinn > > wrote: > > > > > > And I'm > > > back. > > > > > > > So, I had my > > processor running, > no errors, so things were good. > > > > > > > I then add 2 if > > statements with logger messages, run the > same mvn command, > > copied the new NAR > on top of the old one, same versionId and > > all, and now I get a class not found > error, that didn't > > occur before I > stopped, redeployed and started. And I > > can't delete the event in the queue > even though source > > and destination > are stopped. > > > > > > Is there problems when redeploying > a > > custom processor NAR on top of > itself? Even if the NiFi > > instance is > stopped? Serialization issues? > > > > > > > Frustratedly > > yours, > > > John > McGinn > > > > > > > > > >
Scaling source processors in nifi horizontally.
Hi, I am experimenting with nifi for one of our usecases with plans of extending it to various other data routing, ingestion usecases. Right now I need to ingest data from mysql binlogs to hdfs/GCS. We have around 250 different schemas and about 3000 tables to read data from. Volume of the data flow ranges from 500 - 2000 messages per second in different schemas. Right now the problem is mysqlCDC processor can run in only one thread. To overcome this issue I have two options. 1. Use primary node execution, so different processors for each of the schemas. So eventually all processors which reads from mysql will run in single node, which will be a bottleneck no matter how big my nifi cluster is. 2. Another approach is to use multiple nifi instances to pull data and have master nifi cluster for ingestion to various sinks. In this approach I will have to manage all these small nifi instances, and may have to build some kind of tooling on top of it to monitor/provision new processor for newly added schemas etc. Is there any better way to achieve my usecase with nifi ? Please advice me on the architechture. Looking forward for suggestion. - Ashwin
Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?
The L issues previously mentioned have been resolved, and I'm starting the release prep for RC1 now. On Tue, Oct 16, 2018 at 4:52 PM Jeff wrote: > A quick update on RC1 for NiFi 1.8.0. There are some L issues that are > currently being worked, so I'd like to hold off on creating the RC until > they are resolved. Sorry for the delay! > > On Tue, Oct 16, 2018 at 3:46 PM Nathan Gough wrote: > >> So, taking a closer look at this PR and doing some further testing I've >> found that upgrading Guava is going to require some more careful >> assessment, as the changes will affect core NiFi functionality around >> clustering. I will hold off on this change until after the release to have >> more time to upgrade and test. >> >> Nathan >> >> On 10/16/18, 10:58 AM, "Nathan Gough" wrote: >> >> Hi Mike, >> >> Sure I can look at fixing up PR-2977 today. >> >> Nathan >> >> On 10/16/18, 6:13 AM, "Mike Thomsen" wrote: >> >> Does 5562 need to be addressed in 1.8? >> >> https://github.com/apache/nifi/pull/2977 >> >> On Mon, Oct 15, 2018 at 6:33 PM Jeff wrote: >> >> > NiFi Devs, >> > >> > The Release page [1] for 1.8.0 now reports that all issues are >> done! I'd >> > like to start the release candidate preparation tomorrow, >> around 1200 EST. >> > >> > Thanks to everyone for all the great work that's been done! >> 196 issues >> > resolved in this version with some great new features! >> > >> > [1] >> https://issues.apache.org/jira/projects/NIFI/versions/12343482 >> > >> > On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna < >> sivaprasanna...@gmail.com> >> > wrote: >> > >> > > Great. Thanks. :) >> > > >> > > - >> > > Sivaprasanna >> > > >> > > On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura < >> ijokaruma...@gmail.com> >> > > wrote: >> > > >> > > > Jeff, Sivasprasanna, >> > > > >> > > > NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged. >> > > > >> > > > Thanks, >> > > > Koji >> > > > On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura < >> ijokaruma...@gmail.com >> > > >> > > > wrote: >> > > > > >> > > > > Thank you for the fix Sivaprasanna, >> > > > > I have Azure account. Reviewing it now. >> > > > > >> > > > > Koji >> > > > > On Sun, Oct 14, 2018 at 11:21 PM Jeff >> wrote: >> > > > > > >> > > > > > Sivaprasanna, >> > > > > > >> > > > > > Thanks for submitting a pull request for that issue! >> Later today >> > or >> > > > > > tomorrow I'll have to check to see if I've already used >> up my >> > > free-tier >> > > > > > access to Azure. If I still have access, I can review >> your PR and >> > > > we'll >> > > > > > get it into 1.8.0. >> > > > > > >> > > > > > On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna < >> > > > sivaprasanna...@gmail.com> >> > > > > > wrote: >> > > > > > >> > > > > > > All - Just found one bug with DeleteAzureBlobStorage >> processor. >> > It >> > > > was >> > > > > > > shared by one user on StackOverflow [1] and I later >> confirmed it. >> > > It >> > > > looks >> > > > > > > to be introduced by NIFI-4199. I have created a Jira >> [2] and made >> > > the >> > > > > > > necessary changes (not huge, just few lines) and >> raised a PR >> > [3]. I >> > > > think, >> > > > > > > if we can spend a little time in getting it reviewed, >> we can mark >> > > it >> > > > for >> > > > > > > 1.8.0. Thoughts? >> > > > > > > >> > > > > > > [1] - >> > > > > > > >> > > > > > > >> > > > >> > > >> > >> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error >> > > > > > > [2] - https://issues.apache.org/jira/browse/NIFI-5698 >> > > > > > > [3] - https://github.com/apache/nifi/pull/3073 >> > > > > > > >> > > > > > > - >> > > > > > > Sivaprasanna >> > > > > > > >> > > > > > > On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen < >> > > mikerthom...@gmail.com >> > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > 4811 should be ready for review now. Rebased and >> cleaned it up >> > > > with a >> > > > > > > full >> > > > > > > > listing of the Spring dependencies. >> > > > > > > > >> > > > > > > > On Fri, Oct 12, 2018 at 11:23 AM Joe Witt < >> joe.w...@gmail.com> >> > > > wrote: >> > > > > > > > >> > > > > > > > > Jeff, >> > > > > > > > > >> > > > > > > > > I think for
Re: Repeatedly adding trusted certificates
Mariliis, This is not usual behavior. Can you please provide a full log, or at least the log output related to this activity? We can help diagnose what behavior is causing this. Do you have any custom processors installed? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 17, 2018, at 15:34, mariliis.haalme wrote: > > Hello! > > I have org.apache.nifi.StdOut log level set as INFO and recently noticed > that every 30 seconds, the same certificates seem to get added as trusted, > over and over again. Is this normal behaviour? What exactly is happening > that produces these logs? > > Thanks, > Mariliis > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Oracle ODBCConnectionPool driver
Ugh, fat-fingered that. It requires you to have an Oracle account to download. It's free, but you can't just grab it without registering. On Wed, Oct 17, 2018 at 7:32 AM Mike Thomsen wrote: > You can download it from Oracle's site, but it re > > On Wed, Oct 17, 2018 at 7:31 AM Mark Bean wrote: > >> Thanks Mike. Is the driver available as a stand-alone download? Or is it >> only included in the DB software, maybe the Instant Client? >> >> >> > On Oct 17, 2018, at 5:43 AM, Mike Thomsen >> wrote: >> > >> > If you get the latest drive from Oracle, it should work with both. They >> > tend to be pretty good on backward compatibility as far as I know. >> > >> >> On Tue, Oct 16, 2018 at 11:06 PM Mark Bean >> wrote: >> >> >> >> What driver is required to connect to an Oracle database using the >> >> DBCPConnectionPool Controller Service? Will the same driver work for >> both >> >> 11g and 12c databases? >> >> >> >> Thanks, >> >> Mark >> >> >> >
Re: Oracle ODBCConnectionPool driver
You can download it from Oracle's site, but it re On Wed, Oct 17, 2018 at 7:31 AM Mark Bean wrote: > Thanks Mike. Is the driver available as a stand-alone download? Or is it > only included in the DB software, maybe the Instant Client? > > > > On Oct 17, 2018, at 5:43 AM, Mike Thomsen > wrote: > > > > If you get the latest drive from Oracle, it should work with both. They > > tend to be pretty good on backward compatibility as far as I know. > > > >> On Tue, Oct 16, 2018 at 11:06 PM Mark Bean > wrote: > >> > >> What driver is required to connect to an Oracle database using the > >> DBCPConnectionPool Controller Service? Will the same driver work for > both > >> 11g and 12c databases? > >> > >> Thanks, > >> Mark > >> >
Re: Oracle ODBCConnectionPool driver
Thanks Mike. Is the driver available as a stand-alone download? Or is it only included in the DB software, maybe the Instant Client? > On Oct 17, 2018, at 5:43 AM, Mike Thomsen wrote: > > If you get the latest drive from Oracle, it should work with both. They > tend to be pretty good on backward compatibility as far as I know. > >> On Tue, Oct 16, 2018 at 11:06 PM Mark Bean wrote: >> >> What driver is required to connect to an Oracle database using the >> DBCPConnectionPool Controller Service? Will the same driver work for both >> 11g and 12c databases? >> >> Thanks, >> Mark >>
Repeatedly adding trusted certificates
Hello! I have org.apache.nifi.StdOut log level set as INFO and recently noticed that every 30 seconds, the same certificates seem to get added as trusted, over and over again. Is this normal behaviour? What exactly is happening that produces these logs? Thanks, Mariliis -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Mongo license change
Looks like Mongo's drivers are still under ASLv2: https://www.mongodb.com/community/licensing The server is shifting to a new proprietary license based on the AGPLv3 that is essentially a more aggressive, more viral version of the AGPLv3. There are concerns in some places I've been reading like /. and HN that it is so aggressive that it might bring in almost any SaaS that is built on top of Mongo. People using the Java driver should be fine going forward, but anyone using native APIs to access it could potentially be in serious trouble.
Re: Oracle ODBCConnectionPool driver
If you get the latest drive from Oracle, it should work with both. They tend to be pretty good on backward compatibility as far as I know. On Tue, Oct 16, 2018 at 11:06 PM Mark Bean wrote: > What driver is required to connect to an Oracle database using the > DBCPConnectionPool Controller Service? Will the same driver work for both > 11g and 12c databases? > > Thanks, > Mark >
