Re: Apache NiFi RC2 timeline
I think it's actually a great time to include NIFI 1461 to improve the RPM with a non-root user and group. With respect to backward compatibility, 0.5.0 will also include a fix for NIFI-1454, and I'm skeptical that anyone was successfully using the RPM output from the Maven build before that fix. As part of NIFI-1454, we have changed the name of the RPM from "nifi-assembly" to just "nifi", which I believe would break any upgrade chain. Suggested text for the migration notes: The RPM distribution has been significantly improved. The name of the RPM is now "nifi". The NiFi RPM installation will now create a "nifi" user and group to own the installed files, while retaining and reusing existing "nifi" users and groups. RPM users should install the RPM as a new package install, rather than an upgrade, and port their configuration and repository data to the new installation. On Mon, Feb 8, 2016 at 8:24 PM, Tony Kurc <trk...@gmail.com> wrote: > Joe, > We'd need some migration notes changes added 1461. However, it does change > the way the rpm works by a little bit. If people are integrating/deploying > with rpms, it might be a breaking change. > > Tony > > On Mon, Feb 8, 2016 at 11:12 PM, Joe Witt <joe.w...@gmail.com> wrote: > > > Tony, > > > > It was what appears to have become NIFI-1492 that I personally felt > > was important to get with state management because it will limit the > > amount of state we return back to the client/UI. I am concerned about > > the amount of state ListFile can generate in some cases and I think > > that needs to be solved but it doesn't need to hold up the release. > > > > Also, what do you think about including 1461? It was requested by > > Andre, implemented by Puspendu, validated by James Wing. I've not had > > a chance to go through it yet but that it has such solid community > > effort behind it is a huge plus. Will take a look at that first thing > > tomorrow and provide feedback. > > > > Thanks > > Joe > > > > > > > > > > > > > > > > On Mon, Feb 8, 2016 at 8:32 PM, Tony Kurc <trk...@gmail.com> wrote: > > > I've gotten a couple out of band "hold off on the RCs", so I won't be > > > cutting RC2 tonight. > > > > > > On Mon, Feb 8, 2016 at 2:27 PM, Tony Kurc <trk...@gmail.com> wrote: > > > > > >> All, > > >> I'm hoping many of you are trying out the RC1 for functionality and > > >> ideally haven't found any issues beyond those already patched. I was > > >> planning on pulling together an RC2 tonight, so please, if you have > > found > > >> some other critical issues, please bring it up on this dev list. > > >> > > >> Tony > > >> > > >
Trouble with RPMs
Is anyone successfully using NiFi RPMs? I am having trouble building RPMs that result in a working NiFi installation on a clean system. I'm not sure if I'm just doing it wrong, or if I should create a ticket and try to fix it. After building an RPM (mvn clean install -Prpm) and installing it (Amazon Linux, Java 1.7.0_91), I ran into the following error attempting to start NiFi : # ./nifi.sh start Java home: /usr/lib/jvm/java NiFi home: /opt/nifi/nifi-0.4.1 Bootstrap Config File: /opt/nifi/nifi-0.4.1/conf/bootstrap.conf Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2625) at java.lang.Class.getMethod0(Class.java:2866) at java.lang.Class.getMethod(Class.java:1676) at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486) Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 6 more But the .tar.gz assembly file from the same build works just fine. Comparing listings of the two file sets, it appears that the .tar.gz assembly has complete set of jar files in lib/bootstrap: * activation-1.1.jar * antlr-runtime-3.5.2.jar * commons-lang3-3.4.jar * logback-classic-1.1.3.jar * logback-core-1.1.3.jar * mail-1.4.7.jar * nifi-api-0.4.1.jar * nifi-bootstrap-0.4.1.jar * nifi-expression-language-0.4.1.jar * nifi-processor-utils-0.4.1.jar * nifi-security-utils-0.4.1.jar * nifi-utils-0.4.1.jar * slf4j-api-1.7.12.jar while the RPM install's lib/bootstrap has only one of these: * nifi-bootstrap-0.4.1.jar So I tried to troubleshoot why this is. It looks like the nifi-assembly POM is set up to use the same defined set of dependencies for both the maven-assembly-plugin and rpm-maven-plugin plugins. The lib folder plan for both plugins is something like: lib - all dependencies except: * nifi-bootstrap lib/bootstrap - include only: * nifi-bootstrap The problem appears to be a difference between how the two plugins handle transitive dependencies. The maven-assembly-plugin has configuration options allowing us to specify that the exclusions apply to the named plugins AND their transitive dependencies. The rpm-maven-plugin does not have a transitive configuration option, and does not appear to do the same thing by default. In fact, the RPM is built with nifi-bootstrap-0.4.1.jar in the lib/bootstrap folder, but all of its transitive dependencies (such as slf4j from the error) are in the lib folder. I believe the RPM installs LICENSE and NOTICE in the docs folder for the same reason. But that would suggest that NiFi RPMs are not working for anyone else?
Failing Unit Test testShouldValidatePGPPublicKeyringExists
I'm having trouble with a failing unit test in nifi-standard-processors, new in a recent merge. Any helpful tips would be appreciated. Failed tests: TestEncryptContent.testShouldValidatePGPPublicKeyringExists:254 ''Public Keyring File' is invalid because Invalid Public Keyring File src/test/resources/TestEncryptContent/pubring.gpg.missing because java.io.FileNotFoundException: src\test\resources\TestEncryptContent\pubring.gpg.missing (The system cannot find the file specified)' contains ' (No such file or directory)' As I understand the test, * The file is supposed to be missing and an error is expected * The error message is expected to contain the text " (No such file or directory)" * The actual error message does not contain this, text is "'Public Keyring File' is invalid because Invalid Public Keyring File src/test/resources/TestEncryptContent/pubring.gpg.missing because java.io.FileNotFoundException: src\test\resources\TestEncryptContent\pubring.gpg.missing (The system cannot find the file specified)" https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestEncryptContent.java#L254 Am I missing a dependency or something? Things I tried: * mvn clean install (Fails) * cloning a new repo from GitHub, mvn clean install (Fails) * Running a build on a Linux machine (Works!) Perhaps it is an OS or platform issue? Uh...yeah, I have Windows. I guess that might explain different error text coming back from the file system. - Failing on Windows 10, JDK 1.8.0_66, Maven 3.3.9. - Succeded on AWS Linux, JDK 1.7.0_91, Maven 3.3.9 Any ideas? Thanks, James
Re: Failing Unit Test testShouldValidatePGPPublicKeyringExists
Not that I understand the module, but it does seem like "java.io.FileNotFoundException" would be safer. Thank you all for your help, I'll ignore the test for now using Thad's method and move on. Should I create a ticket? On Tue, Jan 12, 2016 at 3:24 PM, Andy LoPresto <alopresto.apa...@gmail.com> wrote: > Sorry, after looking at the code, we don’t get the actual exception > instance from the validation results, but the message does contain the FQ > exception type. > > `vr.toString() == "'Public Keyring File' is invalid because Invalid Public > Keyring File src/test/resources/TestEncryptContent/pubring.gpg.missing > because java.io.FileNotFoundException: > src/test/resources/TestEncryptContent/pubring.gpg.missing (No such file or > directory)”` > > If I change the assert to check for the presence of > “java.io.FileNotFoundException” > is that OK for people? > > Andy LoPresto > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jan 12, 2016, at 3:18 PM, Andy LoPresto <alopresto.apa...@gmail.com> > wrote: > > Perhaps to avoid OS-dependency on the differing messages, we could just > change the inspection to `instanceof FileNotFoundException`? > > Andy LoPresto > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jan 12, 2016, at 3:11 PM, James Wing <jvw...@gmail.com> wrote: > > Yes, my latest commit is 287d3ef5. I believe the new tests were merged in > last night or this morning ( > https://issues.apache.org/jira/browse/NIFI-1324 > or related?). > > On Tue, Jan 12, 2016 at 3:04 PM, Tony Kurc <trk...@gmail.com> wrote: > > James, > What commit are you building off of? 287d3ef5? I believe I did a full > Windows 10 and similar java/maven yesterday, but I can try to replicate. > Any chance you're using cygwin? > On Jan 12, 2016 5:58 PM, "James Wing" <jvw...@gmail.com> wrote: > > I'm having trouble with a failing unit test in nifi-standard-processors, > new in a recent merge. Any helpful tips would be appreciated. > > Failed tests: > TestEncryptContent.testShouldValidatePGPPublicKeyringExists:254 > > ''Public > > Keyring File' is invalid because Invalid Public Keyring File > src/test/resources/TestEncryptContent/pubring.gpg.missing because > java.io.FileNotFoundException: > src\test\resources\TestEncryptContent\pubring.gpg.missing (The system > cannot find the file specified)' contains ' (No such file or directory)' > > As I understand the test, > * The file is supposed to be missing and an error is expected > * The error message is expected to contain the text " (No such file or > directory)" > * The actual error message does not contain this, text is "'Public > > Keyring > > File' is invalid because Invalid Public Keyring File > src/test/resources/TestEncryptContent/pubring.gpg.missing because > java.io.FileNotFoundException: > src\test\resources\TestEncryptContent\pubring.gpg.missing (The system > cannot find the file specified)" > > > > > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestEncryptContent.java#L254 > > > Am I missing a dependency or something? Things I tried: > * mvn clean install (Fails) > * cloning a new repo from GitHub, mvn clean install (Fails) > * Running a build on a Linux machine (Works!) > > Perhaps it is an OS or platform issue? Uh...yeah, I have Windows. I > > guess > > that might explain different error text coming back from the file system. > - Failing on Windows 10, JDK 1.8.0_66, Maven 3.3.9. > - Succeded on AWS Linux, JDK 1.7.0_91, Maven 3.3.9 > > Any ideas? > > Thanks, > > James > > > > >
Re: Nifi-1325 - Enhancing Nifi AWS S3 for cross account access - Refactoring Nifi-AWS Processor credentials to use credentials provider
+1 on converting AbstractAWSProcessor to using AWSCredentialsProvider, and
+1 to adding a controller service for AWS creds. I would like to help.
The current AbstractAWSProcessor.onScheduled method already contains code
to probe for the REGION PropertyDescriptor and modify the AWS client based
on that. Could we use a similar strategy for the controller service and the
upgrade to using createClient with AWSCredentialProvider?
if (getSupportedPropertyDescriptors().contains(REGION)) { ...
My mental sketch of how this might work:
1.) AbstractAWSProcessor declares a PropertyDescriptor for the controller
service, AWS_CREDENTIAL_CONTROLLER_SERVICE, or an equally creative name
2.) The AbstractAWSProcessor.onScheduled method would probe for the
existence of AWS_CREDENTIAL_CONTROLLER_SERVICE.
2.a.) If found, onScheduled would use the new createClient(...,
AWSCredentialsProvider, ...) method, pretty much as Mans Singh describes it
2.b.) If not found, the existing createClient(..., AWSCredentials, ...)
would be used as it is now
3.) As part of the roll out of the controller service, we would modify the
various bundled AWS processors to opt-in via the
AWS_CREDENTIAL_CONTROLLER_SERVICE property.
I believe this would allow unknown custom subclasses of
AbstractAWSProcessor existing in the wild to enjoy the current contract,
while allowing us to roll out a controller service for the bundled
processors.
On the down side, this may be an unintuitive side-effect behavior. There
are very sound reasons to refactor AbstractAWSProcessor to use
AWSCredentialProvider that are not connected to the controller service as
Man Singh has discovered. We could also add a method for subclasses to
explicitly declare their credential strategy, and provide a concrete
default in AbstractAWSProcessor maintaining the existing createClient
behavior.
On Sat, Jan 9, 2016 at 11:29 AM, Aldrin Piri wrote:
> Mans,
>
> Fair points concerning the duplication. I was thinking that in conjunction
> with marking that method deprecated we would also drop the abstract
> classifier and require implementers subclassing the original class to
> provide the override explicitly. It's not ideal, but does alleviate the
> issues concerning excess methods in the interface. Sorry for omission of
> what is certainly a very valid issue.
>
> Outside of that, the items you are establishing sounds like the right
> path. I hashed this out a little more fully to better express my ideas
> [1].
>
> [1] https://gist.github.com/apiri/6a17b71e261f457daecc
>
> On Sat, Jan 9, 2016 at 1:17 PM, M Singh
> wrote:
>
> > Hi Aldrin:
> > Even if we subclass AbstractAWSProcessor and overwrite the onScheduled
> > method, we still have to add (rather then replace createClient with aws
> > creds argument) a createClient method that would take the credential
> > provider argument rather than the aws credentials argument (the current
> > implementation).
> > current nifi aws createClient (with aws credentials)
> > protected abstract ClientType createClient(final ProcessContext
> > context, final AWSCredentials credentials, final ClientConfiguration
> > config);
> > new nifi aws createClient (with aws credentials provider)
> >protected abstract ClientType createClient(final ProcessContext
> > context, final AWSCredentialsProvider credentialsProvider, final
> > ClientConfiguration config);
> > Regarding overwriting onScheduled method. Here is snippet of current
> > AbstractAWSProcessor.onScheduled method (using aws credentials):
> > @OnScheduledpublic void onScheduled(final ProcessContext context)
> > {final ClientType awsClient = createClient(context,
> > getCredentials(context), createConfiguration(context));
> > this.client = awsClient;...
> > So, in the subclass AbstractAWSProcessor.onScheduled method, we will
> check
> > if controller is available and if so, call the create client with the
> > credentials provider method. In this case each of the nifi aws processors
> > (currently s3, sns, and sqs) will have to provide two implementation of
> > create client (one with aws creds - the current one, and one with aws
> creds
> > provider).
> > I might be missing something, but it looks like there will duplication of
> > amazon client creation (one using creds and one using creds provider from
> > the controller) along with two createClient method in Nifi's
> > AbstractAWSProcessor which might causing confusion. But that is just my
> > thought.
> > Let me know what you think.
> >
> > Thanks again.
> > Mans
> >
> >
> >
> > On Saturday, January 9, 2016 9:20 AM, Aldrin Piri <
> > aldrinp...@gmail.com> wrote:
> >
> >
> > Mans,
> >
> > I think the ControllerService is definitely the right play moving
> forward.
> > What I think we can do is subclass the current AWSAbstractProcessor and
> > override onScheduled to provide a way to interact with the
> > ControllerService and, should one not be configured,
