Re: Nifi-1325 - Enhancing Nifi AWS S3 for cross account access - Refactoring Nifi-AWS Processor credentials to use credentials provider
Hi Aldrin: The unfortunate things is that AWSCredentialsProvider does not inherit from AWSCredentials interface. As far as I can see, the provider interface is much more flexible and provides everything with we/anyone can need. As we can see, the creds based constructors (AmazonS3/SQS/SNSClients) internally create a static creds provider instance. If we support both the creds and creds provider based arguments, it could also confusing and error prone for developers extending the class. Even if we have an adapter how would the subclass of AbstractAWSProcessor call createClient allow the two arguments (creds and creds provider) to work seemlessly. Let me know if you have any other thoughts/paths I can investigate. Thanks for the feedback and I am learning a lot with this experience. Mans On Friday, January 8, 2016 11:49 AM, Aldrin Piriwrote: Mans, Thanks for sticking with this and continuing to see things through, the community certainly appreciates it as these are very popular processors and this functionality will help a wide base of users. I am poking around a bit more and thinking we might be able to work something out with a class that adapts an AWSCredentialsProvider to AWSCredentials. The AWSProvider interface is just composition of AWSCredentials with a refresh method. Need to mull things over a bit, and dig through the associated libraries to understand how these are typically used, but this feels like it could be another avenue to consider and where I am directing my attention at the moment. --aldrin On Fri, Jan 8, 2016 at 9:58 AM, M Singh wrote: > Just one more note Joe (as mentioned in the Jira Ticket) > From what I can see, we cannot just deprecate createClient method in the > AbstractAWSProcessor which uses the AWSCredentials argument, since the > subclasses AbstractS3/SNS/SQSProcessor call that to create the respective > clients. We will have to change the argument to AWSCredentialProvider. > If I can assist with any other investigation, please let me know. > Thanks again. > > On Friday, January 8, 2016 5:31 AM, M Singh > wrote: > > > Thanks Joe. > If you think that we can accept the change to creds provider, I will work > on making all the components in nifi aws processors to be consistent. I > think using the creds provider interface is the way to go since it is more > flexible and at this moment we just have 3 aws processors to migrate. > Looking forward to hearing from you/anyone else for advice/feedback. > Mans > > On Friday, January 8, 2016 5:18 AM, Joe Witt > wrote: > > > Mans, > > I am working to put out a proposed roadmap and then probably won't be > very responsive until later tonight. Will try to help then if no one > else has had a chance to. > > That said I see what you mean in terms of a breaking change in the > processor implementation as far as anyone else that has extended it. > There have been some discussions recently about this and I think the > plan is to start annotating everything with the audience and stability > of a given bit of code. Processors are not meant to be locked down > APIs. So, for now, given that it has been ambiguous to the community > the best course is to probably just deprecate a given method if it > cannot be safely repurposed and then use a new one which does meet the > need in the event the controller service is supplied. This last > statement though is not based on me having looked at the code in any > detail yet. > > Thanks > Joe > > On Fri, Jan 8, 2016 at 8:08 AM, M Singh > wrote: > > Hi Joe: > > I have not worked with the controller interface and aws processors so > perhaps you can help me understand it . > > From what I can see (as mentioned in > https://issues.apache.org/jira/browse/NIFI-1325): currently, the Nifi > AbstractAWSProcessor has a method > > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentials credentials, final ClientConfiguration config); > > This method is overridden in the AbstractS3/SNS/SQSProcesors to provide > the respective amazon s3/sns/sqs client using AWSCredentials argument. > > Here is a snippet from AmazonS3Client: > > public AmazonS3Client(AWSCredentials awsCredentials, > ClientConfiguration clientConfiguration) { > super(clientConfiguration); this.awsCredentialsProvider = new > StaticCredentialsProvider(awsCredentials); init(); } > > So, AmazonS3/SNS/SQSClient created with AWSCredentials use > StaticCredentialsProvider in their implementation. > > All the AWSCredentials impls are static creds (Anonymous/Properties > Credentials) except for the STSSessionCredentials which has a refresh > method but is deprecated in favor of the STSSessionCredentialsProvider > interface. AWSCredentials is the interface being used in nifi aws > processors. > > The AWSCredentialsProvider
S3 configuration
Greetings, I'm trying to do a PutS3Object. When I try to use it I get : 2016-01-08 15:43:17,518 INFO [Timer-Driven Process Thread-6] com.amazonaws.http.AmazonHttpClient Unable to execute HTTP request: Connection refused: conjava.net.ConnectException: Connection refused: connect at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_66] ... Can someone give me a clue as to what I'm doing wrong? I set the bucket, access key, secret key, and the region. Everything else I left as the default. Is there more I have to do? thank you, Patrick Jones -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/S3-configuration-tp6122.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Nifi + Oracle redo files?
Hello… I’ve been looking through the documentation and have not found any reference to Nifi reading Oracle redo files. Are you aware of anyone who has done this successfully? …or pursued a similar alternative to directly hitting transactional Oracle tables? Thanks, Jason
Re: Nifi + Oracle redo files?
Jason, I'm not aware of anyone reading Oracle redo logs via NiFi directly yet. More and more we're interacting with relational databases and their related logs as a source but some are more challenging than others to support in an open source capacity. Our existing SQL oriented processors have been shown to work fine on certain Oracle versions for pulling or pushing data into tables but fall short of providing complete change data capture or bulk table export as-is. There is definitely more we can and should do there. Are you aware of any libraries that are open source friendly and support Oracle's redo logs? Thanks Joe On Fri, Jan 8, 2016 at 4:53 PM, Jason.Byerswrote: > Hello… I’ve been looking through the documentation and have not found any > reference to Nifi reading Oracle redo files. > > Are you aware of anyone who has done this successfully? …or pursued a > similar alternative to directly hitting transactional Oracle tables? > > Thanks, > > Jason
Re: Nifi-1325 - Enhancing Nifi AWS S3 for cross account access - Refactoring Nifi-AWS Processor credentials to use credentials provider
An AWS controller service may be a good idea. I've been talking to Tony about and working on state management for the S3 Multipart upload, to make sure that aborted uploads get cleaned up. I'm keeping it in the processor for now, but if there's other benefit to an AWS controller service, such as credential management, it may make sense to move the local and remote state management there too so it will be handled in one central place regardless outside of any processors. On Fri, Jan 8, 2016 at 8:31 AM, M Singhwrote: > Thanks Joe. > If you think that we can accept the change to creds provider, I will work > on making all the components in nifi aws processors to be consistent. I > think using the creds provider interface is the way to go since it is more > flexible and at this moment we just have 3 aws processors to migrate. > Looking forward to hearing from you/anyone else for advice/feedback. > Mans > > On Friday, January 8, 2016 5:18 AM, Joe Witt > wrote: > > > Mans, > > I am working to put out a proposed roadmap and then probably won't be > very responsive until later tonight. Will try to help then if no one > else has had a chance to. > > That said I see what you mean in terms of a breaking change in the > processor implementation as far as anyone else that has extended it. > There have been some discussions recently about this and I think the > plan is to start annotating everything with the audience and stability > of a given bit of code. Processors are not meant to be locked down > APIs. So, for now, given that it has been ambiguous to the community > the best course is to probably just deprecate a given method if it > cannot be safely repurposed and then use a new one which does meet the > need in the event the controller service is supplied. This last > statement though is not based on me having looked at the code in any > detail yet. > > Thanks > Joe > > On Fri, Jan 8, 2016 at 8:08 AM, M Singh > wrote: > > Hi Joe: > > I have not worked with the controller interface and aws processors so > perhaps you can help me understand it . > > From what I can see (as mentioned in > https://issues.apache.org/jira/browse/NIFI-1325): currently, the Nifi > AbstractAWSProcessor has a method > > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentials credentials, final ClientConfiguration config); > > This method is overridden in the AbstractS3/SNS/SQSProcesors to provide > the respective amazon s3/sns/sqs client using AWSCredentials argument. > > Here is a snippet from AmazonS3Client: > >public AmazonS3Client(AWSCredentials awsCredentials, > ClientConfiguration clientConfiguration) { > super(clientConfiguration);this.awsCredentialsProvider = new > StaticCredentialsProvider(awsCredentials);init();} > > So, AmazonS3/SNS/SQSClient created with AWSCredentials use > StaticCredentialsProvider in their implementation. > > All the AWSCredentials impls are static creds (Anonymous/Properties > Credentials) except for the STSSessionCredentials which has a refresh > method but is deprecated in favor of the STSSessionCredentialsProvider > interface. AWSCredentials is the interface being used in nifi aws > processors. > > The AWSCredentialsProvider interface has a fresh method which all it's > subclasses implement appropriately - the static ones (like > PropertyFileCredentialsProvider/StaticCredentialsProvider have a no op for > refresh method) as follows: > >public void refresh() {} > > From what I can see, there is no common interface available for > AWSCredentials and AWSCredentialsProvider that Nifi's > AbstractAWSProcessor.createClient can support. So if we need to use the > controller interface with creds providers, will will have to change > AbstractAWSProcessor.createClient to the following > > > > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentialsProvider credentialsProvider, > > final ClientConfiguration config); > > This appears to be a breaking change for the clients who have extended > the AbstractAWSProcessor.createClient with the AWSCredentials argument > rather that the AWSCredentialsProvider. > > So, can you please elaborate on how the AbstractAWSProcessor will be > able to support both the current impl (ie, invoking aws components with > creds) and the proposed credentials provider interface ? > > Thanks > > Mans > > > >On Thursday, January 7, 2016 9:00 PM, Joe Witt > wrote: > > > > > > Mans, > > > > It appears to me that there is a path for this not to be a breaking > > change for the flow. By creating a controller service to handle the > > credential provider piece you should be able to just update the > > processor to support that controller service interface. If the user > > sets that controller service then you use that and if they don't then > > you revert to using
Re: Nifi-1325 - Enhancing Nifi AWS S3 for cross account access - Refactoring Nifi-AWS Processor credentials to use credentials provider
Mans, I am working to put out a proposed roadmap and then probably won't be very responsive until later tonight. Will try to help then if no one else has had a chance to. That said I see what you mean in terms of a breaking change in the processor implementation as far as anyone else that has extended it. There have been some discussions recently about this and I think the plan is to start annotating everything with the audience and stability of a given bit of code. Processors are not meant to be locked down APIs. So, for now, given that it has been ambiguous to the community the best course is to probably just deprecate a given method if it cannot be safely repurposed and then use a new one which does meet the need in the event the controller service is supplied. This last statement though is not based on me having looked at the code in any detail yet. Thanks Joe On Fri, Jan 8, 2016 at 8:08 AM, M Singhwrote: > Hi Joe: > I have not worked with the controller interface and aws processors so perhaps > you can help me understand it . > From what I can see (as mentioned in > https://issues.apache.org/jira/browse/NIFI-1325): currently, the Nifi > AbstractAWSProcessor has a method > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentials credentials, final ClientConfiguration config); > This method is overridden in the AbstractS3/SNS/SQSProcesors to provide the > respective amazon s3/sns/sqs client using AWSCredentials argument. > Here is a snippet from AmazonS3Client: > public AmazonS3Client(AWSCredentials awsCredentials, ClientConfiguration > clientConfiguration) {super(clientConfiguration); > this.awsCredentialsProvider = new StaticCredentialsProvider(awsCredentials); > init();} > So, AmazonS3/SNS/SQSClient created with AWSCredentials use > StaticCredentialsProvider in their implementation. > All the AWSCredentials impls are static creds (Anonymous/Properties > Credentials) except for the STSSessionCredentials which has a refresh method > but is deprecated in favor of the STSSessionCredentialsProvider interface. > AWSCredentials is the interface being used in nifi aws processors. > The AWSCredentialsProvider interface has a fresh method which all it's > subclasses implement appropriately - the static ones (like > PropertyFileCredentialsProvider/StaticCredentialsProvider have a no op for > refresh method) as follows: > public void refresh() {} > From what I can see, there is no common interface available for > AWSCredentials and AWSCredentialsProvider that Nifi's > AbstractAWSProcessor.createClient can support. So if we need to use the > controller interface with creds providers, will will have to change > AbstractAWSProcessor.createClient to the following > > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentialsProvider credentialsProvider, > final ClientConfiguration config); > This appears to be a breaking change for the clients who have extended the > AbstractAWSProcessor.createClient with the AWSCredentials argument rather > that the AWSCredentialsProvider. > So, can you please elaborate on how the AbstractAWSProcessor will be able to > support both the current impl (ie, invoking aws components with creds) and > the proposed credentials provider interface ? > Thanks > Mans > > On Thursday, January 7, 2016 9:00 PM, Joe Witt wrote: > > > Mans, > > It appears to me that there is a path for this not to be a breaking > change for the flow. By creating a controller service to handle the > credential provider piece you should be able to just update the > processor to support that controller service interface. If the user > sets that controller service then you use that and if they don't then > you revert to using the older properties. We can mark those > properties as no longer the preferred model and deprecate them in the > codebase then when we reach a 1.0 milestone we can remove them. > > Thanks > Joe > > On Thu, Jan 7, 2016 at 9:07 PM, M Singh wrote: >> Hi: >> Just wanted to mention that if we go with the creds provider interface it >> will be breaking change for nifi aws components as mentioned in >> https://issues.apache.org/jira/browse/NIFI-1325. >> Also, I am considering creating one aws creds provider controller which will >> provide creds provider based on property file, basic, anonymous or assume >> role session params. >> Please let me know if there is any additional feedback for me. >> Thanks again. >> Mans >> >>On Thursday, January 7, 2016 2:56 PM, M Singh >> wrote: >> >> >> Hi Joe: >> Based on your feedback I will try to explore the controller interface for >> aws creds provider. >> Thanks for your advice. >> Mans >> >>On Thursday, January 7, 2016 4:15 AM, Joe Witt wrote: >> >> >> Mans >> >>
Re: How to remote debug Jetty UI part
Lei, Are you interested in debugging Jetty or NiFi's UI? Assuming you're referring to the NiFi UI, you'll need to rebuild because by default all the JS/CSS files are aggregated and minified. When you rebuild, you'll need to disable the minify-and-compress maven profile. Assuming your running from the command line it would look something like this: $ mvn clean install -P\!minify-and-compress The ! tells maven to disable the profile while the \ is escaping the ! in your shell. If your building through an IDE then you likely wouldn't need the !. The resulting build ends up in nifi/nifi-assembly/target/nifi--bin/nifi-. Once you launch NiFi you should be able to use the developer tools in your browser to debug the client side Javascript that allows most of the UI to happen. If you're interested in debugging the server side of things (Jetty or NiFi REST API), then you'll want to attach the remote debugging as you've already done and add breakpoints to any Jetty class or the NiFI REST API which is located here: https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api Let me know if you have any more questions. Matt On Fri, Jan 8, 2016 at 4:02 AM, 杨磊wrote: > Hi, > > > I am a newbie to Nifi community. Although i read Hortonwork's announcement > on the dataflow months ago, i just realize how amazing the Nifi is. > However, when i am eager to learn more and more about Nifi, i got some > trouble to debug it. > Could you please tell me how to remote debug the Jetty UI part? Or > introducing the regular way to develop this part to us? > I have uncommented the remote debugging line on conf/boostrap.conf and > attached my Intellij on it. > > > > > Thanks in advance! > Lei
Re: Nifi-1325 - Enhancing Nifi AWS S3 for cross account access - Refactoring Nifi-AWS Processor credentials to use credentials provider
Just one more note Joe (as mentioned in the Jira Ticket) >From what I can see, we cannot just deprecate createClient method in the >AbstractAWSProcessor which uses the AWSCredentials argument, since the >subclasses AbstractS3/SNS/SQSProcessor call that to create the respective >clients. We will have to change the argument to AWSCredentialProvider. If I can assist with any other investigation, please let me know. Thanks again. On Friday, January 8, 2016 5:31 AM, M Singhwrote: Thanks Joe. If you think that we can accept the change to creds provider, I will work on making all the components in nifi aws processors to be consistent. I think using the creds provider interface is the way to go since it is more flexible and at this moment we just have 3 aws processors to migrate. Looking forward to hearing from you/anyone else for advice/feedback. Mans On Friday, January 8, 2016 5:18 AM, Joe Witt wrote: Mans, I am working to put out a proposed roadmap and then probably won't be very responsive until later tonight. Will try to help then if no one else has had a chance to. That said I see what you mean in terms of a breaking change in the processor implementation as far as anyone else that has extended it. There have been some discussions recently about this and I think the plan is to start annotating everything with the audience and stability of a given bit of code. Processors are not meant to be locked down APIs. So, for now, given that it has been ambiguous to the community the best course is to probably just deprecate a given method if it cannot be safely repurposed and then use a new one which does meet the need in the event the controller service is supplied. This last statement though is not based on me having looked at the code in any detail yet. Thanks Joe On Fri, Jan 8, 2016 at 8:08 AM, M Singh wrote: > Hi Joe: > I have not worked with the controller interface and aws processors so perhaps > you can help me understand it . > From what I can see (as mentioned in > https://issues.apache.org/jira/browse/NIFI-1325): currently, the Nifi > AbstractAWSProcessor has a method > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentials credentials, final ClientConfiguration config); > This method is overridden in the AbstractS3/SNS/SQSProcesors to provide the > respective amazon s3/sns/sqs client using AWSCredentials argument. > Here is a snippet from AmazonS3Client: > public AmazonS3Client(AWSCredentials awsCredentials, ClientConfiguration >clientConfiguration) { super(clientConfiguration); >this.awsCredentialsProvider = new StaticCredentialsProvider(awsCredentials); > init(); } > So, AmazonS3/SNS/SQSClient created with AWSCredentials use > StaticCredentialsProvider in their implementation. > All the AWSCredentials impls are static creds (Anonymous/Properties > Credentials) except for the STSSessionCredentials which has a refresh method > but is deprecated in favor of the STSSessionCredentialsProvider interface. > AWSCredentials is the interface being used in nifi aws processors. > The AWSCredentialsProvider interface has a fresh method which all it's > subclasses implement appropriately - the static ones (like > PropertyFileCredentialsProvider/StaticCredentialsProvider have a no op for > refresh method) as follows: > public void refresh() {} > From what I can see, there is no common interface available for > AWSCredentials and AWSCredentialsProvider that Nifi's > AbstractAWSProcessor.createClient can support. So if we need to use the > controller interface with creds providers, will will have to change > AbstractAWSProcessor.createClient to the following > > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentialsProvider credentialsProvider, > final ClientConfiguration config); > This appears to be a breaking change for the clients who have extended the > AbstractAWSProcessor.createClient with the AWSCredentials argument rather > that the AWSCredentialsProvider. > So, can you please elaborate on how the AbstractAWSProcessor will be able to > support both the current impl (ie, invoking aws components with creds) and > the proposed credentials provider interface ? > Thanks > Mans > > On Thursday, January 7, 2016 9:00 PM, Joe Witt wrote: > > > Mans, > > It appears to me that there is a path for this not to be a breaking > change for the flow. By creating a controller service to handle the > credential provider piece you should be able to just update the > processor to support that controller service interface. If the user > sets that controller service then you use that and if they don't then > you revert to using the older properties. We can mark those > properties as no longer the preferred model and deprecate them in the > codebase
Re: [DISCUSS] roadmap for the next 6-12 months
Joe! Looks amazing! Thanks for pulling this together Hello NiFi Community Please review the proposed roadmap and suggest things to add, remove, update, prioritize differently. Over the past year we have had great contributions that improved the quality of NiFi and the community. As we know there are many JIRAs related to improvements or new features. However, it can be difficult for new and existing community members to know when to jump in, how to help, or when things they are interested in might be able to happen. So among other things we should provide and maintain a documented roadmap to help this. The following suggested timeline follows pretty strictly against our previously stated objective of having releases roughly every 6 weeks. We do not need to stay strict to this but a time bound (as opposed to feature bound) release cycle does help provide a regular cadence to follow and helps give contributors a sense of when their efforts might end up in a release. The items below are largely just focused on some of the major muscle movements needed at a framework level. There will of course be many many more JIRAs with each releas. NiFi 0.5.0 (Feb 5) State Management https://cwiki.apache.org/confluence/display/NIFI/State+Management Kerberos https://cwiki.apache.org/confluence/display/NIFI/Pluggable+Authentication Interactive Queue https://cwiki.apache.org/confluence/display/NIFI/Interactive+Queue+Management Scripting Support https://issues.apache.org/jira/browse/NIFI-210 NiFi 0.6.0 (Mar 18) Deterministic Template Export https://issues.apache.org/jira/browse/NIFI-826 Schema/Format Editor (JSON to JSON,…?) Numerous JIRAs related to format/schema conversion. An example implementation: https://github.com/fsauer65/NiFi-Extensions/tree/master/nifi-jsontransform-bundle NiFi 0.7.0 (Apr 29) NiFi 1.0.0 (Jun 17) HA Cluster Management https://cwiki.apache.org/confluence/display/NIFI/Clustering+Redesign HA Data https://cwiki.apache.org/confluence/display/NIFI/High+Availability+Processing Multi-Tenant Authorization https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow Improved Authorization API https://cwiki.apache.org/confluence/display/NIFI/Pluggable+Authentication NiFi's current authority provider gathers data then makes a decision. We should instead support PDP/PEP model [4]. Modernized UX https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface Dependent Properties https://issues.apache.org/jira/browse/NIFI-1121 NiFi 1.1.0 (Sep?) Template/Extension Registry https://cwiki.apache.org/confluence/display/NIFI/Extension%2C+Template%2C+Dataset+Registry Variable Registry https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry The timing of these things is pretty important as far as interdependencies and how long they'll take to develop and validate. State management in 050 sets up an important way to have cluster-wide shared state and introduces zookeeper formally into NiFi which obviously will be important for the distributed systems behaviors we'll need as we restructure how clustering works and introduce data replication for HA data processing. The 060 and 070 releases are intentionally quite light to allow for very focused work on a parallel branch of 1.x which probably should start right on the heels of 050. The 1.0 release will introduce some very key features, offer a chance to remove deprecated classes, ensure all APIs are extremely clearly documented for stability and audience, allow us to introduce UI/UX changes, and so on. Now, of course this is simply a proposal and these timings may not work out. But this should serve as a good basis for discussion and consensus forming. As we progress I plan to update the roadmap [1], product requirements [2], and feature proposals [3] to reflect these things a bit more clearly than they are now. We should also have a discussion on how long we should be committed to supporting the 0.x line and what that means. We need to document a commitment for the community. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851850 [2] https://cwiki.apache.org/confluence/display/NIFI/Product+requirements [3] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals [4] https://en.wikipedia.org/wiki/Common_Open_Policy_Service Thanks Joe