Keren, I'm sorry if my advice is a bit confusing, there have been some changes to AWS credentials over the last few versions. NiFi 0.6.0 does not have the option to use a CLI profile in AWSCredentialsProviderControllerService, I think that was introduced in 0.7.0.
Would it be possible to share some of the log entries? Was there a stack trace associated with the timeout? Also, does your cross-account Role require an External ID? I do not believe that is supported in 0.6.0, but can be required to assume some roles. Thanks, James On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren < keren.tseyt...@capitalone.com> wrote: > Thanks for your responses! > > @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0. > > I¹ve set up debugging, and it shows me that it¹s trying to connect, but it > times out on connecting. It would be awesome if it would also return the > account ID of the credentials it is trying to use. > > Is there any way to see the exact keys/tokens that Nifi is trying to use > to get the S3 object? I¹m not seeing it in the logs. > > I tried to set the Profile in Nifi, but it complains that it¹s not a valid > property. > > Best, > Keren > > On 8/31/16, 6:24 PM, "Andrew Grande" <apere...@gmail.com> wrote: > > >Debug logging can be set in a processor itself in the UI, too. > > > >On Wed, Aug 31, 2016, 5:34 PM James Wing <jvw...@gmail.com> wrote: > > > >> Keren, > >> > >> Which version of NiFi are you using? > >> > >> One thing I noticed in your configuration of FetchS3Object is you are > >> setting both the Access Key and Secret Key properties with the AWS > >> Credentials Provider. When you are using the AWS Credentials Provider > >> Service, you should not specify keys. > >> > >> A more certainly helpful thing to do is enable debug logging for the AWS > >> processor package by adding a line like the following to > >>conf/logback.xml: > >> > >> <logger name="org.apache.nifi.processors.aws" level="DEBUG"/> > >> > >> With the debug logging enabled, there are messages indicating which > >> credential type is being attempted. Your settings for the AWS > >>Credentials > >> Provider look appropriate. The controller service is indeed designed to > >> refresh the STS token automagically using the AWS SDK classes for > >>temporary > >> credentials. > >> > >> Last, you might experiment with configuring > >> AWSCredentialsProviderControllerService to use your named CLI profile > >> "crossaccountrole", which should also work. > >> > >> Thanks, > >> > >> James > >> > >> On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren < > >> keren.tseyt...@capitalone.com> wrote: > >> > >> > Hi All! > >> > > >> > Looking for some help on enabling Cross Account communication within > >> Nifi! > >> > > >> > My goal: There are files stored from CloudTrail in an S3 bucket in > >>VPC B. > >> > My Nifi machines are in VPC A. I want Nifi to be able to get those > >>files > >> > from VPC B. VPC A and VPC B need to be communicating in the > >>FetchS3Object > >> > component. > >> > > >> > See this link for some additional info: http://docs.aws.amazon.com/ > >> > > >>awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html > >> > > >> > I have communication working manually on the Nifi machines in VPC A > >>when > >> I > >> > use the AWS CLI. The process is as follows: > >> > > >> > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a > >>role > >> > I've created in VPC B that is configured to have access to the S3 > >>bucket > >> in > >> > VPC B. > >> > > >> > 2. This will generate temporary keys that need to be refreshed > >>every > >> > hour. There is no way to have assume role create permanent keys. > >>Export > >> the > >> > keys as environment variables. > >> > > >> > 3. Set up ~/.aws/config to have a profile "crossaccountrole" that > >> > connects to the arn of the role created in VPC B. > >> > > >> > 4. Run the following command à "aws s3 cp s3://<object path> > >><object > >> > name locally> --profile crossaccountrole" > >> > > >> > Most importantly, if I ever try to run this without the --profile > >>flag, > >> > then it will not allow me to download the file. It seems like > >>perhaps to > >> > get it to work with Nifi I need a place to pass in the profile that > >>needs > >> > to be used in order for the communication to work. > >> > > >> > I've been trying to implement this in Nifi. Within the FetchS3Object, > >>I > >> > have created an AWSCredentialsProviderService which has the following > >> > properties: > >> > > >> > · Access Key: VPC A access key > >> > > >> > · Secret Key: VPC A secret key > >> > > >> > · Assume Role ARN: VPC B role > >> > > >> > · Assume Role Session Name: crossaccountrole > >> > > >> > · Session Time: 3600 > >> > The general properties in the FetchS3Object are as follows: > >> > > >> > · Bucket: VPC B bucket name > >> > > >> > · Object: Filename of VPC B bucket object > >> > > >> > · Access Key: VPC A access key > >> > > >> > · Secret Key: VPC A secret key > >> > > >> > · AWS Credentials Provider Service: <see above> > >> > > >> > However, when this tries to run I get Access Denied. I've been going > >> > through the source code for Nifi and I'm not sure if short-lived > >>tokens > >> get > >> > passed through. Can anyone please provide me some guidance or > >>suggestions > >> > on how to get this to work? J > >> > > >> > Best, > >> > Keren > >> > ________________________________________________________ > >> > > >> > The information contained in this e-mail is confidential and/or > >> > proprietary to Capital One and/or its affiliates and may only be used > >> > solely in performance of work or services for Capital One. The > >> information > >> > transmitted herewith is intended only for use by the individual or > >>entity > >> > to which it is addressed. If the reader of this message is not the > >> intended > >> > recipient, you are hereby notified that any review, retransmission, > >> > dissemination, distribution, copying or other use of, or taking of any > >> > action in reliance upon this information is strictly prohibited. If > >>you > >> > have received this communication in error, please contact the sender > >>and > >> > delete the material from your computer. > >> > > >> > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > >
