[jira] [Comment Edited] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504352#comment-17504352 ] John Wise edited comment on NIFI-7900 at 7/18/22 4:30 PM: -- [~turcsanyip] - Although it's a bit unwieldy, the above is a viable use case for supporting temporary AWS credentials in the AWSCredentialsProviderControllerService. I presume that session token support would also require updates to most/all of the AWS processors, but it should be a minimal lift to add an additional configuration field & header. We're prohibited from creating new user accounts to acquire static access & secret keys; no, that doesn't make sense at all, but that's what we've been stuck with for several years now. Without that session token support, anyone required to use temporary credentials is essentially unable to use any AWS services with NiFi. was (Author: john.wise): [~turcsanyip] - Although it's a bit unwieldy, the above is a viable use case for supporting temporary AWS credentials in the AWSCredentialsProviderControllerService. I presume that session token support would also require updates to most/all of the AWS processors, but it should be a minimal lift to add an additional configuration field & header. Due to a draconian policy decreed by our ill-informed & stubborn security organization, we're prohibited from creating new user accounts to acquire static access & secret keys; no, that doesn't make sense at all, but that's what we've been stuck with for several years now. Without that session token support, anyone required to use temporary credentials is essentially unable to use any AWS services with NiFi. > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17556874#comment-17556874 ] iain smith edited comment on NIFI-7900 at 6/21/22 12:39 PM: [~john.wise] - Assuming your use case above is not running on an EC2 instance, could you also use default credentials, by setting 'Use Default Credentials' to 'true' then configuring your periodic STS-endpoint credentials retrieving flow to write the access key id, secret access key and session token into the AWS default credentials file $HOME/.aws/credentials (in the format shown below)? [default] aws_access_key_id = aws_secret_access_key = aws_session_token = X or set them as environment variables with the correct names maybe? That should also be picked up as long as 'Use Default Credentials' is 'true' was (Author: JIRAUSER291337): [~john.wise] - Could this be used in your use case above, by setting 'Use Default Credentials' to 'true' then configuring your periodic STS-endpoint credentials retrieving flow to write the access key id, secret access key and session token into the AWS default credentials file $HOME/.aws/credentials (in the format shown below)? [default] aws_access_key_id = aws_secret_access_key = aws_session_token = X or set them as environment variables with the correct names maybe? That should also be picked up as long as 'Use Default Credentials' is 'true' > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Comment Edited] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17556793#comment-17556793 ] iain smith edited comment on NIFI-7900 at 6/21/22 12:24 PM: update - the above use case (accessing an EC2 instance's temporary AWS credentials granted by its associated instance profile) is already supported, as per the documentation at [https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.15.3/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html] - so there is no need to try to retrieve and manually use the credentials from the EC2 metadata. All that is required is to set the AWSCredentialsProviderControllerService 'Use Default Credentials' to 'true', and ensure that no other conflicting properties are set (eg. 'Credentials File'). was (Author: JIRAUSER291337): I've just noticed on [https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.15.3/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html] "Uses default credentials without configuration. Default credentials support EC2 instance profile/role, default user profile, environment variables, etc" Is using the temporary credentials from an EC2 instance profile/role already supported? (going to test this) > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Comment Edited] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17556793#comment-17556793 ] iain smith edited comment on NIFI-7900 at 6/21/22 11:02 AM: I've just noticed on [https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.15.3/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html] "Uses default credentials without configuration. Default credentials support EC2 instance profile/role, default user profile, environment variables, etc" Is using the temporary credentials from an EC2 instance profile/role already supported? (going to test this) was (Author: JIRAUSER291337): I've just noticed on [https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.15.3/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html] "Uses default credentials without configuration. Default credentials support EC2 instance profile/role, default user profile, environment variables, etc" Is using the temporary credentials from an EC2 instance profile/role already supported? If so, should they be being automatically detected and used as long as no key id / access key is configured in nifi? > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Comment Edited] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[
https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503064#comment-17503064
]
John Wise edited comment on NIFI-7900 at 3/8/22, 5:06 PM:
--
FWIW, I've created a flow which queries the AWS API at the STS endpoint every 4
hours for temporary credentials, grabs the access key, secret key, session
token, and expiration into attributes, updates an
AWSCredentialsProviderControllerService with the keys, then restarts it.
Unfortunately, without the ability to pass in & use the session token, none of
the attached AWS processors can authenticate, even though the access & secret
keys are current at the time the flow runs.
>From Amazon's "[Using temporary credentials with AWS resources
>page|https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html]":
* When you make a call using temporary security credentials, the call {*}must
include a session token{*}, which is returned along with those temporary
credentials. AWS uses the session token to validate the temporary security
credentials.
was (Author: john.wise):
FWIW, I've created a flow which queries the AWS API at the STS endpoint every 4
hours for temporary credentials, grabs the access key, secret key, session
token, and expiration into attributes, updates an
AWSCredentialsProviderControllerService with the keys, then restarts it.
Unfortunately, without the ability to pass in & use the session token, none of
the attached AWS processors can authenticate, even though the access & secret
keys are current at the time the flow runs.
> Add AWS session token to AWSCredentialsProvider
> ---
>
> Key: NIFI-7900
> URL: https://issues.apache.org/jira/browse/NIFI-7900
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.9.2, 1.12.1
>Reporter: Jody
>Assignee: Peter Turcsanyi
>Priority: Major
>
> As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor,
> with temporary credentials to allow connecting to secure AWS environments
> that make use of the AWS Security Token Service.
>
> The NiFi AWSCredentialsProviderControllerService is giving an option to add
> the required fields for using temporary credentials. While access key id and
> secret access key properties can be configured, the property "session token"
> is not available. The session token property must be provided when temporary
> credentials are used. If the session token is not presented, an error will be
> thrown: "The AWS Access Key Id you provided does not exist in our records.
> (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId"
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503064#comment-17503064 ] John Wise edited comment on NIFI-7900 at 3/8/22, 5:03 PM: -- FWIW, I've created a flow which queries the AWS API at the STS endpoint every 4 hours for temporary credentials, grabs the access key, secret key, session token, and expiration into attributes, updates an AWSCredentialsProviderControllerService with the keys, then restarts it. Unfortunately, without the ability to pass in & use the session token, none of the attached AWS processors can authenticate, even though the access & secret keys are current at the time the flow runs. was (Author: john.wise): FWIW, I've created a flow which queries the AWS API every 4 hours for temporary credentials, grabs the access key, secret key, session token, and expiration into attributes, updates an AWSCredentialsProviderControllerService with the keys, then restarts it. Unfortunately, without the ability to pass in & use the session token, none of the attached AWS processors can authenticate, even though the access & secret keys are current at the time the flow runs. > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.1#820001)
