[
https://issues.apache.org/jira/browse/NIFI-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18018329#comment-18018329
]
Nick edited comment on NIFI-14927 at 9/5/25 4:08 PM:
-----------------------------------------------------
In the meantime, I tried to access via a named profile which can assume the IAM
Role.
Funnily enough, this works for MSK, but doesn't appear to work with the AWS
Credential Provider (which I initially used with S3 for testing).
Anyway, I thought I'd write the steps down here as it was a little convoluted.
I needed to create an AWS config (like the one that works with the AWS CLI), eg:
~/.aws/config:
{code:bash}
[profile pod-role]
# Call a script to get the credentials for the PodIdentity
# For whatever reason, even though the credentials are set in the pod,
# the custom profile is not able to infer them.
# Assume HOME=/app
credential_process=/app/.aws/get-token.sh
[profile msk-role]
role_arn=arn:aws:iam::123456789012:role/nifi-node-msk
role_session_name=AssumeFromProfile
source_profile=pod-role {code}
get-token.sh
{code:bash}
#!/usr/bin/env sh
# Need to rename Token to SessionToken, and add a Version.
# See:
https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#sourcing-credentials-from-external-processes
# There is no jq in the container, so we'll need to use sed (or similar) to fix
the JSON structure
# curl -H "Authorization: $(cat $AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE)"
"$AWS_CONTAINER_CREDENTIALS_FULL_URI" \
# | jq -c '{AccessKeyId: .AccessKeyId, SecretAccessKey: .SecretAccessKey,
SessionToken: .Token, Expiration: .Expiration, Version: 1}'
curl -H "Authorization: $(cat $AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE)"
"$AWS_CONTAINER_CREDENTIALS_FULL_URI" \
| sed -e 's|Token|SessionToken|' | sed -e 's|}|, "Version": 1}|'{code}
In my case, I added those to a ConfigMap, then mounted them into the pod.
Unfortunately I had to do multiple mounts because the CLI (or perhaps the AWS
SDK) needs write access to ~/.aws/.
{code:yaml}
volumes:
# AWS CLI config
# aws cli needs to be able to write into ~/.aws/cli when assuming
roles,
# hence emptyDir
- name: aws
emptyDir: {}
- name: aws-config
configMap:
name: aws-profile
items:
- key: config
path: config
- name: aws-get-token
configMap:
name: aws-profile
defaultMode: 0777 # need the script to be executable
items:
- key: get-token.sh
path: get-token.sh
containers:
- name: app
volumeMounts:
- name: aws
# $HOME/.aws
mountPath: /app/.aws
- name: aws-config
# $HOME/.aws/config
mountPath: /app/.aws/config
subPath: config
- name: aws-get-token
# $HOME/.aws/get-token.sh
mountPath: /app/.aws/get-token.sh
subPath: get-token.sh {code}
Test with the aws CLI:
{code:json}
simple-nifi-node-default-0 /app $ aws --profile pod-role sts get-caller-identity
{
"UserId":
"AROAXXXXXXXXXXXXXXX2N:eks-my-cluster-simple-nif-6a2607ae-ded1-40fe-85ae-3e4372c68432",
"Account": "123456789012",
"Arn":
"arn:aws:sts::123456789012:assumed-role/nifi-nod/eks-my-cluster-simple-nif-6a2607ae-ded1-40fe-85ae-3e4372c68432"
}
simple-nifi-node-default-0 /app $ aws --profile msk-role sts get-caller-identity
{
"UserId": "AROAXXXXXXXXXXXXXXXKM:AssumeFromProfile",
"Account": "123456789012",
"Arn":
"arn:aws:sts::123456789012:assumed-role/nifi-node-msk/AssumeFromProfile"
} {code}
Use the name profile in the AmazonMSKConnectionService:
!image-2025-09-05-11-11-06-768.png|width=573,height=426!
Then test using PublishKafka:
!image-2025-09-05-11-13-03-732.png|width=572,height=386!
was (Author: JIRAUSER306966):
In the meantime, I tried to access via a named profile which can assume the IAM
Role.
Funnily enough, this works for MSK, but doesn't appear to work with the AWS
Credential Provider (which I initially used with S3 for testing).
Anyway, I thought I'd write the steps down here as it was a little convoluted.
I needed to create an AWS config (like the one that works with the AWS CLI), eg:
~/.aws/config:
{code:bash}
[profile pod-role]
# Call a script to get the credentials for the PodIdentity
# For whatever reason, even though the credentials are set in the pod,
# the custom profile is not able to infer them.
# Assume HOME=/app
credential_process=/app/.aws/get-token.sh
[profile msk-role]
role_arn=arn:aws:iam::123456789012:role/nifi-node-msk
role_session_name=AssumeFromProfile
source_profile=pod-role {code}
get-token.sh
{code:bash}
#!/usr/bin/env sh
# Need to rename Token to SessionToken, and add a Version.
# See:
https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#sourcing-credentials-from-external-processes
# There is no jq in the container, so we'll need to use sed (or similar) to fix
the JSON structure
# curl -H "Authorization: $(cat $AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE)"
"$AWS_CONTAINER_CREDENTIALS_FULL_URI" \
# | jq -c '{AccessKeyId: .AccessKeyId, SecretAccessKey: .SecretAccessKey,
SessionToken: .Token, Expiration: .Expiration, Version: 1}'
curl -H "Authorization: $(cat $AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE)"
"$AWS_CONTAINER_CREDENTIALS_FULL_URI" \
| sed -e 's|Token|SessionToken|' | sed -e 's|}|, "Version": 1}|'{code}
In my case, I added those to a ConfigMap, then mounted them into the pod.
Unfortunately I had to do multiple mounts because the CLI (or perhaps the AWS
SDK) needs write access to ~/.aws/.
{code:yaml}
volumes:
# AWS CLI config
# aws cli needs to be able to write into ~/.aws/cli when assuming
roles,
# hence emptyDir
- name: aws
emptyDir: {}
- name: aws-config
configMap:
name: aws-profile
items:
- key: config
path: config
- name: aws-get-token
configMap:
name: aws-profile
defaultMode: 0777 # need the script to be executable
items:
- key: get-token.sh
path: get-token.sh
containers:
- name: app
volumeMounts:
- name: aws
# $HOME/.aws
mountPath: /app/.aws
- name: aws-config
# $HOME/.aws/config
mountPath: /app/.aws/config
subPath: config
- name: aws-get-token
# $HOME/.aws/get-token.sh
mountPath: /app/.aws/get-token.sh
subPath: get-token.sh {code}
Test with the aws CLI:
{code:json}
simple-nifi-node-default-0 /app $ aws --profile pod-role sts get-caller-identity
{
"UserId":
"AROAXXXXXXXXXXXXXXX2N:eks-my-cluster-simple-nif-6a2607ae-ded1-40fe-85ae-3e4372c68432",
"Account": "123456789012",
"Arn":
"arn:aws:sts::123456789012:assumed-role/nifi-nod/eks-my-cluster-simple-nif-6a2607ae-ded1-40fe-85ae-3e4372c68432"
}
simple-nifi-node-default-0 /app $ aws --profile msk-role sts get-caller-identity
{
"UserId": "AROAXX5AQFS2G2QEGKTKM:AssumeFromProfile",
"Account": "123456789012",
"Arn":
"arn:aws:sts::123456789012:assumed-role/nifi-node-msk/AssumeFromProfile"
} {code}
Use the name profile in the AmazonMSKConnectionService:
!image-2025-09-05-11-11-06-768.png|width=573,height=426!
Then test using PublishKafka:
!image-2025-09-05-11-13-03-732.png|width=572,height=386!
> Make use of AWSCredentialsProviderControllerService for
> AmazonMSKConnectionService
> ----------------------------------------------------------------------------------
>
> Key: NIFI-14927
> URL: https://issues.apache.org/jira/browse/NIFI-14927
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Nick
> Assignee: Pierre Villard
> Priority: Major
> Attachments: image-2025-09-01-21-50-48-663.png,
> image-2025-09-01-21-52-34-971.png, image-2025-09-02-11-21-31-147.png,
> image-2025-09-05-11-11-06-768.png, image-2025-09-05-11-13-03-732.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, the AmazonMSKConnectionService lacks the authentication settings
> that are available through the AWSCredentialsProviderControllerService on the
> other AWS providers (to allow things like Assume Role).
> !image-2025-09-01-21-50-48-663.png|width=614,height=461!
> Currently this means we need to set permissions on the AWS IAM Role that is
> mapped to the PodIdentity. Instead, we would rather configure each provider
> with the applicable IAM Role to be assumed from the PodIdentity Role.
> ListS3, AwsSecretsManagerParameterProvider (and others) allow a more
> versatile and expected configuration using the
> AWSCredentialsProviderControllerService:
> !image-2025-09-02-11-21-31-147.png|width=649,height=361!
> !image-2025-09-01-21-52-34-971.png|width=647,height=501!
> Can the same be applied to AmazonMSKConnectionService?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
