Hi, I don't have in-depth knowledge on AWS, but I can provide some guidance on enabling programmatic access. Consider creating an IAM user with the necessary roles and permissions. Once you've established the user, you can generate access keys for them and integrate them with your service.
IAM users are entities in AWS that can be specifically configured to have access to AWS resources, such as your OpenSearch cluster. For more detailed information on IAM roles and user creation, you can refer to the AWS documentation, which provides comprehensive guidance on setting up IAM roles and users, as well as best practices for access management. Regards, Lehel ________________________________ From: James McMahon <jsmcmah...@gmail.com> Sent: Sunday, October 22, 2023 18:34 To: lehe...@hotmail.com <lehe...@hotmail.com> Cc: users@nifi.apache.org <users@nifi.apache.org> Subject: Re: curl from ExecuteStreamCommand This looks promising, Lehel. Visiting my OpenSearch home page in the AWS dashboard, I do see that I have an IAM role associated with it. That role is AWSServiceRoleFoirAmazonOpenSearchService. I select that role, but don't see that I have an ACCESS_KEY or SECRET_KEY associated with it. Its Type is AWS Managed. Looking at the top of the dashboard for this role, it appears that the role has a REGION of Global. My OpenSearch Service has a REGION of US East (N. Virginia). When I created my OpenSearch service, I did so only with a master user name and master user password. Should I instead explicitly create access and secret keys for my Amazon OpenSearch Service? Can you say a few words regarding how I get to these keys? On Sat, Oct 21, 2023 at 9:51 PM Lehel Boér <lehe...@hotmail.com<mailto:lehe...@hotmail.com>> wrote: Hi James, I'm not sure if the username/password authentication is enough in this case. The AWS CLI automatically handles the authentication and authorization for you, using the credentials you have configured for your CLI. This looks like an authorization issue between curl and the AWS service. Curl supports --aws-sigv-4 requests which you can use with the access key of the IAM role set in OpenSearch. I managed to get it working for GET requests. https://how.wtf/aws-sigv4-requests-with-curl.html -XGET;https://$DOMAIN/$PATH;-H;'Content-Type: application/json';--user;$ACCES_KEY:$SECRET_KEY;--aws-sigv4;aws:amz:$REGION:es Kind Regards, Lehel ________________________________ From: James McMahon <jsmcmah...@gmail.com<mailto:jsmcmah...@gmail.com>> Sent: Saturday, October 21, 2023 20:25 To: users <users@nifi.apache.org<mailto:users@nifi.apache.org>> Subject: curl from ExecuteStreamCommand I have tested this curl from my ec2 command line: curl -XPUT -u 'myusernm:myuserpw' 'https://vpc-rampart-test-opensearch-nrqyb7jjpvmji6cp2qcvmyhcgq.us-east-1.es.amazonaws.com/movies/_doc/1' -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json' It successfully puts the json into my Amazon OpenSearch domain. That domain in the URL above is the Domain endpoint shown on the AWS dashboard for its OpenSearch service. In NiFi the JSON is my flowfile content. I am trying to get my ExecuteStreamCommand to run the curl, but it fails. NiFi indicates it gets back this: "message": "Request forbidden by administrative rules" This is how I have the processor configured. Command Arguments - -XINPUT;-u;myusernm:myuserpw;https://vpc-rampart-test-opensearch-nrqyb7jjpvmji6cp2qcvmyhcgq.us-east-1.es.amazonaws.com/movies/_doc/1;-H;'Content-Type: application/json' Command Path - /usr/bin/curl Ignore STDIN - false Argument Delimiter - ; Max Attribute Length - 256 How can this be configured in the ExecuteStreamCommand processor to run successfully? If the ExecuteStreamCommand executes the command just as if we were at the command line, what is getting in the way here when I try to run this from NiFi?
