Hi Otto, and thank you for your analysis. I am pretty sure though that both region and resource are correct and working (I have tested them with success with Postman), it must mean that I am doing something wrong in configuring the InvokeAWSGatewayApi processor, even if I can’t imagine what.
Anyway, I have found an alternative solution using an ExecuteProcess processor to send the API request call via a python script, based on this example here: https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html#sig-v4-examples-get-auth-header where I use the following parameters (plus Access Key and Secret Key): ________________________________ method = 'GET' service = 'execute-api' host = 'zsslnu1awf.execute-api.eu-west-1.amazonaws.com' region = 'eu-west-1' endpoint = 'https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/prod/test' request_parameters = '' ________________________________ Thank you, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<http://www.dedagroup.it/public-services> [cid:image001.jpg@01D76C19.550EE260]<http://www.dedagroup.it/home> Da: Otto Fowler <ottobackwa...@gmail.com> Inviato: venerdì 18 giugno 2021 20:50 A: users@nifi.apache.org Oggetto: Re: How to access to an Amazon API? Luca, I have a setup of the petstore sample from aws: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-from-example.html<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Fapigateway%2Flatest%2Fdeveloperguide%2Fapi-gateway-create-api-from-example.html&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649577210%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Z8GpXdMH5qJAxvhIGnHM7Antm7FJt7VQWapJil2m3jY%3D&reserved=0> The only way that I can reproduce your issue is if: a. the region is incorrect b. the resource is wrong If I have those correct, then I get the api response I expect Standard FlowFile Attributes Key: 'entryDate' Value: 'Fri Jun 18 14:45:57 EDT 2021' Key: 'lineageStartDate' Value: 'Fri Jun 18 14:45:57 EDT 2021' Key: 'fileSize' Value: '184' FlowFile Attribute Map Content Key: 'Access-Control-Allow-Origin' Value: '*' Key: 'Connection' Value: 'keep-alive' Key: 'Content-Length' Value: '184' Key: 'Content-Type' Value: 'application/json' Key: 'Date' Value: 'Fri, 18 Jun 2021 18:45:57 GMT' Key: 'X-Amzn-Trace-Id' Value: ' Key: 'aws.gateway.api.endpoint' Value: 'execute-api' Key: 'aws.gateway.api.resource' Value: '/pets' Key: 'aws.gateway.api.status.code' Value: '200' Key: 'aws.gateway.api.status.message' Value: 'OK' -------------------------------------------------- [ { "id": 1, "type": "dog", "price": 249.99 }, { "id": 2, "type": "cat", "price": 124.99 }, { "id": 3, "type": "fish", "price": 0.99 } ] On Jun 18, 2021, at 13:00, Luca Giovannini <luca.giovann...@dedagroup.it<mailto:luca.giovann...@dedagroup.it>> wrote: Thank you for your help, Otto! In the meantime I thought to get around the obstacle by trying to reach the API via a python script using the AWS SDK (to be then run by NiFi in an ExecuteScript processor), but that proved to be not at all straightforward and I am still struggling. The closest thing I found to a short example on how to do this is the following: https://gist.github.com/bjinwright/8c0d1e3a65017e1479d61e7dbbffb79f<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fbjinwright%2F8c0d1e3a65017e1479d61e7dbbffb79f&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649587161%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9socRI3zUoeBahx5wHfI4AVImIeou3ZWzo5R4NwBRN4%3D&reserved=0> I will try to adapt it to my case and hope it works. But this leaves me with a big question: I HAVE to be doing something wrong myself with the InvokeAWSGatewayApi… how can it be that nobody else has had my problem? It seems to me that I am using it for such a simple request! Nobody in the NiFi community has ever used it to access to a ‘execute-api’ resource in the ‘eu-west-1’ region? Really? Or am I not understanding the error that I am getting? Sorry, I am a little frustrated by this… any insight will be really appreciated! 😊 Thank you all very much, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649587161%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=m8ionceG9SObNDe%2FBd3XIb7EjlHim1E9SiEtse7uD6Q%3D&reserved=0> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649597122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1E03j4gzVRW16bY%2BxwJDauOXZPZHw92P8g%2F9Byswb8Q%3D&reserved=0> <image001.jpg><https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649597122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=m2tmHYfO%2BkjSWxvgY%2Fl6fzDxHXqCvU18j2Q7xEgFFTE%3D&reserved=0> Da: Otto Fowler <ottobackwa...@gmail.com<mailto:ottobackwa...@gmail.com>> Inviato: mercoledì 16 giugno 2021 21:37 A: users@nifi.apache.org<mailto:users@nifi.apache.org> Oggetto: Re: How to access to an Amazon API? I hope to be able to try it tomorrow. On Jun 16, 2021, at 09:01, Luca Giovannini <luca.giovann...@dedagroup.it<mailto:luca.giovann...@dedagroup.it>> wrote: Thank you Otto, please let me know if you need any additional information to set up the processor with my configuration! Just an additional element: I cloned the flow in a different machine with NiFi 1.11.4 on it and the same exact problem shows up (same error in logs, same absence of the raw API call in the provenance data). Thank you again for the help you might be able to provide me…I am really stuck here! ☹ Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649597122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Z405pFLiRtcbngJpnmD5rDHN%2BdC0Q7nNGR538IrTEwI%3D&reserved=0> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649607074%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FFeapugjEqdX8zzOqpxQEW2ny40UcP12XwV10a%2BpL88%3D&reserved=0> <image001.jpg><https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649607074%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=h%2B6Yrc1cgz2YUDuu7TopXvcLg16B268Ij%2FgG%2BVKcho4%3D&reserved=0> Da: Otto Fowler <ottobackwa...@gmail.com<mailto:ottobackwa...@gmail.com>> Inviato: martedì 15 giugno 2021 17:04 A: users@nifi.apache.org<mailto:users@nifi.apache.org> Oggetto: Re: How to access to an Amazon API? That error means that we passed info into AWS api ( asked the region to find the resource ) and it said nope, and tried to guess. I would have to try to setup the processor with your configuration ( enough to try to parse, ok if it fails later because of cred or whatever ). On Jun 15, 2021, at 06:26, Luca Giovannini <luca.giovann...@dedagroup.it<mailto:luca.giovann...@dedagroup.it>> wrote: Thank you Otto, I can confirm that the resource API is correct: I tested it successfully with Postman by doing a GET on the request URL (https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/prod/test<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzsslnu1awf.execute-api.eu-west-1.amazonaws.com%2Fprod%2Ftest&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649617032%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vWzZ3nHEVzPmIxw%2BdU0MlGxn2qPAIdItJEyf%2BqpfGxE%3D&reserved=0>) and providing AccessKey, SecretKey and AWS Region in the “Authorization” tab of the Postman client. I also checked log/nifi-app.log and the only informative line that I found is: 2021-06-15 11:59:24,046 INFO [Timer-Driven Process Thread-8] c.a.i.DefaultServiceEndpointBuilder {execute-api, eu-west-1} was not found in region metadata, trying to construct an endpoint using the standard pattern for this region: 'execute-api.eu-west-1.amazonaws.com<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexecute-api.eu-west-1.amazonaws.com%2F&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649617032%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XlxT%2B6Fz8uQQOxEHbOnZhUBoJB13r%2Bc18VZhkgValh8%3D&reserved=0>'. Which seems to indicate that the processor is not using the endpoint I provided as parameter (https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/)<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzsslnu1awf.execute-api.eu-west-1.amazonaws.com%2F)&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649626988%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=e%2FPCX2pIJ47ItbLSYv6zFaHULa6qeNLj6sUKrmBbjHk%3D&reserved=0>... I also checked the provenance data (both from the “Provenance” button of the output FlowFile listing and from the “Data Provenance” element of the main menu) for records of the call, but I could not find any provenance data at all for the processor InvokeAWSGatewayApi under scrutiny (my flow right now is made up just by that processor alone and a LogMessage one). Is this new information shedding any more light on the problem for you? Thank you very much, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649626988%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iLXTMpVfXrMBERGkioC1ublFiCFbXDJoe%2BYhonLhJ3c%3D&reserved=0> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649636941%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=c185woZh5ZvnS6nie%2B0gUCcS3ewT%2FyzJ4i6gsHH9QJA%3D&reserved=0> <image001.jpg><https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649636941%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NZYEawBfiCoL604T2LKyw0uXHqg3EQmo2L7aBy3V6vQ%3D&reserved=0> Da: Otto Fowler <ottobackwa...@gmail.com<mailto:ottobackwa...@gmail.com>> Inviato: lunedì 14 giugno 2021 19:48 A: users@nifi.apache.org<mailto:users@nifi.apache.org> Oggetto: Re: How to access to an Amazon API? **ATTENZIONE** Questo messaggio proviene da un ACCOUNT ESTERNO, presta attenzione ad eventuali link o allegati al suo interno. You should also confirm the resource api you are trying to call is correct. On Jun 14, 2021, at 12:04, Luca Giovannini <luca.giovann...@dedagroup.it<mailto:luca.giovann...@dedagroup.it>> wrote: Hello everybody! I have no experience with Amazon services but I now have to access to an API hosted on AWS. I have been given the following parameters: * HTTP Method GET * Amazon Region eu-west-1 * Access Key ID **** (hidden but known) * Secret Access Key **** (hidden but known) * Amazon Gateway Api ResourceName /test * Amazon Gateway Api Endpoint https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/xxxx<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzsslnu1awf.execute-api.eu-west-1.amazonaws.com%2Fxxxx&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649646905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Ur70MJ6sb1eTF3Ajxml3%2BwLrMjfr%2FTN27RVsjf3Qm9s%3D&reserved=0> (anonymized) I have used the “InvokeAWSGatewayApi” and set the previous 6 parameters (and only those), but I got {"message": "Internal server error"} as an answer. Maybe I did not format the ResourceName correctly (I see here [*] that the ResourceName is usually much longer)? I am using NiFi 1.12.1. Thank you very much for your help! Luca Giovannini [*] https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Fgeneral%2Flatest%2Fgr%2Faws-arns-and-namespaces.html&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649646905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7kivyUVIAZyZa9DUtfv9c3TZ4N%2Bi5%2BcqFAdzkBeNwNI%3D&reserved=0> Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649656855%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0uoOFqqYtl1tTfTNn9hQ0JIoQ4tlidtqcUjLiHrkJ3w%3D&reserved=0> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649656855%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EOG7KnAk3UGKBZcaUBw5jDlcCR3Jw3gZC7M%2BZ1kkv7Y%3D&reserved=0> <image001.jpg><https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d433336a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649666802%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MUVPwHblcrr07B%2B0x4PdEHLCw6t8sUbiglaWQKOX1%2Bw%3D&reserved=0> Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente. The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.
