R: How to see the request header created by InvokeHTTP?
Thank you so much, Kevin!! With that I could solve it in a minute! The issue was that InvokeHTTP was sending a "user-agent" header with an empty value and that caused the endpoint to issue and error message. By setting the "useragent" parameter of InvokeHTTP to a random non-null value (i.e. "none"), the endpoint responded instead with the expected answer. Thanks again, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<http://www.dedagroup.it/public-services> [cid:image001.jpg@01D83EBD.FA0D3420]<http://www.dedagroup.it/home> Da: Kevin Doran Inviato: mercoledì 2 marzo 2022 16:39 A: users@nifi.apache.org Oggetto: Re: How to see the request header created by InvokeHTTP? **ATTENZIONE** Questo messaggio proviene da un ACCOUNT ESTERNO, presta attenzione ad eventuali link o allegati al suo interno. Hi Luca, Based on looking at the code here [1], I believe you need to enable debug-level logging for this logger: org.apache.nifi.processors.standard.InvokeHTTP Instructions for changing log level for individual component loggers can be found here [2]. (Ignore the part about attaching a remote debugger if you are just interested in the logging part.) Basically, you need to add this line to your NiFi conf/logback.xml file: [1] https://github.com/apache/nifi/blob/4cc20e6c0687277394f021ca1f3ececd430c5a0e/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L1277<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fblob%2F4cc20e6c0687277394f021ca1f3ececd430c5a0e%2Fnifi-nar-bundles%2Fnifi-standard-bundle%2Fnifi-standard-processors%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fnifi%2Fprocessors%2Fstandard%2FInvokeHTTP.java%23L1277=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ce1a122e0f71b45a1c49908d9fc62c06d%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637818323354462607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=hWA12fczMaiCmi7xXiYQI1oyYzrz2SCcX8Rv%2Bj7iQbw%3D=0> [2] https://community.cloudera.com/t5/Community-Articles/NiFi-Debugging-Tutorial/ta-p/246082<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.cloudera.com%2Ft5%2FCommunity-Articles%2FNiFi-Debugging-Tutorial%2Fta-p%2F246082=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ce1a122e0f71b45a1c49908d9fc62c06d%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637818323354462607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=mOI4b0IRYfbrfuYVtbMVQNkl%2BgVhOvodiie5zNL7mss%3D=0> Hope this helps! Kevin On Mar 2, 2022 at 04:26:10, Luca Giovannini mailto:luca.giovann...@dedagroup.it>> wrote: Dear All, I am having troubles replicating with InvokeHTTP (1.12.1) a call to an HTTP endpoint that I am successfully able to make with Postman. I use InvokeHTTP with its default values and I just add two attributes for 'Authorization' and 'Host'. I would like to see the actual request header created by InvokeHTTP, so that I can compare it with the one created by Postman and try to have some insights on my problem from the comparison... So, how can I access the request header created by InvokeHTTP? I had a look in the nifi-app log but no detail on the call was provided. Thank you for your help! Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ce1a122e0f71b45a1c49908d9fc62c06d%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637818323354462607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=OlfQuGLwbYUy97U6%2F8%2BPdbmiq1tIKQwbNmgg4DjWnz8%3D=0> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ce1a122e0f71b45a1c49908d9fc62c06d%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637818323354462607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=xFLpV%2Bm7C6qrPcY5Nzs9lUM%2BCZdVxFhM%2FR4s4fuMDs0%3D=0> [cid:image001.jpg@01D83EBD.FA0D3420]<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ce1a122e0f71b45a1c49908d9fc62c06d%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637818323354462607%7CUnkn
How to see the request header created by InvokeHTTP?
Dear All, I am having troubles replicating with InvokeHTTP (1.12.1) a call to an HTTP endpoint that I am successfully able to make with Postman. I use InvokeHTTP with its default values and I just add two attributes for 'Authorization' and 'Host'. I would like to see the actual request header created by InvokeHTTP, so that I can compare it with the one created by Postman and try to have some insights on my problem from the comparison... So, how can I access the request header created by InvokeHTTP? I had a look in the nifi-app log but no detail on the call was provided. Thank you for your help! Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<http://www.dedagroup.it/public-services> [cid:image001.jpg@01D82E1C.8B4422D0]<http://www.dedagroup.it/home> Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente. The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.
R: How to access to an Amazon API?
Hi Otto,
and thank you for your analysis.
I am pretty sure though that both region and resource are correct and working
(I have tested them with success with Postman), it must mean that I am doing
something wrong in configuring the InvokeAWSGatewayApi processor, even if I
can’t imagine what.
Anyway, I have found an alternative solution using an ExecuteProcess processor
to send the API request call via a python script, based on this example here:
https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html#sig-v4-examples-get-auth-header
where I use the following parameters (plus Access Key and Secret Key):
method = 'GET'
service = 'execute-api'
host = 'zsslnu1awf.execute-api.eu-west-1.amazonaws.com'
region = 'eu-west-1'
endpoint = 'https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/prod/test'
request_parameters = ''
Thank you,
Luca
Luca Giovannini
Information Systems Analyst
Dedagroup Public Services
www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/>
T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67
www.dedagroup.it/public-services<http://www.dedagroup.it/public-services>
[cid:image001.jpg@01D76C19.550EE260]<http://www.dedagroup.it/home>
Da: Otto Fowler
Inviato: venerdì 18 giugno 2021 20:50
A: users@nifi.apache.org
Oggetto: Re: How to access to an Amazon API?
Luca,
I have a setup of the petstore sample from aws:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-from-example.html<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Fapigateway%2Flatest%2Fdeveloperguide%2Fapi-gateway-create-api-from-example.html=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d46a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649577210%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=Z8GpXdMH5qJAxvhIGnHM7Antm7FJt7VQWapJil2m3jY%3D=0>
The only way that I can reproduce your issue is if:
a. the region is incorrect
b. the resource is wrong
If I have those correct, then I get the api response I expect
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Fri Jun 18 14:45:57 EDT 2021'
Key: 'lineageStartDate'
Value: 'Fri Jun 18 14:45:57 EDT 2021'
Key: 'fileSize'
Value: '184'
FlowFile Attribute Map Content
Key: 'Access-Control-Allow-Origin'
Value: '*'
Key: 'Connection'
Value: 'keep-alive'
Key: 'Content-Length'
Value: '184'
Key: 'Content-Type'
Value: 'application/json'
Key: 'Date'
Value: 'Fri, 18 Jun 2021 18:45:57 GMT'
Key: 'X-Amzn-Trace-Id'
Value: '
Key: 'aws.gateway.api.endpoint'
Value: 'execute-api'
Key: 'aws.gateway.api.resource'
Value: '/pets'
Key: 'aws.gateway.api.status.code'
Value: '200'
Key: 'aws.gateway.api.status.message'
Value: 'OK'
--
[
{
"id": 1,
"type": "dog",
"price": 249.99
},
{
"id": 2,
"type": "cat",
"price": 124.99
},
{
"id": 3,
"type": "fish",
"price": 0.99
}
]
On Jun 18, 2021, at 13:00, Luca Giovannini
mailto:luca.giovann...@dedagroup.it>> wrote:
Thank you for your help, Otto!
In the meantime I thought to get around the obstacle by trying to reach the API
via a python script using the AWS SDK (to be then run by NiFi in an
ExecuteScript processor), but that proved to be not at all straightforward and
I am still struggling. The closest thing I found to a short example on how to
do this is the following:
https://gist.github.com/bjinwright/8c0d1e3a65017e1479d61e7dbbffb79f<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fbjinwright%2F8c0d1e3a65017e1479d61e7dbbffb79f=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cfac27c74738d46a108d9328a0365%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637596390649587161%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=9socRI3zUoeBahx5wHfI4AVImIeou3ZWzo5R4NwBRN4%3D=0>
I will try to adapt it to my case and hope it works.
But this leaves me with a big question:
I HAVE to be doing something wrong myself with the InvokeAWSGatewayApi… how can
it be that nobody else has had my problem?
It seems to me that I am using it for such a simple request! Nobody in the NiFi
community has ever used it to access to a ‘execute-api’ resource in the
‘eu-west-1’ region? Really?
Or am I not understanding the error that I am getting?
Sorry, I am a little frustrated by this… any insight will be really
appr
R: How to access to an Amazon API?
Thank you for your help, Otto! In the meantime I thought to get around the obstacle by trying to reach the API via a python script using the AWS SDK (to be then run by NiFi in an ExecuteScript processor), but that proved to be not at all straightforward and I am still struggling. The closest thing I found to a short example on how to do this is the following: https://gist.github.com/bjinwright/8c0d1e3a65017e1479d61e7dbbffb79f I will try to adapt it to my case and hope it works. But this leaves me with a big question: I HAVE to be doing something wrong myself with the InvokeAWSGatewayApi… how can it be that nobody else has had my problem? It seems to me that I am using it for such a simple request! Nobody in the NiFi community has ever used it to access to a ‘execute-api’ resource in the ‘eu-west-1’ region? Really? Or am I not understanding the error that I am getting? Sorry, I am a little frustrated by this… any insight will be really appreciated! Thank you all very much, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<http://www.dedagroup.it/public-services> [cid:image001.jpg@01D76473.2946C740]<http://www.dedagroup.it/home> Da: Otto Fowler Inviato: mercoledì 16 giugno 2021 21:37 A: users@nifi.apache.org Oggetto: Re: How to access to an Amazon API? I hope to be able to try it tomorrow. On Jun 16, 2021, at 09:01, Luca Giovannini mailto:luca.giovann...@dedagroup.it>> wrote: Thank you Otto, please let me know if you need any additional information to set up the processor with my configuration! Just an additional element: I cloned the flow in a different machine with NiFi 1.11.4 on it and the same exact problem shows up (same error in logs, same absence of the raw API call in the provenance data). Thank you again for the help you might be able to provide me…I am really stuck here! ☹ Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ccae5d160c8fa4e0f7e1208d930fe1be2%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637594690240328254%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=ddCKIhrWdvGZ91f4ZZM4B%2FM1WUfHLe6hXLN9%2Bik0kgc%3D=0> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ccae5d160c8fa4e0f7e1208d930fe1be2%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637594690240338212%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=U%2FHdfYaeuKonyAi2w2fPuqwHAUUu7bq8y4Ug2M4eAFE%3D=0> <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ccae5d160c8fa4e0f7e1208d930fe1be2%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637594690240338212%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=Pf6yTHHxPqi1BSK%2BNi7GhsOtzr5q5%2Bp0k1xFDiBbk30%3D=0> Da: Otto Fowler mailto:ottobackwa...@gmail.com>> Inviato: martedì 15 giugno 2021 17:04 A: users@nifi.apache.org<mailto:users@nifi.apache.org> Oggetto: Re: How to access to an Amazon API? That error means that we passed info into AWS api ( asked the region to find the resource ) and it said nope, and tried to guess. I would have to try to setup the processor with your configuration ( enough to try to parse, ok if it fails later because of cred or whatever ). On Jun 15, 2021, at 06:26, Luca Giovannini mailto:luca.giovann...@dedagroup.it>> wrote: Thank you Otto, I can confirm that the resource API is correct: I tested it successfully with Postman by doing a GET on the request URL (https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/prod/test<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzsslnu1awf.execute-api.eu-west-1.amazonaws.com%2Fprod%2Ftest=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ccae5d160c8fa4e0f7e1208d930fe1be2%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637594690240338212%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=73TmFt24FwsmDA4XItR3rNsRAzRWeUKkkjJ6oBNfBIU%3D=0>) and providing AccessKey, SecretKey and AWS Region in the “Authorization” tab of the Postman client. I also checked log/nifi-app.log and the only informative line that I fou
R: How to access to an Amazon API?
Thank you Otto,
please let me know if you need any additional information to set up the
processor with my configuration!
Just an additional element: I cloned the flow in a different machine with NiFi
1.11.4 on it and the same exact problem shows up (same error in logs, same
absence of the raw API call in the provenance data).
Thank you again for the help you might be able to provide me…I am really stuck
here! ☹
Luca
Luca Giovannini
Information Systems Analyst
Dedagroup Public Services
www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/>
T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67
www.dedagroup.it/public-services<http://www.dedagroup.it/public-services>
[cid:image001.jpg@01D762BE.880DC0F0]<http://www.dedagroup.it/home>
Da: Otto Fowler
Inviato: martedì 15 giugno 2021 17:04
A: users@nifi.apache.org
Oggetto: Re: How to access to an Amazon API?
That error means that we passed info into AWS api ( asked the region to find
the resource ) and it said nope, and tried to guess.
I would have to try to setup the processor with your configuration ( enough to
try to parse, ok if it fails later because of cred or whatever ).
On Jun 15, 2021, at 06:26, Luca Giovannini
mailto:luca.giovann...@dedagroup.it>> wrote:
Thank you Otto,
I can confirm that the resource API is correct: I tested it successfully with
Postman by doing a GET on the request URL
(https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/prod/test<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzsslnu1awf.execute-api.eu-west-1.amazonaws.com%2Fprod%2Ftest=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cd7c5327ecb2f4d502c3308d9300ecf9c%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637593662448072299%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=UgKYU8cuDkNR7KcUjOEJfmAOiYnlzxsltoQvACjWWBw%3D=0>)
and providing AccessKey, SecretKey and AWS Region in the “Authorization” tab
of the Postman client.
I also checked log/nifi-app.log and the only informative line that I found is:
2021-06-15 11:59:24,046 INFO [Timer-Driven Process Thread-8]
c.a.i.DefaultServiceEndpointBuilder {execute-api, eu-west-1} was not found in
region metadata, trying to construct an endpoint using the standard pattern for
this region:
'execute-api.eu-west-1.amazonaws.com<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexecute-api.eu-west-1.amazonaws.com%2F=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cd7c5327ecb2f4d502c3308d9300ecf9c%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637593662448072299%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=6Snijmjgzpk3iXb0LAla0kUe%2FXOgu3iWRaSgQVoY3js%3D=0>'.
Which seems to indicate that the processor is not using the endpoint I provided
as parameter
(https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/)<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzsslnu1awf.execute-api.eu-west-1.amazonaws.com%2F)=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cd7c5327ecb2f4d502c3308d9300ecf9c%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637593662448082260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=ogBUugUkELwI5%2FictxdlybPHmONqCs5rxcBdSNoB6T4%3D=0>...
I also checked the provenance data (both from the “Provenance” button of the
output FlowFile listing and from the “Data Provenance” element of the main
menu) for records of the call, but I could not find any provenance data at all
for the processor InvokeAWSGatewayApi under scrutiny (my flow right now is made
up just by that processor alone and a LogMessage one).
Is this new information shedding any more light on the problem for you?
Thank you very much,
Luca
Luca Giovannini
Information Systems Analyst
Dedagroup Public Services
www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cd7c5327ecb2f4d502c3308d9300ecf9c%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637593662448082260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=lyA%2FBjtUB7jJ2GrhS5ddEnIi2dGAqlFGXRrX3v3WIiM%3D=0>
T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67
www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cd7c5327ecb2f4d502c3308d9300ecf9c%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637593662448092220%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=UuwxvW8qUhkAf3jqfcLkzDTPz9F1KTUfkjOyQZc7NTw%3D=0>
<https://eur01.safelinks.protection.outl
R: How to access to an Amazon API?
Thank you Otto,
I can confirm that the resource API is correct: I tested it successfully with
Postman by doing a GET on the request URL
(https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/prod/test) and
providing AccessKey, SecretKey and AWS Region in the "Authorization" tab of the
Postman client.
I also checked log/nifi-app.log and the only informative line that I found is:
2021-06-15 11:59:24,046 INFO [Timer-Driven Process Thread-8]
c.a.i.DefaultServiceEndpointBuilder {execute-api, eu-west-1} was not found in
region metadata, trying to construct an endpoint using the standard pattern for
this region: 'execute-api.eu-west-1.amazonaws.com'.
Which seems to indicate that the processor is not using the endpoint I provided
as parameter (https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/)...
I also checked the provenance data (both from the "Provenance" button of the
output FlowFile listing and from the "Data Provenance" element of the main
menu) for records of the call, but I could not find any provenance data at all
for the processor InvokeAWSGatewayApi under scrutiny (my flow right now is made
up just by that processor alone and a LogMessage one).
Is this new information shedding any more light on the problem for you?
Thank you very much,
Luca
Luca Giovannini
Information Systems Analyst
Dedagroup Public Services
www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/>
T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67
www.dedagroup.it/public-services<http://www.dedagroup.it/public-services>
[cid:image001.jpg@01D761E0.29244390]<http://www.dedagroup.it/home>
Da: Otto Fowler
Inviato: lunedì 14 giugno 2021 19:48
A: users@nifi.apache.org
Oggetto: Re: How to access to an Amazon API?
**ATTENZIONE** Questo messaggio proviene da un ACCOUNT ESTERNO, presta
attenzione ad eventuali link o allegati al suo interno.
You should also confirm the resource api you are trying to call is correct.
On Jun 14, 2021, at 12:04, Luca Giovannini
mailto:luca.giovann...@dedagroup.it>> wrote:
Hello everybody!
I have no experience with Amazon services but I now have to access to an API
hosted on AWS.
I have been given the following parameters:
* HTTP Method GET
* Amazon Region eu-west-1
* Access Key ID
(hidden but known)
* Secret Access Key (hidden but
known)
* Amazon Gateway Api ResourceName/test
* Amazon Gateway Api Endpoint
https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzsslnu1awf.execute-api.eu-west-1.amazonaws.com%2F=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cf6a7b92cbcd14c47f89f08d92f5c8a62%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637592896770527023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=IQS9bBXxoooneM0UjHWU5ixemx32WXJ4GMgiSIaFKiU%3D=0>
(anonymized)
I have used the "InvokeAWSGatewayApi" and set the previous 6 parameters (and
only those), but I got {"message": "Internal server error"} as an answer.
Maybe I did not format the ResourceName correctly (I see here [*] that the
ResourceName is usually much longer)?
I am using NiFi 1.12.1.
Thank you very much for your help!
Luca Giovannini
[*]
https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Fgeneral%2Flatest%2Fgr%2Faws-arns-and-namespaces.html=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cf6a7b92cbcd14c47f89f08d92f5c8a62%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637592896770536966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=Dtf1zlPZU56NuLe0yq5Q73WJaJ5Po%2B3RDGzSBcDprxo%3D=0>
Luca Giovannini
Information Systems Analyst
Dedagroup Public Services
www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cf6a7b92cbcd14c47f89f08d92f5c8a62%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637592896770536966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=2xckDJtCVAY%2FgRxfYycfUU%2FxsbH9fILRvUy%2Bx0D9qkE%3D=0>
T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67
www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Cf
How to access to an Amazon API?
Hello everybody!
I have no experience with Amazon services but I now have to access to an API
hosted on AWS.
I have been given the following parameters:
* HTTP Method GET
* Amazon Region eu-west-1
* Access Key ID
(hidden but known)
* Secret Access Key (hidden but
known)
* Amazon Gateway Api ResourceName/test
* Amazon Gateway Api Endpoint
https://zsslnu1awf.execute-api.eu-west-1.amazonaws.com/ (anonymized)
I have used the "InvokeAWSGatewayApi" and set the previous 6 parameters (and
only those), but I got {"message": "Internal server error"} as an answer.
Maybe I did not format the ResourceName correctly (I see here [*] that the
ResourceName is usually much longer)?
I am using NiFi 1.12.1.
Thank you very much for your help!
Luca Giovannini
[*] https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html
Luca Giovannini
Information Systems Analyst
Dedagroup Public Services
www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/>
T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67
www.dedagroup.it/public-services<http://www.dedagroup.it/public-services>
[cid:image001.jpg@01D7613E.F2FB48A0]<http://www.dedagroup.it/home>
Le informazioni contenute in questo messaggio di posta elettronica sono
riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o
forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La
invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione
al mittente.
The information included in this e-mail and any attachments are confidential
and may also be privileged. If you are not the correct recipient, you are
kindly requested to notify the sender immediately, to cancel it and not to
disclose the contents to any other person.
R: Wait processor doesn't route expired flowfiles to 'expired' relationship
Thanks a lot, Koji, for your insight! I will definitely look into it and see if it solves my problem. Best regards, Luca -Messaggio originale- Da: Koji Kawamura Inviato: lunedì 21 dicembre 2020 03:50 A: users@nifi.apache.org Oggetto: Re: Wait processor doesn't route expired flowfiles to 'expired' relationship **ATTENZIONE** Questo messaggio proviene da un ACCOUNT ESTERNO, presta attenzione ad eventuali link o allegati al suo interno. Hi Luca, Sorry to hear that you are having an issue with Wait processor. By looking at the code and testing it locally, I couldn't find a cause of the issue nor reproduce it. However, theoretically such a situation can happen when there are too many queued FlowFiles in the connection in front of the Wait processor exceeding the processor's throughput. Especially where the Wait processor is busy processing the recently added FlowFiles and not being able to take care of older FlowFiles. By default, the order of processing queued FlowFiles is undefined. To eliminate one of the possible uncertainties, I'd recommend using OldestFlowFileFirstPrioritizer at the incoming connection, so that older FlowFiles can be processed first. Hope this helps, Koji On Thu, Dec 17, 2020 at 1:36 AM Luca Giovannini wrote: > > > > Hi All, > > > > I am using the Wait/Notify couple of processors, with a setting of the Wait > processor as follows: > > Wait mode = keep in the upstream connection Expiration duration = 10 > min Wait penalty duration = 30 sec > > > > I use the Wait/Notify in a split/merge situation where al the split parts > have to be processed before the Wait is released. > > Occasionally, one of the branches of the split fails or takes too much time > and that is taken care with an “expired” branch of the Wait processor. > > > > The flow runs well for days but then slowly the incoming connection to the > Wait processor starts to grow, and checking the “queued duration” and > “lineage duration” of the flowfiles in the queue list I see that they are > much older than the “Expiration duration” setting (days vs. 10 minutes). I > have also noticed that, when this happens, if I change the “expiration > duration” setting to a smaller value the Wait processor starts to clean up > the queue by routing the flowfiles to expired. I have seen this problem > happening with all sorts of different values for “Expiration duration” and > “Wait penalty”, the ones that I provided above are just an example. > > I am using NiFi 1.11.4. > > > > What can be happening? > > This unpredictable behaviour is blocking the whole flow and making the > Wait processor unusable for me ☹ > > > > Thank you very much for your help and support! > > > > Luca > > > > Le informazioni contenute in questo messaggio di posta elettronica sono > riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o > forma. Qualora Lei non fosse la persona destinataria del presente messaggio, > La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente > comunicazione al mittente. > > The information included in this e-mail and any attachments are confidential > and may also be privileged. If you are not the correct recipient, you are > kindly requested to notify the sender immediately, to cancel it and not to > disclose the contents to any other person.
Wait processor doesn't route expired flowfiles to 'expired' relationship
Hi All, I am using the Wait/Notify couple of processors, with a setting of the Wait processor as follows: * Wait mode = keep in the upstream connection * Expiration duration = 10 min * Wait penalty duration = 30 sec I use the Wait/Notify in a split/merge situation where al the split parts have to be processed before the Wait is released. Occasionally, one of the branches of the split fails or takes too much time and that is taken care with an “expired” branch of the Wait processor. The flow runs well for days but then slowly the incoming connection to the Wait processor starts to grow, and checking the “queued duration” and “lineage duration” of the flowfiles in the queue list I see that they are much older than the “Expiration duration” setting (days vs. 10 minutes). I have also noticed that, when this happens, if I change the “expiration duration” setting to a smaller value the Wait processor starts to clean up the queue by routing the flowfiles to expired. I have seen this problem happening with all sorts of different values for “Expiration duration” and “Wait penalty”, the ones that I provided above are just an example. I am using NiFi 1.11.4. What can be happening? This unpredictable behaviour is blocking the whole flow and making the Wait processor unusable for me ☹ Thank you very much for your help and support! Luca Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente. The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.
R: Consuming a dynamically variable number of AMQP queues
Thank you, Bryan!
Luca
Da: Bryan Bende
Inviato: giovedì 12 novembre 2020 17:14
A: users@nifi.apache.org
Oggetto: Re: Consuming a dynamically variable number of AMQP queues
**ATTENZIONE** Questo messaggio proviene da un ACCOUNT ESTERNO, presta
attenzione ad eventuali link o allegati al suo interno.
Hello,
This idea/problem has come up a few times. The issue is that if you allow
incoming connections with dynamic queue/topic on these "consume" processors,
then there is potentially an unbounded number of topics/queues to consume from.
How does the processor ever know to stop consuming from one of them?
I would suggest implementing your own custom processor which can make more
assumptions for your specific use case.
Another option would be to have some type of script that used NiFi's REST API
to add/remove ConsumeAMQP processors based on some external knowledge.
-Bryan
On Thu, Nov 12, 2020 at 5:56 AM Luca Giovannini
mailto:luca.giovann...@dedagroup.it>> wrote:
Hi All!
I have to implement a flow with NiFi that involves consuming different AMQP
queues whose number may change in time.
One of the requirements is that when a new queue is available (of an old one is
no more available) the NiFi flow adjusts to the new situation without the need
for any manual adjustments.
In order to implement this I thought of having a simple configuration table in
a postgres DB, listing the names of the active queues (that an operator would
update when needed), and then use NiFi to go check periodically the queue names
list and feed it parametrically to a single ConsumeAMQP processor.
However, I realized that my plan is not applicable because:
1. ConsumeAMQP does not allow incoming relationships
2. Its “Queue” property does not allow values in Expression Language (so
putting a variable value like ${queue_name} wouldn’t work, right?)
Do you have an alternative, doable approach in NiFi to suggest me?
Thank you very much,
Luca
Luca Giovannini
Information Systems Analyst
Dedagroup Public Services
www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F=04%7C01%7CLuca.Giovannini%40dedagroup.it%7C283d86afb05448c8fcbb08d887261276%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637407944884240018%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=XL71PpPWJNZkgaV9HXrta%2BlVVSGmBCGTTgw%2BEfx4%2FO4%3D=0>
T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67
www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services=04%7C01%7CLuca.Giovannini%40dedagroup.it%7C283d86afb05448c8fcbb08d887261276%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637407944884249969%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=yYuilwfCK%2BpxRa%2FhfZKnCy%2BFNJFS3q9Efl1xM9cKflk%3D=0>
[cid:image001.jpg@01D6BF30.132D0020]<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome=04%7C01%7CLuca.Giovannini%40dedagroup.it%7C283d86afb05448c8fcbb08d887261276%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637407944884249969%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=IgiOSfdM2X%2Fy0qcRH2Hzb7LemFeXfwrUei9yyTP0zAU%3D=0>
Le informazioni contenute in questo messaggio di posta elettronica sono
riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o
forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La
invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione
al mittente.
The information included in this e-mail and any attachments are confidential
and may also be privileged. If you are not the correct recipient, you are
kindly requested to notify the sender immediately, to cancel it and not to
disclose the contents to any other person.
Consuming a dynamically variable number of AMQP queues
Hi All!
I have to implement a flow with NiFi that involves consuming different AMQP
queues whose number may change in time.
One of the requirements is that when a new queue is available (of an old one is
no more available) the NiFi flow adjusts to the new situation without the need
for any manual adjustments.
In order to implement this I thought of having a simple configuration table in
a postgres DB, listing the names of the active queues (that an operator would
update when needed), and then use NiFi to go check periodically the queue names
list and feed it parametrically to a single ConsumeAMQP processor.
However, I realized that my plan is not applicable because:
1. ConsumeAMQP does not allow incoming relationships
2. Its "Queue" property does not allow values in Expression Language (so
putting a variable value like ${queue_name} wouldn't work, right?)
Do you have an alternative, doable approach in NiFi to suggest me?
Thank you very much,
Luca
Luca Giovannini
Information Systems Analyst
Dedagroup Public Services
www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/>
T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67
www.dedagroup.it/public-services<http://www.dedagroup.it/public-services>
[cid:image001.jpg@01D6B8E7.9C7E08F0]<http://www.dedagroup.it/home>
Le informazioni contenute in questo messaggio di posta elettronica sono
riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o
forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La
invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione
al mittente.
The information included in this e-mail and any attachments are confidential
and may also be privileged. If you are not the correct recipient, you are
kindly requested to notify the sender immediately, to cancel it and not to
disclose the contents to any other person.
R: Error with FetchFTP when filename has non-ASCII charachters
Thank you, Matt! As you suggested I opened this improvement Jira: https://issues.apache.org/jira/browse/NIFI-7685 In the meantime I wrote a python script to fetch the files with non-ASCII names and used it with an ExecuteScript processor. Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<http://www.dedagroup.it/public-services> [cid:image001.jpg@01D6658C.B31D5880]<http://www.dedagroup.it/home> Da: Matt Burgess Inviato: martedì 14 luglio 2020 16:19 A: users@nifi.apache.org Oggetto: Re: Error with FetchFTP when filename has non-ASCII charachters **ATTENZIONE** Questo messaggio proviene da un ACCOUNT ESTERNO, presta attenzione ad eventuali link o allegati al suo interno. Luca, I'm guessing the issue is the same as the one in [1] but it just wasn't fixed for FetchFTP. Please feel free to write an improvement Jira [2] to add this to FetchFTP as well. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-4137<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-4137=02%7C01%7CLuca.Giovannini%40dedagroup.it%7C35da53a2e1db4aadb69a08d82800df73%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637303331502181485=ETqdW2puaeZKwDl1cr4raz8oll29qGFSN6fqVKO9%2FI4%3D=0> [2] https://issues.apache.org/jira/browse/NIFI<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI=02%7C01%7CLuca.Giovannini%40dedagroup.it%7C35da53a2e1db4aadb69a08d82800df73%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637303331502181485=FQGrhPEAmCUAtJGqGi2OsKP4xWdIeiuR6qTVfX0Qd6c%3D=0> On Tue, Jul 14, 2020 at 9:50 AM Luca Giovannini mailto:luca.giovann...@dedagroup.it>> wrote: When FetchFTP tries to fetch a file with name containing non-ASCII characters (e.g. “Xml_APE001_unitàimmobiliareguarda.xml”), the process succeeds but the corresponding flowfile is empty. Is there a way to make the FetchFTP processor succeed for real and really fetch the file? Thank you very much, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F=02%7C01%7CLuca.Giovannini%40dedagroup.it%7C35da53a2e1db4aadb69a08d82800df73%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637303331502191442=L9EPR6hB19XnxAIVOdV3gd2WiQO2jQO1e8DShQqwIvo%3D=0> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services=02%7C01%7CLuca.Giovannini%40dedagroup.it%7C35da53a2e1db4aadb69a08d82800df73%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637303331502191442=ijycbIjpmc%2F9DSel0ktLh9MiYQrk32RjSOxS0iNeDwY%3D=0> [cid:image001.jpg@01D6658C.B31D5880]<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome=02%7C01%7CLuca.Giovannini%40dedagroup.it%7C35da53a2e1db4aadb69a08d82800df73%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637303331502191442=lKVa6NH0XXek86iA9HeMXhq9p9HBXJzA6kpFB4AEZWA%3D=0> Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente. The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.
Error with FetchFTP when filename has non-ASCII charachters
When FetchFTP tries to fetch a file with name containing non-ASCII characters (e.g. "Xml_APE001_unitàimmobiliareguarda.xml"), the process succeeds but the corresponding flowfile is empty. Is there a way to make the FetchFTP processor succeed for real and really fetch the file? Thank you very much, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<http://www.dedagroup.it/public-services> [cid:image001.jpg@01D659F6.7F3A1C70]<http://www.dedagroup.it/home> Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente. The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.
R: Triggered fetching of multiple files from FTP - how?
Thank you, Harald. I ended up solving my problem with an ExecuteScript in Python that does the listing on demand and can be triggered by a flowfile. Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<http://www.dedagroup.it/public-services> [cid:image001.jpg@01D659F5.BF00AB40]<http://www.dedagroup.it/home> Da: Dobbernack, Harald (Key-Work) Inviato: giovedì 9 luglio 2020 16:00 A: users@nifi.apache.org Oggetto: AW: Triggered fetching of multiple files from FTP - how? Hi, you can use a listftp to know the filenames and then trigger fetchfile via wait processor as soon as it's ok to fetch them Kind Regards harald Von: Luca Giovannini mailto:luca.giovann...@dedagroup.it>> Gesendet: Donnerstag, 9. Juli 2020 15:25 An: users@nifi.apache.org<mailto:users@nifi.apache.org> Betreff: Triggered fetching of multiple files from FTP - how? Hi All, I am writing for ask for ideas and suggestion on this topic: I have to fetch all the .XML files that are present in a given FTP folder (and subfolders) and I have to fetch them only if a certain event happens. Therefore I cannot use the GetFTP processor because it doesn't allow to be triggered by an incoming flowfile. And I cannot use the FetchFTP either because I don't know the names of the files. Any suggestions on how else I can do it? Thank you very much, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F=02%7C01%7CLuca.Giovannini%40dedagroup.it%7Cab70cc4c266846c2a53c08d824107088%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637299000318534301=QCduzTt2qFzqZHQ7LRRuJhM61CSo3Pqh3hgbgJkhrOo%3D=0> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fpublic-services=02%7C01%7CLuca.Giovannini%40dedagroup.it%7Cab70cc4c266846c2a53c08d824107088%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637299000318544295=1T3o0ZH9%2B1gRS7M0RCKbjTwQH6s2w5iaxodbInF9m%2F0%3D=0> [cid:image001.jpg@01D659F5.BF00AB40]<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dedagroup.it%2Fhome=02%7C01%7CLuca.Giovannini%40dedagroup.it%7Cab70cc4c266846c2a53c08d824107088%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637299000318544295=zPHWjGafWdmQG0%2FfT8sE3AM6ASFlBaLVR63lyTcHNrI%3D=0> Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente. The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person. Harald Dobbernack Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | www.key-work.de<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.key-work.de%2F=02%7C01%7CLuca.Giovannini%40dedagroup.it%7Cab70cc4c266846c2a53c08d824107088%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637299000318554290=VwUIuT091yWWW%2FiRDwaNRttFArk4M9sjn2%2B%2BuWR2%2Big%3D=0> | Datenschutz<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.key-work.de%2Fde%2Ffooter%2Fdatenschutz.html=02%7C01%7CLuca.Giovannini%40dedagroup.it%7Cab70cc4c266846c2a53c08d824107088%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637299000318554290=2wu4TiTebjKdqRU07rUk0aq5Z3P%2BXwKymhJbDclT6%2F0%3D=0> Fon: +49-721-78203-264 | E-Mail: harald.dobbern...@key-work.de<mailto:harald.dobbern...@key-work.de> | Fax: +49-721-78203-10 Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim Geschäftsführer: Andreas Stappert, Tobin Wotring
Triggered fetching of multiple files from FTP - how?
Hi All, I am writing for ask for ideas and suggestion on this topic: I have to fetch all the .XML files that are present in a given FTP folder (and subfolders) and I have to fetch them only if a certain event happens. Therefore I cannot use the GetFTP processor because it doesn't allow to be triggered by an incoming flowfile. And I cannot use the FetchFTP either because I don't know the names of the files. Any suggestions on how else I can do it? Thank you very much, Luca Luca Giovannini Information Systems Analyst Dedagroup Public Services www.linkedin.com/in/lucagio/<http://www.linkedin.com/in/lucagio/> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128 Dedagroup Public Services Srl - Sede di Casalecchio di Reno, Via del Lavoro 67 www.dedagroup.it/public-services<http://www.dedagroup.it/public-services> [cid:image001.jpg@01D65603.8A5D76C0]<http://www.dedagroup.it/home> Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente. The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.
