PutAzureEventHub Errors in NiFi 1.1.0
Hello All: I tested the PutAzureEventHub processor in NiFi 1.1.0 and kept getting the following errors: 2016-12-09 10:41:05,435 WARN [StandardProcessScheduler Thread-8] o.a.n.controller.StandardProcessorNode Timed out while waiting for OnScheduled of 'PutAzureEventHub' processor to finish. An attempt is made to cancel the task via Thread.interrupt(). However it does not guarantee that the task will be canceled since the code inside current OnScheduled operation may have been written to ignore interrupts which may result in a runaway thread. This could lead to more issues, eventually requiring NiFi to be restarted. This is usually a bug in the target Processor 'PutAzureEventHub[id=4d6fe16b-b252-48de-4fd1-88ed63c335ea]' that needs to be documented, reported and eventually fixed. 2016-12-09 10:41:05,435 ERROR [StandardProcessScheduler Thread-4] o.a.n.p.azure.eventhub.PutAzureEventHub PutAzureEventHub[id=4d6fe16b-b252-48de-4fd1-88ed63c335ea] Failed to create EventHubClient due to java.lang.InterruptedException 2016-12-09 10:41:05,435 ERROR [StandardProcessScheduler Thread-8] o.a.n.p.azure.eventhub.PutAzureEventHub PutAzureEventHub[id=4d6fe16b-b252-48de-4fd1-88ed63c335ea] PutAzureEventHub[id=4d6fe16b-b252-48de-4fd1-88ed63c335ea] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Timed out while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Timed out while executing one of processor's OnScheduled task. 2016-12-09 10:41:05,436 ERROR [StandardProcessScheduler Thread-4] o.a.n.p.azure.eventhub.PutAzureEventHub java.lang.InterruptedException: null at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) ~[na:1.8.0_74] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[na:1.8.0_74] at org.apache.nifi.processors.azure.eventhub.PutAzureEventHub.createEventHubClient(PutAzureEventHub.java:185) [nifi-azure-processors-1.1.0.jar:1.1.0] at org.apache.nifi.processors.azure.eventhub.PutAzureEventHub.setupClient(PutAzureEventHub.java:141) [nifi-azure-processors-1.1.0.jar:1.1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_74] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_74] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_74] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_74] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) [nifi-framework-core-1.1.0.jar:1.1.0] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) [nifi-framework-core-1.1.0.jar:1.1.0] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) [nifi-framework-core-1.1.0.jar:1.1.0] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) [nifi-framework-core-1.1.0.jar:1.1.0] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1279) [nifi-framework-core-1.1.0.jar:1.1.0] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1275) [nifi-framework-core-1.1.0.jar:1.1.0] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_74] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_74] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_74] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] 2016-12-09 10:41:05,436 ERROR [StandardProcessScheduler Thread-8] o.a.n.p.azure.eventhub.PutAzureEventHub java.lang.RuntimeException: Timed out while executing one of processor's OnScheduled task. at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1447) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode.access$100(StandardProcessorNode.java:100) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1275) ~[na:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_74] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_74] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_74] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThrea
Re: NiFi Supports for Kerberized Kafka Cluster
Thank you, Oleg. I'll be testing it out once it is available next week. Regards, Indus On Thu, Apr 7, 2016 at 10:48 AM, Oleg Zhurakousky < ozhurakou...@hortonworks.com> wrote: > Not at the moment unless you are using HDP distribution of Kafka broker > 0.8 which has Kerberos support while ASF version of 0.8 Kafka doesn't. > That said, we are working on new pair of Kafka processors that will rely > on Kafka 0.9 API and broker where such support is provided. It is slated > for 0.7 release, but it should be available in master some time next week > if you'd like to try NIFI-1296. > > Cheers > Oleg > > > > > Sent from my iPhone > > > On Apr 7, 2016, at 11:39, indus well wrote: > > > > Hello All: > > > > Does current version of NiFi support kerberized Kafka cluster? I am > getting time-out errors trying to use PutKafka and GetKafka processors. > > > > Please advise. > > > > Thanks, > > > > Indus >
NiFi Supports for Kerberized Kafka Cluster
Hello All: Does current version of NiFi support kerberized Kafka cluster? I am getting time-out errors trying to use PutKafka and GetKafka processors. Please advise. Thanks, Indus
PutKafka Processor Time-out Errors with Guarantee Replicated Delivery on NiFi 0.5.1 and Kafka 0.8.2
Hello NiFi Experts: I am getting time-out errors from the PutKafka processor when using the Guarantee Replicated Delivery option in the Guarantee Delivery property on NiFi 0.5.1 with Kafka 0.8.2 cluster. However, everything is working as normal when I switched to the Best Effort option. In addition, NiFi 0.4.1 works just fine with either option. Here's the error: 19:33:02 CDTERROR478f6a4f-4b79-4253-afe4-ecda8174df67 PutKafka[id=478f6a4f-4b79-4253-afe4-ecda8174df67] Failed to send StandardFlowFileRecord[uuid=ebcd-5220-45c0-a713-f3d68fac633f,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1458692540256-122, container=default, section=122], offset=110349, length=122],offset=0,name=1033351446389479,size=122] to Kafka; routing to 'failure'; last failure reason reported was org.apache.kafka.common.errors.TimeoutException: Batch Expired;: org.apache.kafka.common.errors.TimeoutException: Batch Expired Is this issue related to Joe's comment in the following thread: http://mail-archives.apache.org/mod_mbox/nifi-users/201603.mbox/%3CCALJK9a4h%3D_A-LZPsAvWaAKcGu3qR78fWh2572gDh53Z9LeuLFQ%40mail.gmail.com%3E Please advise. Thanks, Indus
Re: Binary Build for nifi-0.5.0-RC3
Excellent! Thank you very much, Tony. On Mon, Feb 15, 2016 at 9:42 PM, Tony Kurc wrote: > The binaries are staged here: > https://dist.apache.org/repos/dist/dev/nifi/nifi-0.5.0/ > > After the release, they will be moved and available via the web page. > > On Mon, Feb 15, 2016 at 10:31 PM, indus well wrote: > >> Hello NiFi: >> >> I have troubles building nifi-0.5.0-RC3 from source. Is there a build >> available for me to download? Please help. >> >> Thanks, >> >> Indus >> > >
Binary Build for nifi-0.5.0-RC3
Hello NiFi: I have troubles building nifi-0.5.0-RC3 from source. Is there a build available for me to download? Please help. Thanks, Indus
ExecuteSQL Processor Usage
Hi NiFiers: Is there a sample data flow to show the usage of the ExecuteSQL processor? I can connect to the database and execute the SQL SELECT query and I can see the executesql.row.count value, but I don't know how to get the data content returned from the query. Please help. Thanks, Indus
Re: InvokeHTTP Content Type
I am currently running on 0.4.0-SNAPSHOT which I downloaded and built on 11/30. Would it include your change? On Fri, Dec 4, 2015 at 3:30 PM, Joe Percivall wrote: > Sorry, what version are you running? Adding the property like that is > something I added for 0.4.0. > > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com > > > > > On Friday, December 4, 2015 4:27 PM, indus well > wrote: > > > > Yes, I passed a header with key "Accept" and value "application/json". I > followed your instruction and add a new property in the InvokeHTTP process > called "Accept" and value of "application/json", but I now had an error > said "'Accept' validated against 'application/json' is invalid because > 'Accept' is not a supported property." > > Did I do something wrong? > > Thanks, > > Indus > > > On Fri, Dec 4, 2015 at 3:12 PM, Joe Percivall > wrote: > > I see, as for the request I'm guessing you mean you pass a header with key > "Accept" and value "application/json"? If so in the configuration tab for > the processor properties create a new dynamic one (using the plus in the > top right) with a key of "Accept" and value of "application/json". > > > >You use the "Attributes to Send" property when you already have the > header value as an attribute of the flowfile and want to send an attribute > with that name and it's value. > > > >Joe > > > >- - - - - - > >Joseph Percivall > >linkedin.com/in/Percivall > >e: joeperciv...@yahoo.com > > > > > > > > > > > >On Friday, December 4, 2015 4:06 PM, indus well > wrote: > > > > > > > >Thanks, Joe. My use case involves calling a REST service that returns XML > response by default. However, it also has option for JSON which I can pass > "Accept: application/json" along with the request using CURL for example. I > tried to set that on the "Attributes to Send" property in the InvokeHTTP > processor, but it did not work. It always returns XML. > > > >Thanks, > > > >Indus > > > > > >On Fri, Dec 4, 2015 at 2:49 PM, Joe Percivall > wrote: > > > >Hello Indus, > >> > >>Are you receiving a specific error when you attempt to receive a JSON > response? > >> > >>I can configure an InvokeHTTP processor to GET " > http://api.randomuser.me/"; by just setting the URL and it properly > receives back a JSON response. > >> > >>Joe > >> > >> > >>- - - - - - > >>Joseph Percivall > >>linkedin.com/in/Percivall > >>e: joeperciv...@yahoo.com > >> > >> > >> > >> > >> > >>On Friday, December 4, 2015 3:27 PM, indus well > wrote: > >> > >> > >> > >>Hi All: > >> > >>How would I configure the InvokeHTTP processor to receive content type > of JSON response? > >> > >>Thanks, > >> > >>Indus Well > >> > > >
Re: InvokeHTTP Content Type
Yes, I passed a header with key "Accept" and value "application/json". I followed your instruction and add a new property in the InvokeHTTP process called "Accept" and value of "application/json", but I now had an error said "'Accept' validated against 'application/json' is invalid because 'Accept' is not a supported property." Did I do something wrong? Thanks, Indus On Fri, Dec 4, 2015 at 3:12 PM, Joe Percivall wrote: > I see, as for the request I'm guessing you mean you pass a header with key > "Accept" and value "application/json"? If so in the configuration tab for > the processor properties create a new dynamic one (using the plus in the > top right) with a key of "Accept" and value of "application/json". > > You use the "Attributes to Send" property when you already have the header > value as an attribute of the flowfile and want to send an attribute with > that name and it's value. > > Joe > > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com > > > > > On Friday, December 4, 2015 4:06 PM, indus well > wrote: > > > > Thanks, Joe. My use case involves calling a REST service that returns XML > response by default. However, it also has option for JSON which I can pass > "Accept: application/json" along with the request using CURL for example. I > tried to set that on the "Attributes to Send" property in the InvokeHTTP > processor, but it did not work. It always returns XML. > > Thanks, > > Indus > > > On Fri, Dec 4, 2015 at 2:49 PM, Joe Percivall > wrote: > > Hello Indus, > > > >Are you receiving a specific error when you attempt to receive a JSON > response? > > > >I can configure an InvokeHTTP processor to GET "http://api.randomuser.me/"; > by just setting the URL and it properly receives back a JSON response. > > > >Joe > > > > > >- - - - - - > >Joseph Percivall > >linkedin.com/in/Percivall > >e: joeperciv...@yahoo.com > > > > > > > > > > > >On Friday, December 4, 2015 3:27 PM, indus well > wrote: > > > > > > > >Hi All: > > > >How would I configure the InvokeHTTP processor to receive content type of > JSON response? > > > >Thanks, > > > >Indus Well > > >
Re: InvokeHTTP Content Type
Thanks, Joe. My use case involves calling a REST service that returns XML response by default. However, it also has option for JSON which I can pass "Accept: application/json" along with the request using CURL for example. I tried to set that on the "Attributes to Send" property in the InvokeHTTP processor, but it did not work. It always returns XML. Thanks, Indus On Fri, Dec 4, 2015 at 2:49 PM, Joe Percivall wrote: > Hello Indus, > > Are you receiving a specific error when you attempt to receive a JSON > response? > > I can configure an InvokeHTTP processor to GET "http://api.randomuser.me/"; > by just setting the URL and it properly receives back a JSON response. > > Joe > > > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com > > > > > On Friday, December 4, 2015 3:27 PM, indus well > wrote: > > > > Hi All: > > How would I configure the InvokeHTTP processor to receive content type of > JSON response? > > Thanks, > > Indus Well >
InvokeHTTP Content Type
Hi All: How would I configure the InvokeHTTP processor to receive content type of JSON response? Thanks, Indus Well
Re: Database Lookup Approach
Thanks, Joe. Yes, the referenced lookup data does not change frequently so your built-in caching strategy sounds great. Would you expand on how to implement the built-in caching mechanism as well as refreshing the cache in NiFi please? Thanks, Indus On Sat, Nov 14, 2015 at 12:24 PM, Joe Witt wrote: > Indus, > > Let's project out one order of magnitude. That puts you at just a bit > more than 500 lookups per second. You will want to consider the > properties of the data itself and the properties of the database. > This helps you decide on the appropriate caching and querying > strategy. If the data as-is lends itself to highly effective caching > behavior then you're good. If not you may choose to intentionally > merge/batch data together to gain higher cache hits. With regard to > the database itself if it is being updated frequently then you'll need > a tighter cache refresh policy and so on. Anyway such cases and > fairly wide combination of scenarios have been played out with great > effect in NiFi. We have some of the tools you'll need out of the box > to do this and some you'll possibly need to build for your database or > caching strategy or if you want to edit the JSON data based on the > results of the database lookup. > > So general flow logic would be like this (not specific to any existing > processor) > > 1) Consume JSON object > 2) Extract username from JSON object as a flow file attribute > 3) Execute SQL statement to lookup results of query with username. > This could be a single processor which has a built-in caching > mechanism or a processor that first looks up in a cache then if misses > looks up against the actual database and then populates the cache. > 4) The attributes of the flow file now contain the results of the > query. Update the JSON object content if necessary using attributes > or just make an attribute based routing decision > > The key part of this is really 3. Many ways to go after that. > > Thanks > Joe > > On Sat, Nov 14, 2015 at 12:46 PM, indus well wrote: > > Hi NiFi Experts: > > > > I have a use case where I consume a Json message containing an userID and > > need to lookup the value from a database table. The transaction volume is > > averaging around 5 million per day and growing. What would be the best > > approach for this scenario. > > > > Thanks, > > > > Indus > > > > >
Fwd: Database Lookup Approach
Hi NiFi Experts: I have a use case where I consume a Json message containing an userID and need to lookup the value from a database table. The transaction volume is averaging around 5 million per day and growing. What would be the best approach for this scenario. Thanks, Indus
Fwd: GetKafka Processor Issue
Hello NiFi Team: This issue keeps happening. Please see the attached log file for the full stack dump. Thanks, Indus On Sun, Oct 4, 2015 at 11:31 PM, indus well wrote: > Thanks, Joe. Not sure what happened, but it appeared to be working when I > turned both on. I will keep monitoring the flows and will get the stack > dump for you if it happens again. > > On Sun, Oct 4, 2015 at 9:34 PM, Joe Witt wrote: > >> Indus, >> >> Can you please have them both running to get it back into the bad >> state then run /bin/nifi.sh dump >> >> This will write out the stack dump to the application log. Can you >> send us this? The stack trace could be quite telling. >> >> Thanks >> Joe >> >> On Sun, Oct 4, 2015 at 10:30 PM, indus well wrote: >> > Hello NiFi Experts: >> > >> > I have two workflows using GetKafka processors to consume two different >> > topics from a single node Kafka server and encountering an issue. If I >> turn >> > on both workflows, no message from the two topics would flow through. >> All >> > statuses show 0 byte or no activity. However, as soon as I stop one >> > workflow, messages would start streaming in. So I currently can only >> consume >> > one Kafka topic at a time. >> > >> > Have you seen this issue? Am I missing something in the settings? Please >> > advise. >> > >> > Thanks, >> > >> > Indus Well >> > >> > > nifi-bootstrap.log Description: Binary data
Re: GetKafka Processor Issue
Joe, it finally happened again after I turned off and turned on one of the GetKafka processor. Please review the attached stack dump file and let me know if you have any questions. Thanks, Indus On Sun, Oct 4, 2015 at 11:31 PM, indus well wrote: > Thanks, Joe. Not sure what happened, but it appeared to be working when I > turned both on. I will keep monitoring the flows and will get the stack > dump for you if it happens again. > > On Sun, Oct 4, 2015 at 9:34 PM, Joe Witt wrote: > >> Indus, >> >> Can you please have them both running to get it back into the bad >> state then run /bin/nifi.sh dump >> >> This will write out the stack dump to the application log. Can you >> send us this? The stack trace could be quite telling. >> >> Thanks >> Joe >> >> On Sun, Oct 4, 2015 at 10:30 PM, indus well wrote: >> > Hello NiFi Experts: >> > >> > I have two workflows using GetKafka processors to consume two different >> > topics from a single node Kafka server and encountering an issue. If I >> turn >> > on both workflows, no message from the two topics would flow through. >> All >> > statuses show 0 byte or no activity. However, as soon as I stop one >> > workflow, messages would start streaming in. So I currently can only >> consume >> > one Kafka topic at a time. >> > >> > Have you seen this issue? Am I missing something in the settings? Please >> > advise. >> > >> > Thanks, >> > >> > Indus Well >> > >> > > nifi-bootstrap.log Description: Binary data
Re: GetKafka Processor Issue
Thanks, Joe. Not sure what happened, but it appeared to be working when I turned both on. I will keep monitoring the flows and will get the stack dump for you if it happens again. On Sun, Oct 4, 2015 at 9:34 PM, Joe Witt wrote: > Indus, > > Can you please have them both running to get it back into the bad > state then run /bin/nifi.sh dump > > This will write out the stack dump to the application log. Can you > send us this? The stack trace could be quite telling. > > Thanks > Joe > > On Sun, Oct 4, 2015 at 10:30 PM, indus well wrote: > > Hello NiFi Experts: > > > > I have two workflows using GetKafka processors to consume two different > > topics from a single node Kafka server and encountering an issue. If I > turn > > on both workflows, no message from the two topics would flow through. All > > statuses show 0 byte or no activity. However, as soon as I stop one > > workflow, messages would start streaming in. So I currently can only > consume > > one Kafka topic at a time. > > > > Have you seen this issue? Am I missing something in the settings? Please > > advise. > > > > Thanks, > > > > Indus Well > > >
GetKafka Processor Issue
Hello NiFi Experts: I have two workflows using GetKafka processors to consume two different topics from a single node Kafka server and encountering an issue. If I turn on both workflows, no message from the two topics would flow through. All statuses show 0 byte or no activity. However, as soon as I stop one workflow, messages would start streaming in. So I currently can only consume one Kafka topic at a time. Have you seen this issue? Am I missing something in the settings? Please advise. Thanks, Indus Well
Re: GetHTTP Usage with Dynamic Filenames
Thanks, Joe. That did it, exactly what I was looking for.
You guys and NiFi rock!
On Wed, Sep 23, 2015 at 10:43 PM, Joe Witt wrote:
> Indus,
>
> We should probably have another JIRA to have the InvokeHTTP processor
> capture response headers and store them as flow file attributes or at
> least as a single attribute with the headers on it - something.
> However, the core need you have is totally doable. I'd recommend the
> following flow:
>
> GenerateFlowFile -> UpdateAttribute -> InvokeHTTP -> go do great
> things with the content
>
> In UpdateAttribute create a property called 'filename' and give it a value
> of
> ${now():format('MMdd')}.zip
>
> Then in the invoke HTTP use that for the expression language in the URL:
> http://example.com/${filename}
>
> Now you'll get your content and have retained the filename you used to
> pull it.
>
> Thanks
> Joe
>
> On Wed, Sep 23, 2015 at 8:29 PM, indus well wrote:
> > Worked like a charm! Thank you for your quick response, Daryl and Mark.
> >
> > The content of the downloaded file is stored in the file generated
> FlowFile
> > with a random filename, how would I rename the output file to actual
> > downloaded filename?
> >
> > By the way, the ticket is a good enhancement.
> >
> > Thanks,
> >
> > Indus
> >
> > On Wed, Sep 23, 2015 at 8:27 PM, Mark Payne
> wrote:
> >>
> >> Indus,
> >>
> >> As Daryl mentioned, InvokeHTTP will allow you to use the Expression
> >> Language to do an HTTP GET.
> >> It works a bit differently, though, because GetHTTP is a "Source
> >> Processor" whereas InvokeHTTP needs to
> >> be fed a FlowFile to do anything. So you can use GenerateFlowFile as a
> >> source and have it generate a
> >> 0 byte FlowFile (set the File Size property to "0 B"). Then just connect
> >> GenerateFlowFile to InvokeHTTP.
> >>
> >> Having said that, it is a bit awkward to have to use a GenerateFlowFile
> to
> >> trigger InvokeHTTP to run, so
> >> I went ahead and created a ticket [1] to allow GetHTTP to evaluate the
> >> Expression Language for the URL
> >> property. In the meantime, though, GenerateFlowFile -> InvokeHTTP should
> >> provide you with
> >> the capability you're looking for.
> >>
> >> Thanks
> >> -Mark
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-993
> >>
> >> On Sep 23, 2015, at 9:14 PM, Daryl Teo wrote:
> >>
> >> Indus - use InvokeHTTP
> >>
> >> Daryl
> >>
> >> On Thu, Sep 24, 2015 at 11:07 AM, indus well
> wrote:
> >>>
> >>> Hello NiFi Experts:
> >>>
> >>> The GetHTTP processor works fine with static filename when getting
> files
> >>> from a website. However, I have a use case where I need to download a
> file
> >>> daily and the filename is the date of today, ie: 09222015.zip. Since
> the URL
> >>> property of the GetHTTP does not support expression language, I cannot
> do
> >>> something like http://example.com/${now():format('MMdd')}.zip. Is
> there
> >>> a way I can specify the filename dynamically? Or using other processor
> to
> >>> make this work. Please advise.
> >>>
> >>> Thanks,
> >>>
> >>> Indus
> >>
> >>
> >>
> >>
> >> --
> >> Daryl Teo
> >>
> >> Tech Lead
> >>
> >>
> >>
> >>
> >> 122 Cremorne Street
> >> Richmond VIC 3121
> >> Australia
> >>
> >> T +61 (3)8669 1821
> >> M +61 (0)432 413 752
> >>
> >> www.listcorp.com
> >>
> >>
> >>
> >> This email (including any attachments) is confidential and may be
> >> privileged. It may be read, copied and used only by the intended
> recipient.
> >> If you have received it in error, please contact the sender immediately
> by
> >> return email. Please then delete both emails and do not disclose their
> >> contents to any person. We believe, but do not warrant, that this email
> and
> >> any attachments are virus free. You should take full responsibility for
> >> virus checking. Listcorp. reserves the right to monitor all email
> >> communications. If the content of this email is personal or unconnected
> with
> >> our business, we accept no liability or responsibility for it.
> >>
> >>
> >
>
Re: GetHTTP Usage with Dynamic Filenames
Worked like a charm! Thank you for your quick response, Daryl and Mark.
The content of the downloaded file is stored in the file generated FlowFile
with a random filename, how would I rename the output file to actual
downloaded filename?
By the way, the ticket is a good enhancement.
Thanks,
Indus
On Wed, Sep 23, 2015 at 8:27 PM, Mark Payne wrote:
> Indus,
>
> As Daryl mentioned, InvokeHTTP will allow you to use the Expression
> Language to do an HTTP GET.
> It works a bit differently, though, because GetHTTP is a "Source
> Processor" whereas InvokeHTTP needs to
> be fed a FlowFile to do anything. So you can use GenerateFlowFile as a
> source and have it generate a
> 0 byte FlowFile (set the File Size property to "0 B"). Then just connect
> GenerateFlowFile to InvokeHTTP.
>
> Having said that, it is a bit awkward to have to use a GenerateFlowFile to
> trigger InvokeHTTP to run, so
> I went ahead and created a ticket [1] to allow GetHTTP to evaluate the
> Expression Language for the URL
> property. In the meantime, though, GenerateFlowFile -> InvokeHTTP should
> provide you with
> the capability you're looking for.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-993
>
> On Sep 23, 2015, at 9:14 PM, Daryl Teo wrote:
>
> Indus - use InvokeHTTP
>
> Daryl
>
> On Thu, Sep 24, 2015 at 11:07 AM, indus well wrote:
>
>> Hello NiFi Experts:
>>
>> The GetHTTP processor works fine with static filename when getting files
>> from a website. However, I have a use case where I need to download a file
>> daily and the filename is the date of today, ie: 09222015.zip. Since the
>> URL property of the GetHTTP does not support expression language, I cannot
>> do something like http://example.com/${now():format('MMdd')}.zip. Is
>> there a way I can specify the filename dynamically? Or using other
>> processor to make this work. Please advise.
>>
>> Thanks,
>>
>> Indus
>>
>
>
>
> --
> *Daryl Teo*
>
> Tech Lead
>
>
>
>
> 122 Cremorne Street
> Richmond VIC 3121
> Australia
>
> *T *+61 (3)8669 1821
> *M * +61 (0)432 413 752
>
> www.listcorp.com
>
>
>
> This email (including any attachments) is confidential and may be
> privileged. It may be read, copied and used only by the intended recipient.
> If you have received it in error, please contact the sender immediately by
> return email. Please then delete both emails and do not disclose their
> contents to any person. We believe, but do not warrant, that this email and
> any attachments are virus free. You should take full responsibility for
> virus checking. Listcorp. reserves the right to monitor all email
> communications. If the content of this email is personal or unconnected
> with our business, we accept no liability or responsibility for it.
>
>
>
GetHTTP Usage with Dynamic Filenames
Hello NiFi Experts:
The GetHTTP processor works fine with static filename when getting files
from a website. However, I have a use case where I need to download a file
daily and the filename is the date of today, ie: 09222015.zip. Since the
URL property of the GetHTTP does not support expression language, I cannot
do something like http://example.com/${now():format('MMdd')}.zip. Is
there a way I can specify the filename dynamically? Or using other
processor to make this work. Please advise.
Thanks,
Indus
