Re: How to get ftp file according to Current date?

[prabhu Mahendran]

Hello Pierre,

Thanks for your suggestions.

I have tried one more way for my requirments.

I have used* GenerateFlowFile-->FetchFTP *processor for fetch the file with
current date by using "*RemoteFile*" Attribute (Expression language
supported).

In FetchFTP processor gets data from FTP Server and route it into "
*commons.failure" not into "success" relationship.*
For that issue i have found JIRA Ticket.

*https://issues.apache.org/jira/browse/NIFI-3553
*
I have applied patches available in that task in NiFi.But it leads
IOException error and also route entire file into *"commons.failure"*
relationship.

ERROR [Timer-Driven Process Thread-4] o.a.nifi.processors.standard.FetchFTP
FetchFTP[id=9ff253e1-015b-1000-4af3-9701cd126b69] Failed to fetch content
for
StandardFlowFileRecord[uuid=1a685b21-b431-4786-a171-1b3772e576d0,claim=,offset=0,name=520303376677591,size=0]
from filename /FTPFolder/20170425InputFile.csv on remote host
ftp.host.domain:21 due to
org.apache.nifi.processor.exception.ProcessException: IOException thrown
from FetchFTP[id=9ff253e1-015b-1000-4af3-9701cd126b69]:
java.net.SocketTimeoutException: Read timed out; routing to comms.failure:
org.apache.nifi.processor.exception.ProcessException: IOException thrown
from FetchFTP[id=9ff253e1-015b-1000-4af3-9701cd126b69]:
java.net.SocketTimeoutException: Read timed out
2017-04-25 09:57:13,812 ERROR [Timer-Driven Process Thread-4]
o.a.nifi.processors.standard.FetchFTP
org.apache.nifi.processor.exception.ProcessException: IOException thrown
from FetchFTP[id=9ff253e1-015b-1000-4af3-9701cd126b69]:
java.net.SocketTimeoutException: Read timed out
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2348)
~[na:na]
at
org.apache.nifi.processors.standard.FetchFileTransfer.onTrigger(FetchFileTransfer.java:238)
~[nifi-standard-processors-1.1.1.jar:1.1.1]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.1.jar:1.1.1]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.1.jar:1.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_91]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_91]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_91]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[na:1.8.0_91]
at java.net.SocketInputStream.read(SocketInputStream.java:170)
~[na:1.8.0_91]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[na:1.8.0_91]

i don't know why this error has shown in processor.

Can you help me regarding this?

Thanks,
Mahendran

On Mon, Apr 24, 2017 at 9:32 PM, Pierre Villard  wrote:

> Hi,
>
> You can file a JIRA and ask for an improvement to add this feature to
> GetFTP and ListFTP, or modify the processor on your side and re-generate
> the NAR containing the processor. This should not be a huge change.
>
> 2017-04-24 9:11 GMT+02:00 prabhu Mahendran :
>
>> Pierre,
>>
>> Thanks for your reply.
>>
>> I have tried your suggestions.
>>
>> i am having 1 lakhs files in FTP Directory.
>>
>> For *ListFTP* It check all filenames and process into queue it could
>> take long time.
>>
>> So it can take more time for fetch particular FTP file from lakhs.
>>
>> Can you guide way to reduce time taken for fetch file?
>>
>> On Fri, Apr 21, 2017 at 8:40 PM, Pierre Villard <
>> pierre.villard...@gmail.com> wrote:
>>
>>> You could use the combination of ListFTP and FetchFTP (this is, most of
>>> the time, a better approach), and between the two processors you could do a
>>> RouteOnAttribute and only keep the flow files with the filename you are
>>> looking for.
>>>
>>> Pierre.
>>>
>>> 2017-04-21 13:29 GMT+02:00 prabhu Mahendran :
>>>
  I have tried that "GetFTP" processor in which downloads file from FTP
 accoding to the two attributes

 1

Re: How to get ftp file according to Current date?

[Pierre Villard]

Hi,

You can file a JIRA and ask for an improvement to add this feature to
GetFTP and ListFTP, or modify the processor on your side and re-generate
the NAR containing the processor. This should not be a huge change.

2017-04-24 9:11 GMT+02:00 prabhu Mahendran :

> Pierre,
>
> Thanks for your reply.
>
> I have tried your suggestions.
>
> i am having 1 lakhs files in FTP Directory.
>
> For *ListFTP* It check all filenames and process into queue it could take
> long time.
>
> So it can take more time for fetch particular FTP file from lakhs.
>
> Can you guide way to reduce time taken for fetch file?
>
> On Fri, Apr 21, 2017 at 8:40 PM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> You could use the combination of ListFTP and FetchFTP (this is, most of
>> the time, a better approach), and between the two processors you could do a
>> RouteOnAttribute and only keep the flow files with the filename you are
>> looking for.
>>
>> Pierre.
>>
>> 2017-04-21 13:29 GMT+02:00 prabhu Mahendran :
>>
>>>  I have tried that "GetFTP" processor in which downloads file from FTP
>>> accoding to the two attributes
>>>
>>> 1."FileFilterRegex" -Name of file in FTP
>>> 2."RemotePath"-Path of an FTP file.
>>>
>>> I wants to download the File from FTP Server only if it having today's
>>> date which is append with filename.
>>>
>>> *For example:*
>>>
>>> File name is *20170421TempFile.txt *
>>> which is in FTP Server.
>>>
>>> Now i need to give that system date only to be append with filename.It
>>> should get the current system date automatically *instead of given date
>>> value directly*.
>>>
>>> So i have find that ${now()} gets the current system date but i cannot
>>> give it in *"FileFilterRegex"* attribute due to it doesn't have
>>> expression language support.
>>>
>>> Finally i need to get particular file with current date.
>>>
>>>  Anyone give some idea/guide me to achieve my requirements?
>>>
>>
>>
>

Re: How to get ftp file according to Current date?

[prabhu Mahendran]

Pierre,

Thanks for your reply.

I have tried your suggestions.

i am having 1 lakhs files in FTP Directory.

For *ListFTP* It check all filenames and process into queue it could take
long time.

So it can take more time for fetch particular FTP file from lakhs.

Can you guide way to reduce time taken for fetch file?

On Fri, Apr 21, 2017 at 8:40 PM, Pierre Villard  wrote:

> You could use the combination of ListFTP and FetchFTP (this is, most of
> the time, a better approach), and between the two processors you could do a
> RouteOnAttribute and only keep the flow files with the filename you are
> looking for.
>
> Pierre.
>
> 2017-04-21 13:29 GMT+02:00 prabhu Mahendran :
>
>>  I have tried that "GetFTP" processor in which downloads file from FTP
>> accoding to the two attributes
>>
>> 1."FileFilterRegex" -Name of file in FTP
>> 2."RemotePath"-Path of an FTP file.
>>
>> I wants to download the File from FTP Server only if it having today's
>> date which is append with filename.
>>
>> *For example:*
>>
>> File name is *20170421TempFile.txt *
>> which is in FTP Server.
>>
>> Now i need to give that system date only to be append with filename.It
>> should get the current system date automatically *instead of given date
>> value directly*.
>>
>> So i have find that ${now()} gets the current system date but i cannot
>> give it in *"FileFilterRegex"* attribute due to it doesn't have
>> expression language support.
>>
>> Finally i need to get particular file with current date.
>>
>>  Anyone give some idea/guide me to achieve my requirements?
>>
>
>

Re: How to get ftp file according to Current date?

[Pierre Villard]

You could use the combination of ListFTP and FetchFTP (this is, most of the
time, a better approach), and between the two processors you could do a
RouteOnAttribute and only keep the flow files with the filename you are
looking for.

Pierre.

2017-04-21 13:29 GMT+02:00 prabhu Mahendran :

>  I have tried that "GetFTP" processor in which downloads file from FTP
> accoding to the two attributes
>
> 1."FileFilterRegex" -Name of file in FTP
> 2."RemotePath"-Path of an FTP file.
>
> I wants to download the File from FTP Server only if it having today's
> date which is append with filename.
>
> *For example:*
>
> File name is *20170421TempFile.txt *
> which is in FTP Server.
>
> Now i need to give that system date only to be append with filename.It
> should get the current system date automatically *instead of given date
> value directly*.
>
> So i have find that ${now()} gets the current system date but i cannot
> give it in *"FileFilterRegex"* attribute due to it doesn't have
> expression language support.
>
> Finally i need to get particular file with current date.
>
>  Anyone give some idea/guide me to achieve my requirements?
>

How to get ftp file according to Current date?

[prabhu Mahendran]

 I have tried that "GetFTP" processor in which downloads file from FTP
accoding to the two attributes

1."FileFilterRegex" -Name of file in FTP
2."RemotePath"-Path of an FTP file.

I wants to download the File from FTP Server only if it having today's date
which is append with filename.

*For example:*

File name is *20170421TempFile.txt *
which is in FTP Server.

Now i need to give that system date only to be append with filename.It
should get the current system date automatically *instead of given date
value directly*.

So i have find that ${now()} gets the current system date but i cannot give
it in *"FileFilterRegex"* attribute due to it doesn't have expression
language support.

Finally i need to get particular file with current date.

 Anyone give some idea/guide me to achieve my requirements?