Mark, Does zookeeper process runs inside Nifi JVM? If yes, what is the default path of zookeeper data directory?
Regards, Sourav Gulati -----Original Message----- From: Mark Payne [mailto:marka...@hotmail.com] Sent: Wednesday, May 11, 2016 6:37 PM To: dev@nifi.apache.org Subject: Re: Reg: Get files from ftp Sourav, If your NiFi instance is clustered, it will store the information in ZooKeeper. If not clustered, it will store the state in a local file. This is done because in a cluster, you typically want to run your List*** Processors on Primary Node only, and this allows another node to pick up where the previous one left off if the Primary Node changes. Of course, storing all of the files that have been listed can become very verbose so it stores only a small amount of data -- the timestamp of the latest file discovered and the timestamp of the latest file process/listed. It can then use this information to determine if files are new or modified without storing much info. Thanks -Mark > On May 11, 2016, at 12:39 AM, Sourav Gulati <sourav.gul...@impetus.co.in> > wrote: > > Thanks Matthew, > A quick question: Where does it store the state of files already listed? > > > Regards, > Sourav Gulati > > -----Original Message----- > From: Matthew Clarke [mailto:matt.clarke....@gmail.com] > Sent: Wednesday, May 11, 2016 3:37 AM > To: dev@nifi.apache.org > Subject: Re: Reg: Get files from ftp > > The list type processors are designed to use NiFi state management to keep > from listing the same files twice. The fetch type processors with retrieve > files based on the FlowFiles it is fed. Typically those FlowFiles it works > from come from the corresponding list processor. > On May 10, 2016 8:56 AM, "Mark Payne" <marka...@hotmail.com> wrote: > >> Sourav, >> >> Sure. Within the nifi-standard-processors bundle are a few classes >> that would be important here. >> First is the AbstractListProcessor. You'll want to use this as your >> base class for ListFTP. Also, FetchFileTransfer will be the class >> that you'll extend for the FetchFTP processor. >> >> The ListSFTP and FetchSFTP are great examples to look at as examples. >> >> Additionally, the GetFTP and GetSFTP are good examples to look at as >> to how the FTP & SFTP implementations differ. They basically differ >> in the Property Descriptors provided and the FileTransfer object that >> is used. >> >> If you have any questions, please feel free to reach out to this >> mailing list. Very happy to help however we can! >> >> Thanks >> -Mark >> >> >>> On May 10, 2016, at 1:30 AM, Sourav Gulati >>> <sourav.gul...@impetus.co.in> >> wrote: >>> >>> Sure Mark. I am interested to work on it. Please provide some >>> pointers >> regarding that. >>> >>> Also, I will check if Sftp can be used. So ListSFTP / FetchSFTP >>> won't >> pick files more than once? >>> >>> Regards, >>> Sourav Gulati >>> >>> -----Original Message----- >>> From: Mark Payne [mailto:marka...@hotmail.com] >>> Sent: Monday, May 09, 2016 5:34 PM >>> To: dev@nifi.apache.org >>> Subject: Re: Reg: Get files from ftp >>> >>> Sourav, >>> >>> We have begun transitioning from many of the Get*** Processors to >> List*** and Fetch*** Processors. >>> There is a ListSFTP / FetchSFTP processor set but not currently a >> List/Fetch FTP. Is SFTP a possibility for you? Would you be >> interested in working on a List/Fetch FTP Processor set? >>> >>> Thanks >>> -Mark >>> >>>> On May 9, 2016, at 5:48 AM, Sourav Gulati >>>> <sourav.gul...@impetus.co.in> >> wrote: >>>> >>>> Hi Team, >>>> >>>> I need a suggestion. >>>> >>>> I want to get files from ftp server for which GetFtp processor is >> available. However, as I cannot delete files from source, I need to >> put a check that this processor does not pick a file more than once. >> What is the best way to do that? >>>> >>>> Regards, >>>> Sourav Gulati >>>> >>>> >>>> ________________________________ >>>> >>>> >>>> >>>> >>>> >>>> >>>> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited >> when received in error. Impetus does not represent, warrant and/or >> guarantee, that the integrity of this communication has been >> maintained nor that the communication is free of errors, virus, interception >> or interference. >>> >>> >>> ________________________________ >>> >>> >>> >>> >>> >>> >>> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited >> when received in error. Impetus does not represent, warrant and/or >> guarantee, that the integrity of this communication has been >> maintained nor that the communication is free of errors, virus, interception >> or interference. >> >> > > ________________________________ > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference. ________________________________ NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
