Rai, There are incremental data movement processors in NiFi depending on your source/target. For example, if your sources are files, you can use ListFile in combination with FetchFile, the former will keep track of which files it has found thus far, so if you put new files into the location (or update existing ones), only those new/updated files will be processed the next time.
For database (RDBMS) sources, there are the QueryDatabaseTable and GenerateTableFetch processors, which support the idea of "maximum value columns", such that for each of said columns, the processor(s) will keep track of the maximum value observed in that column, then for future executions of the processor, only rows whose values in those columns exceed the currently-observed maximum will be retrieved, then the maximum will be updated, and so forth. The Usage documentation for these processors can be found at https://nifi.apache.org/docs.html (left-hand side under Processors). Regards, Matt On Thu, Oct 13, 2016 at 11:05 AM, Gop Krr <[email protected]> wrote: > Hi All, > I am learning NiFi as well as trying to deploy it in production for few use > cases. One of the use case is ETL and another use case is, using NiFi as a > backup solution, where it takes the data from one source and moves to > another database|file. Is anyone using NiFi for this purpose? Does NiFi > support incremental data move? > It would be awesome if someone can point me to right documentation. > Thanks > Rai
