Khaja, There are two options in NiFi for incremental database fetch: QueryDatabaseTable and GenerateTableFetch. The former is more often used on a standalone NiFi cluster for single tables (as it does not accept an incoming connection). It generates the SQL needed to do incremental fetching, then executes the statements and writes out the rows to the outgoing flowfile(s). GenerateTableFetch is meant for a cluster or for multiple tables (as it does accept an incoming connection), and does the "first half" of what QueryDatabaseTable does, generating SQL statements but not executing them. The statements are written out as FlowFiles to be executed downstream.
In a cluster, these processors are meant to run on the primary node only, otherwise each node will fetch the same information from the database and handle it in its own copy of the flow. Set the processor to run on the Primary Node Only. If using GenerateTableFetch, you can distribute the generated SQL statements using a Remote Process Group -> Input Port or a Load-Balanced Connection to an ExecuteSQL processor downstream, which will parallelize the actual fetching among the cluster nodes. Regards, Matt On Wed, Feb 24, 2021 at 3:46 PM KhajaAsmath Mohammed <mdkhajaasm...@gmail.com> wrote: > > Hi, > > I have a use case where I need to do incremental fetch on the oracle tables. > Is there a easy way to do this? I saw some posts about querydatabase table. > want to check if there is any efficient way to do this? > > Thanks, > Khaja
