Dear All,
Greetings of the Day!
I hope you are doing well.
I am currently working on a data integration task where I need to perform an
Incremental JOIN between a transactional table and a master table using Apache
NiFi processors. Despite multiple attempts and research, I am facing challenges
in implementing it correctly.
Here is a brief overview of the requirement:
*
Transactional Table: Contains newly generated or updated records.
*
Master Table: Holds historical or master data.
*
Objective: Incrementally join the transactional table's latest changes with the
master data without reprocessing the entire dataset.
*
Challenge:
*
Identifying and fetching only the new/updated records efficiently.
*
Performing a JOIN operation inside NiFi flow processors (like
QueryDatabaseTable, ExecuteSQL, LookupRecord, etc.).
*
Maintaining synchronization between incremental loads to ensure no data
duplication or data loss.
I would sincerely appreciate any guidance, best practices, or examples on how
to achieve an Incremental JOIN in a resource-optimized and reliable manner
using NiFi.
Please let me know if you need any additional details regarding the current
processor setup or the schema structure.
Thank you in advance for your support!
Best Regards,
Deepanshu
