Hi, I just want to give a little update about what I have been up to recently. I have completed the design and implementation of the FileSystemFeedAdapter. The adapter basically gets a list of directories as a parameter and start ingesting records in files that exists within the specified directories.
I am using the Java WatchService API to monitor the selected directories. With this API, the system receives three types of notifications: 1. File Create notification. This notification is sent whenever a new file is created or a file has been renamed. 2. File Delete notification. This notification is sent whenever an existing file is deleted or a file has been renamed. 3. File Modification notification. This notification is sent whenever changes to an existing file are saved. We would like to avoid reading files which are being written and so we added another parameter that the user is expected to supply, "matching-expression". This expression is matched against File Create notifications and will start immediately ingesting data from the file. The user is expected to write their data in a file with a name that doesn't match the expression (i.e., something that ends with .tmp, .staging, etc) and rename it to something that matches the expression when the file is ready to be read. I think many other systems do the same. The feed stops working if all the watched directories have been removed or when the user disconnect the feed. The only remaining piece of this is how we can address feed continuation in case of system failure as right now, I am not storing progress on disk at all. This however will be added in another iteration. A change request will be submitted for this today and I am thinking about selecting @Murtadha, @Ian for review. Please, confirm that you have time to do it :-) and thanks in advance :-). I would've picked @Raman but he seems very busy these days. If any of this doesn't make sense or if there is a better way to do any of this, please let me know. Cheers, Abdullah.
