Hi Gary, I would need something in the run() method of FileAlterationMonitor. I need the full list of new files to sort after it has looped through the observers, but before it does another pass through the folders. We are monitoring 100 different folders, each of which is for an external source. For each folder, there could be multiple files created in a very short time, and the data for that source has to be imported in the correct order. The users have to review the data after it is imported, so keeping all the files in order is more of a courtesy to allow the users who sent their data first to be the first in the queue.
I have to change what I said earlier. I will need to check for existing files. In my current system, existing files would only be present if they sent data while the system was down, which they shouldn't do. It doesn't work correctly because the files are detected before the system is fully started. In the new design, the files could exist if the connection was lost, and the observer had to be restarted. I will need to think through the scenarios to figure this out. I’m thinking that I only need to check the on the first pass for the observer. We have a different application that uses WatchService to monitor a Windows shared folder on a different server. It does not work well if the network connection is lost even for a second. It may be something that could be fixed in the code. I don't know what our new disk storage solution will be yet, and we may have an interim solution before we get a permanent one. Based on my reading, WatchService only works well for certain platforms. It doesn't always detect new files. I think I’m locked into a specific version of library because it is used for other tools that we are using. I don't think I could upgrade to a newer version even if it had the features we need. Thanks again for your help, Patrick R. Thomas -----Original Message----- From: Gary Gregory <garydgreg...@gmail.com> Sent: Tuesday, January 30, 2024 1:51 PM To: Commons Users List <user@commons.apache.org> Subject: Re: Evaluating FileAlterationMonitor CAUTION! This email originated outside of Quest Diagnostics. DO NOT click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: ph...@questdiagnostics.com BTW, what are your findings that make WatchSerice not work well? Gary On Tue, Jan 30, 2024, 11:57 AM Gary D. Gregory <ggreg...@apache.org> wrote: > Patrick, > > ATM, I think you could create a subclass of the FileAlterationObserver > and re-implement FileAlterationObserver.checkAndNotify(). You'll have > to decide what to do in the case of file trees. Do you want to sort by > timestamp for all files in the tree or just sort files in the current > folder being processed. At least, that's the consideration needed for > a general solution. You'll notice that any solution will likely need a > second file comparator that sorts files for presentation to listeners > in the customized order. The current comparator is needed as is since > it is used for event and event type detection. > > Let me know what you think and you are considering providing a PR on > GitHub. > > TY! > > Gary > > On 2024/01/29 23:15:17 Gary Gregory wrote: > > YW. I think you could sort files if I refactor the monitor a bit > > such > that > > you can plug in a custom comparator (it's already there internally)... > > > > Gary > > > > On Mon, Jan 29, 2024, 5:19 PM Thomas, Patrick R > > <patrick.r.tho...@questdiagnostics.com.invalid> wrote: > > > > > Thanks for the quick response. I will need to test its behavior > > > for > lost > > > connections. > > > > > > Based on what I have discovered so far, I will have to create my > > > own monitor. I need to create a list of new files from all of the > > > monitored folders during each pass and then sort them by date/time > > > to make sure > they > > > are processed in the correct order. > > > > > > It doesn't seem to notify about files that already exist. I will > > > have > to > > > add this as well, although it is optional because it doesn't work > correctly > > > in my current system. > > > > > > Thank you, > > > > > > Patrick R. Thomas > > > > > > -----Original Message----- > > > From: Gary Gregory <garydgreg...@gmail.com> > > > Sent: Monday, January 29, 2024 2:05 PM > > > To: Commons Users List <user@commons.apache.org> > > > Subject: Re: Evaluating FileAlterationMonitor > > > > > > CAUTION! This email originated outside of Quest Diagnostics. DO > > > NOT > click > > > links or open attachments unless you recognize the sender and know > > > the content is safe. Please report suspicious emails to: > > > ph...@questdiagnostics.com > > > > > > > > > The documentation we have is on the site and the Java document. If > > > you have improvements on the code or documentation, we welcome > > > them as PRs > on > > > GitHub. > > > > > > The monitor thread wakes up, looks around, and fires events. If a > > > file > is > > > not there because a network connection is broken, this will show > > > up as > a > > > deleted file. > > > > > > YYMV since we do not test this use case in unit tests. > > > > > > Gary > > > > > > > > > On Mon, Jan 29, 2024, 1:45 PM Thomas, Patrick R > > > <patrick.r.tho...@questdiagnostics.com.invalid> wrote: > > > > > > > Hello, > > > > > > > > Our application monitors multiple folders for new files to import. > > > > Once a new file is detected, it is moved to a working folder > > > > where it is processed and then moved to an archive folder. Files > > > > are never updated, and ones that are deleted before processing > > > > are skipped. The current implementation has the folders on the > > > > same server as the application. We are using Java's WatchService to do > > > > the monitoring. > > > > For security reasons, we have to move to a new design where the > > > > folders will be on a different server. My understanding is that > > > > WatchService will not work well for this. I am evaluating > > > > FileAlterationMonitor as an alternative. I have several > > > > questions. Is there additional documentation I can reference? If > > > > the folder being monitored is on a mapped network drive and the > > > > connection is lost, what happens? Can the monitor detect files > > > > that already exist when it starts? It is also important that the > > > > files remain in the order they > > > were created to be imported. > > > > > > > > Any help would be greatly appreciated. > > > > > > > > Thank you, > > > > > > > > Patrick R. Thomas > > > > Software Engineer > > > > > > > > > > > > > ______________________________________________________________________ > > > > The contents of this message, together with any attachments, are > > > > intended only for the use of the person(s) to which they are > addressed > > > > and may contain confidential and/or privileged information. > > > > Further, any medical information herein is confidential and protected > > > > by law. > > > > It is unlawful for unauthorized persons to use, review, copy, > > > > disclose, or disseminate confidential medical information. If > > > > you are not the intended recipient, immediately advise the > > > > sender and delete > > > this message and any attachments. > > > > Any distribution, or copying of this message, or any attachment, > > > > is prohibited. > > > > > > __________________________________________________________________ > > > ____ The contents of this message, together with any attachments, > > > are > intended > > > only for the use of the person(s) to which they are addressed and > > > may contain confidential and/or privileged information. Further, > > > any > medical > > > information herein is confidential and protected by law. It is > unlawful for > > > unauthorized persons to use, review, copy, disclose, or > > > disseminate confidential medical information. If you are not the > > > intended > recipient, > > > immediately advise the sender and delete this message and any > attachments. > > > Any distribution, or copying of this message, or any attachment, > > > is prohibited. > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > For additional commands, e-mail: user-h...@commons.apache.org > > ______________________________________________________________________ The contents of this message, together with any attachments, are intended only for the use of the person(s) to which they are addressed and may contain confidential and/or privileged information. Further, any medical information herein is confidential and protected by law. It is unlawful for unauthorized persons to use, review, copy, disclose, or disseminate confidential medical information. If you are not the intended recipient, immediately advise the sender and delete this message and any attachments. Any distribution, or copying of this message, or any attachment, is prohibited.
