I had a very cursory glance at the source code, and it seems like the WatchfolderSource just checks if a file exists (using a periodic task in Celery if I'm not mistaken), and processes it if it does. So I think that could be improved.
@Roberto, do you think it would be worth it to have a standalone watchfolder daemon that can be configured to push files to Mayan e.g. using the API, which means it could also be run on a local machine and push to a remote Mayan instance, or do you prefer to keep it a Celery task? If it's a celery task, I don't think you can use some file watcher like pyinotify (which requires a continuously running process which is not really what Celery is intended for), but you could still have the task wait for the files it encounters to stop growing before processing them further. On Thu, May 24, 2018 at 7:19 PM, Jesaja Everling <jeverl...@gmail.com> wrote: > Hi Eddi, > > I'm not familiar with the code that handles the watchfolder monitoring, > and it has been a while that I worked with watchfolders myself. I think you > have probably two options - make sure the watcher acts on the appropriate > event, e.g. IN_CLOSE_WRITE in http://seb.dbzteam.org/ > pyinotify/pyinotify-module.html#IN_CLOSE_WRITE, or you could use a > spin-loop to check if the file is still growing and only process it further > when it has stopped growing for some time (which sounds less reliable then > relying on pyinotify or watchdog or a similar solution). > > Don't know how much that helps, hope it does. > > Best Regards, > > Jesaja Everling > > > On Thu, May 24, 2018 at 4:44 PM, Eddi <efreite...@gmail.com> wrote: > >> Hi, >> >> I am using Mayan 2.7.3. I have a watch folder setup, where a scanner >> saves files to. >> >> When the scanner starts scanning, it will create a file in the folder. >> When watching the folder via CLI while the scanner is scanning, I can see >> the file grow in size. >> >> The issue comes, when the scanner is scanning, and the Mayan watch >> function checks the folder. The file is present (but not completed because >> the scanner is still scanning), but Mayan EDMS grabs the file. >> The end result is a corrupt file (I use mostly PDF extensions). In Mayan >> it shows up as a red question mark. >> >> How can I work around this? >> >> Cheers, >> >> Eddi >> >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "Mayan EDMS" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to mayan-edms+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- --- You received this message because you are subscribed to the Google Groups "Mayan EDMS" group. To unsubscribe from this group and stop receiving emails from it, send an email to mayan-edms+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
