Okay, this was a bit more complex, and a bit simpler than I thought.
1) The Django app __init__ is the only place you can place the signal to stop
the watchdog function.
2) You can’t perform any app model specific work in the __init__ because the
app’s model frame work wasn’t yet finished initializing. So I ended up with a
chicken & egg situation.
Now, this does *exactly* what I needed to do, so this might not be needed for
anyone else, but it without a doubt simplifies any sort of file caching issue,
since you can invalidate the cache with whatever granularity you wanted (eg
File Level, or in my case, Directory level).
I created a class to contain the watchdog specific code, and a table that
stored the list of cached directories (to invalidate, I just remove the
directory name from that table).
The existing scanning code is just bypassed if it’s already been scanned, if
it’s invalidated, it’ll just refresh the listing(s) in the database.
eg.
in watchdogmon.py:
class watchdog_monitor():
def __init__(self):
self.my_observer = Observer()
self.my_event_handler = None
def on_event(self, event):
pass
def startup(self, monitor_path, created=None,
deleted=None, modified=None,
moved=None):
print("Monitoring :", monitor_path)
patterns = ["*"]
ignore_patterns = None
ignore_directories = False
case_sensitive = False
self.my_event_handler = PatternMatchingEventHandler(patterns,
ignore_patterns,
ignore_directories,
case_sensitive)
self.my_event_handler.on_created = created
self.my_event_handler.on_deleted = deleted
self.my_event_handler.on_modified = modified
self.my_event_handler.on_moved = moved
go_recursively = True
self.my_observer = Observer()
self.my_observer.schedule(self.my_event_handler,
monitor_path,
recursive=go_recursively)
self.my_observer.start()
def shutdown(self, *args):
if os.environ.get('RUN_MAIN') == 'true':
print("Shutting down")
self.my_observer.stop()
self.my_observer.join()
# signal.send('system')
sys.exit(0) # So runserver does try to exit
watchdog = watchdog_monitor()
In __init__.py:
signal.signal(signal.SIGINT, watchdogmon.watchdog.shutdown)
In views.py:
print("Clearing all entries from Cache Tracking")
Cache_Tracking.objects.all().delete()
print("Starting Watchdog")
from frontend.utilities import delete_from_cache_tracking
watchdog.startup(monitor_path=os.path.join(configdata["locations"]["albums_path"],
"albums"),
created=delete_from_cache_tracking,
deleted=delete_from_cache_tracking,
modified=delete_from_cache_tracking,
moved=delete_from_cache_tracking)
where delete_from_cache_tracking, just deletes the event.src_path from the
cache_Tracking table.
Now, the drawback here is that watchdog utilizes the fsevents framework under
Mac OS X, so there is no practical limit to what it can track, the OS is doing
the work watchdog is wrapping around OS provided functionality. (Or at least
that is my understanding)
So under Mac OS X, it’s tracking probably millions or at least 10’s of
thousands of files, and directories.
Under Windows, or Unix, my understanding is that watchdog requires a file
handle per directory? So some tweaking maybe required to prevent from running
out of file handles under a large directory structure / number of files.
Hope this helps someone else that is trying to integrate watchdog into Django.
- BEnjamin
> On May 20, 2021, at 10:14 PM, Benjamin Schollnick
> <[email protected]> wrote:
>
> Folks,
>
> I’m working on revamping the Gallery/viewer application that I wrote in
> Django.
>
> One issue I’ve had is that it’s a dynamic on the fly thumbnail creator, but
> I’ve used a homebrew (fairly successfully) cached system to keep track of the
> directories & file contents. Unlike many of the other gallery style
> applications I don’t require deliberate pre-scanning, the files are detected,
> and thumbnailed in realtime.
>
> But I realized that I can use Watchdog to simplify that logic, and eliminate
> the need for any caching.
>
> My idea is to have a table that is cleared at every startup, and it just
> contains the directories that have been scanned for that execution of the
> program. If watchdog has detected a change in a directory, it’s just removed
> from that table, which makes it a candidate for rescanning the next time
> someone views that directory.
>
> But the big kicker is that I can’t find any documentation that can help me
> add watchdog monitoring into Django.
>
> Now, I know that Django does something similar in the Dev server (which is
> what I primarily use right now)… And I’ve found a stackoverflow, whichs that
> I can use Signal to capture the SIGINT on shutdown (allowing me to stop
> watchdog )… I should be able to start watchdog in the __init__ file in the
> APP directory as well…
>
> Now this is the sample app that I used:
>
> import sys
> import time
> from watchdog.observers import Observer
> from watchdog.events import PatternMatchingEventHandler
>
> def on_created(event):
> print(f"hey, {event.src_path} has been created!")
>
> def on_deleted(event):
> print(f"what the f**k! Someone deleted {event.src_path}!")
>
> def on_modified(event):
> print(f"hey buddy, {event.src_path} has been modified")
>
> def on_moved(event):
> print(f"ok ok ok, someone moved {event.src_path} to {event.dest_path}")
>
> if __name__ == "__main__”:
>
> # <— startup —>
> patterns = ["*"]
> ignore_patterns = None
> ignore_directories = False
> case_sensitive = True
> my_event_handler = PatternMatchingEventHandler(patterns, ignore_patterns,
> ignore_directories, case_sensitive)
>
> my_event_handler.on_created = on_created
> my_event_handler.on_deleted = on_deleted
> my_event_handler.on_modified = on_modified
> my_event_handler.on_moved = on_moved
>
> path = sys.argv[1]#"."
> go_recursively = True
> my_observer = Observer()
> my_observer.schedule(my_event_handler, path, recursive=go_recursively)
>
> my_observer.start()
>
> # stop startup
>
> try:
> while True:
> time.sleep(1)
> except KeyboardInterrupt:
> # shutdown
> my_observer.stop()
> my_observer.join()
>
>
> Nothing fancy, just a proof of concept based on code that I found online.
>
> Anyone have any suggestions?
>
> I’m thinking that I could add the startup .. stop startup to an function in
> the __init__ and run that at startup.
> And that capturing the SIGINT, and when SIGINT is tripped, I can do the
> observer.stop & join?
>
> Would it be that simple? Am I missing something? Is there a better way to
> do this in Django?
> I don’t want to have to stand up a celery server, if I can run this, or
> something equivalent inside of Django…
>
> I haven’t been able to (yet) trackdown STATRELOADER, which appears to be the
> Django built-in equivalent, so I can’t tell if I can repurpose that to do
> this instead?
>
> I’m open to suggestions…
>
> - Benjamin
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/5300158C-831E-407C-A8C8-617C40D84E1B%40schollnick.net.
