https://bugs.kde.org/show_bug.cgi?id=384880
Bug ID: 384880
Summary: Dirwatcher inefficiency when using QFileSystemWatcher
Product: kdevplatform
Version: unspecified
Platform: Compiled Sources
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: NOR
Component: filemanager
Assignee: kdevelop-bugs-n...@kde.org
Reporter: rjvber...@gmail.com
Target Milestone: ---
Creating dirwatcher is largely negligible in terms of cost on Linux when
inotify is (can be used).
For instance, when importing a pre-existing project for the GCC 7.2.0 source
tree on Linux (1.6Ghz N3150 CPU, ZFS on an SSHD), with dirwatcher creation
deferred to after the initial project load and full-project parsing off:
kdevplatform.shell: Starting project import timer
kdevplatform.shell: All projects imported in 17.948 seconds
kdevplatform.filemanager: Project dir monitoring is enabled for project
"gcc-7.2.0-MP"
kdevplatform.shell: creating dir watchers and/or starting parse jobs took
an additional 1.189 seconds
kdevplatform.filemanager: Deleting dir watcher took 0.024 seconds for project
"gcc-7.2.0-MP"
31.069 user_cpu 5.016 kernel_cpu 0:34.10 total_time 105.7%CPU
When forcing the use of QFileSystemWatcher (using KDIRWATCH_METHOD=QFSWatch),
those figures become
kdevplatform.shell: Starting project import timer
kdevplatform.shell: All projects imported in 17.701 seconds
kdevplatform.filemanager: Project dir monitoring is enabled for project
"gcc-7.2.0-MP"
kdevplatform.shell: creating dir watchers and/or starting parse jobs took
an additional 179.013 seconds
kdevplatform.filemanager: Deleting dir watcher took 256.16 seconds for project
"gcc-7.2.0-MP"
470.119 user_cpu 19.310 kernel_cpu 7:39.48 total_time 106.5%CPU
(On Mac OS X with a 2.7Ghz i7 and HFS on a regular 7200RPM HDD,
creation:destruction take roughly 85:176 seconds)
These long wait times are spent entirely in KDirWatch method, blocking the
thread's event loop (and the UI in this case) and running at 100%CPU (on Mac
and Linux alike).
Deferring dirwatcher creation means the host OS has had opportunity to cache
the directory structure, and indeed profiling the operation on OS X shows that
the time is mostly spent in memory allocation and string comparison.
I think the extent of the overhead is due to an inefficiency in the way
dirwatching is implemented. A project nothing but a directory which will
usually contain other directories. Rather than doing actual dirwatching
(watching the project directory for changes), KDevelop installs watchers on
each and every item in the project tree. Notifications are only used for
keeping the tree in the project toolview uptodate. The KDirWatch::created
signal can lead to emitting a fileAdded signal, but that KDirWatch signal
(quote) " is not emitted when creating a file is created in a watched
directory". IOW and IIUC that signal is thus only emitted when a watch was
added for a non-existent file, which KDevelop doesn't do.
All this to say that I think it should be possible to rewrite this
implementation to watch only project directories for change, and invoke
`eventuallyReadFolder()` when a change is detected. This is an automated
version of how I have always worked with KDevelop on Mac (where dirwatching
doesn't work), and I don't have the impression that I'm missing any features
because of it.
Adding only project directories to the project dirwatcher will 1) speed up its
creation on all platforms and 2) will help avoid file descriptor depletion on
BSD variants (the kqueue-based QFSW backend uses 2 filedescriptors per watched
item, if memory serves me well, and depletion is not just a theoretical
possibility as I've observed with Qt4/KDevelop4 on Mac - OS X is a BSD variant
too). And finally 3) it should be possible to make the non-recursive
KDirWatch::addDir() call in eventuallyReadFolder or even in the
FileManagerListJob instance it creates. That could improve efficiency even more
by reducing the number of directory traversals.
NB: change detection in open documents does not depend on project dirwatching
so will not be affected by the proposed change.
--
You are receiving this mail because:
You are watching all bug changes.
