QFileSystemWatcher internals in Qt 5
tl;dr, all in all, a lot of code was deleted, and things still function more or less the same, except a bit better. That's quite a common story for Qt 5, I hope... :)
First of all, platform support: as with Qt 5 itself, Symbian support is no longer a goal. Since I wanted to make some changes to internals, and wasn't able to even remotely come close to building the Symbian code, it was removed.
On Linux, the (ancient, and no longer used by default) dnotify backend also met its maker. Since inotify has been around for some 6-7 years, it was about time, especially as the dnotify backend had some interesting bugs in behaviour.
The OS X FSEvents backend (also unused for quite some time, due to bugs, and not being a recommended way of working apparently) joined to make for a trinity of dead implementations. OS X's watching is survived by kqueue, which it shares with BSD platforms.
The currently supported backends are:
In the past, QFileSystemWatcher engines used to be run in a thread. I'm not sure why this was done originally, but it pretty much never made much sense - monitoring file changes is not a particularly intensive operation, so this is just a waste of resources (thread stack, time to start the thread, etc) - which was compounded by this being a thread per engine, meaning that if you have a few different libraries monitoring files, they'd each start their own thread.
This thread has now been removed, so changes are implicitly rate-limited to the thread the QFileSystemWatcher lives in, meaning that all of these are no longer a problem. Kudos should also go to Bradley Hughes for fixing a few issues which I missed on platforms other than Linux after it was integrated.
Brad also took this work a step further: QFileSystemWatcher has never been documented as being thread-safe, but the engines may have happened to be more or less thread-safe thanks to living on a different thread to the QFileSystemWatcher, through mutexing. One part inside Qt itself actually needed this for autotests to function correctly, too: QFileSystemModel. He fixed this requirement, and was thus able to remove the mutexes from the engines. Thanks!
I'd also like to thank Brad, João Abecasis, and anyone I've forgotten for helping to review these changes and get them integrated.
(One thing I neglected to mention above - the thread story is a little more complicated on Windows. Windows still has threads inside the engine (although the engine itself is no longer a thread, so there's still one less). This is necessary because WaitForMultipleObjects can only process up to MAXIMUM_WAIT_OBJECT handles at a time, unless you use multiple threads to do the monitoring, so that's exactly what it does. It spawns multiple threads on-demand as soon as it can't find a thread with a spare slot. But this is nothing new.)
On Linux, the (ancient, and no longer used by default) dnotify backend also met its maker. Since inotify has been around for some 6-7 years, it was about time, especially as the dnotify backend had some interesting bugs in behaviour.
The OS X FSEvents backend (also unused for quite some time, due to bugs, and not being a recommended way of working apparently) joined to make for a trinity of dead implementations. OS X's watching is survived by kqueue, which it shares with BSD platforms.
The currently supported backends are:
- inotify (on Linux)
- kqueue (on BSD and OS X)
- WaitForMultipleObjects on Windows, which I need to become more familiar with. Not having a Windows machine has meant that I'm not really able to do much here...
Aside from backend support, there were some more 'fun' changes which went in. First, some detail on implementation. Each QFileSystemWatcher has an 'engine' associated with it, which is backend-specific, and does the actual monitoring. The backend is responsible for communicating changes to the 'frontend' QFileSystemWatcher, which then sends the notifications to the API user.
In the past, QFileSystemWatcher engines used to be run in a thread. I'm not sure why this was done originally, but it pretty much never made much sense - monitoring file changes is not a particularly intensive operation, so this is just a waste of resources (thread stack, time to start the thread, etc) - which was compounded by this being a thread per engine, meaning that if you have a few different libraries monitoring files, they'd each start their own thread.
Another nasty side effect of this thread was resource consumption caused by monitoring. If you monitored a large number of paths, but couldn't consume events faster than the OS was throwing them at you, then that engine thread would happily sit there and keep on reading them and turning them into Qt signals for the QFileSystemWatcher/user code. But because that code was on a different thread, and unable to keep up, you'd just keep getting more, and more, and more signals, and memory usage would keep growing and growing.
This thread has now been removed, so changes are implicitly rate-limited to the thread the QFileSystemWatcher lives in, meaning that all of these are no longer a problem. Kudos should also go to Bradley Hughes for fixing a few issues which I missed on platforms other than Linux after it was integrated.
Brad also took this work a step further: QFileSystemWatcher has never been documented as being thread-safe, but the engines may have happened to be more or less thread-safe thanks to living on a different thread to the QFileSystemWatcher, through mutexing. One part inside Qt itself actually needed this for autotests to function correctly, too: QFileSystemModel. He fixed this requirement, and was thus able to remove the mutexes from the engines. Thanks!
I'd also like to thank Brad, João Abecasis, and anyone I've forgotten for helping to review these changes and get them integrated.
(One thing I neglected to mention above - the thread story is a little more complicated on Windows. Windows still has threads inside the engine (although the engine itself is no longer a thread, so there's still one less). This is necessary because WaitForMultipleObjects can only process up to MAXIMUM_WAIT_OBJECT handles at a time, unless you use multiple threads to do the monitoring, so that's exactly what it does. It spawns multiple threads on-demand as soon as it can't find a thread with a spare slot. But this is nothing new.)