Sunday, 22 January 2012

QFileSystemWatcher internals in Qt 5

Just thought I'd share some details on some of the recent changes I've pushed to Qt 5 a few weeks ago. (Yes, this post is rather overdue, I've been a bit slack with writing it). If you were in Tampere when I gave a short, completely underprepared Q&A on Qt 5 a few days ago, this won't be news to you, but I will go into a bit more detail.

tl;dr, all in all, a lot of code was deleted, and things still function more or less the same, except a bit better. That's quite a common story for Qt 5, I hope... :)

First of all, platform support: as with Qt 5 itself, Symbian support is no longer a goal. Since I wanted to make some changes to internals, and wasn't able to even remotely come close to building the Symbian code, it was removed.

On Linux, the (ancient, and no longer used by default) dnotify backend also met its maker. Since inotify has been around for some 6-7 years, it was about time, especially as the dnotify backend had some interesting bugs in behaviour.

The OS X FSEvents backend (also unused for quite some time, due to bugs, and not being a recommended way of working apparently) joined to make for a trinity of dead implementations. OS X's watching is survived by kqueue, which it shares with BSD platforms.

The currently supported backends are:
  • inotify (on Linux)
  • kqueue (on BSD and OS X)
  • WaitForMultipleObjects on Windows, which I need to become more familiar with. Not having a Windows machine has meant that I'm not really able to do much here...
Aside from backend support, there were some more 'fun' changes which went in. First, some detail on implementation. Each QFileSystemWatcher has an 'engine' associated with it, which is backend-specific, and does the actual monitoring. The backend is responsible for communicating changes to the 'frontend' QFileSystemWatcher, which then sends the notifications to the API user.

In the past, QFileSystemWatcher engines used to be run in a thread. I'm not sure why this was done originally, but it pretty much never made much sense - monitoring file changes is not a particularly intensive operation, so this is just a waste of resources (thread stack, time to start the thread, etc) - which was compounded by this being a thread per engine, meaning that if you have a few different libraries monitoring files, they'd each start their own thread.

Another nasty side effect of this thread was resource consumption caused by monitoring. If you monitored a large number of paths, but couldn't consume events faster than the OS was throwing them at you, then that engine thread would happily sit there and keep on reading them and turning them into Qt signals for the QFileSystemWatcher/user code. But because that code was on a different thread, and unable to keep up, you'd just keep getting more, and more, and more signals, and memory usage would keep growing and growing.

This thread has now been removed, so changes are implicitly rate-limited to the thread the QFileSystemWatcher lives in, meaning that all of these are no longer a problem. Kudos should also go to Bradley Hughes for fixing a few issues which I missed on platforms other than Linux after it was integrated.

Brad also took this work a step further: QFileSystemWatcher has never been documented as being thread-safe, but the engines may have happened to be more or less thread-safe thanks to living on a different thread to the QFileSystemWatcher, through mutexing. One part inside Qt itself actually needed this for autotests to function correctly, too: QFileSystemModel. He fixed this requirement, and was thus able to remove the mutexes from the engines. Thanks!

I'd also like to thank Brad, João Abecasis, and anyone I've forgotten for helping to review these changes and get them integrated.

(One thing I neglected to mention above - the thread story is a little more complicated on Windows. Windows still has threads inside the engine (although the engine itself is no longer a thread, so there's still one less). This is necessary because WaitForMultipleObjects can only process up to MAXIMUM_WAIT_OBJECT handles at a time, unless you use multiple threads to do the monitoring, so that's exactly what it does. It spawns multiple threads on-demand as soon as it can't find a thread with a spare slot. But this is nothing new.)

7 comments:

  1. Sounds like some good improvements!

    Note: your site is a real huge pain to read with white on (nearly) black.. It hurts my eyes!

    ReplyDelete
  2. > The OS X FSEvents backend (also unused for quite some time, due to bugs,
    > and not being a recommended way of working apparently) joined to make for a
    > trinity of dead implementations.

    FSEvents and kqueues have different pros/cons. I don't think it makes sense to try and hide them from the Qt user as just "the backend on a particular platform". FSEvents is good when you want to watch a potentially large directory hierarchy with cheap initialization and low resource usage and be able to get change notifications for changes that happened when your app was not running. The downside is coarse-grained notifications (directory level only). Kqueues on the other hand provides much more fine-grained notifications but uses a file handle for every directory watched.

    The QFileSystemWatcher documentation in Qt 4.7 states: " Mac OS X 10.5 and up use a different backend and do not suffer from this issue.". I believe this refers to the FSEvents backend, which isn't used so the documentation is incorrect - file descriptor limits are a problem on Mac with QFSW if you are potentially going to monitor a lot of directories.

    ReplyDelete
  3. Robert, you're right in that the documentation is incorrect. If you want to submit a patch for that, I'll happily approve it (or I'll try get to it when I next remember). As for FSEvents, I actually like the idea a lot, but when it doesn't work, it just plain doesn't work - and FSEvents was disabled for 4.7 too.

    As for max FDs being a problem - at least on my OS X system (10.6), they're over 10,000. That's more than the default max_user_watches inotify setting on my Linux system, so I think that may be another case of the documentation being outdated or just plain incorrect.

    That having been said, I would like to add a method to allow fallback to polling when monitoring paths for 'worst case' type monitoring. We'll see.

    ReplyDelete
  4. Hope QFileSystemWatcher will also have more information about which file changed (as for directoryChanged()) and what actually changed (for fileChanged()). :)

    ReplyDelete
  5. ahh, i can read your site now without hurting my eyes :) Thanx for that.

    Anyway, how about custom monitoring backends? With that i mean for example the KIO package in KDE. They have a range of different protocols to do all different kind of file magic on, but can a backend be made to listen for file/folder changes on __for_example__
    - http
    - smb
    - ftp
    - nfs
    - ... a dozen others
    ??

    Just wondering if that's possible. Even if it is, i'm not exactly sure yet how that will benefit KIO. Other then that, some protocols probable don't even have a way to notify the user of file changes (http? ftp?)

    Thanx,
    Mark

    ReplyDelete
  6. Re threaded QFileSystemWatcher; back then (iirc pre 2.6.31) Linux 2.6 inotify wasn't behaving that well under heavy HD load. If for example a KDirWatch (which doesn't only listen to a single file but to a whole directory including sub-directories) was listening to your build-dir applications could freeze for fractions of seconds while KDirWatch was busy processing lots of data (see https://svn.reviewboard.kde.org/r/1963/diff/ which never land cause of missing time to polish means fix problems on my end). Maybe that was the reason QFileSystemWatcher was made threaded? In any case I agree it should be better fixed (and to my knowledge was) down the stack.

    ReplyDelete
  7. By reading the gio code I wonder if ReadDirectoryChangesW would be a more useful Win32 API than WaitForMultipleObjects: It seems to be asynchronous and will report what kind of change happend to a directory (file added, removed, modified, ...)...

    ReplyDelete