Fix watching for files that are removed and then re-added

We need to remove removed files from the set of seen paths. Otherwise,
those files will be ignored when they are re-added, e.g. when a secret
key is first removed and then re-imported.

GnuPG-bug-id: 5546