[Notifications] Don't add IDs to be removed multiple times

In case an application happens to issue multiple close commands in the
time between we record the first close and it actually being processed.