[Notifications] When application responds to cancel request, stop kill timer

This ensures that an error message the app might have sent in response to the
cancel request is still shown.

Also, while at it deduplicate code by calling into `terminate` from the
service watcher.