Notifier: Fix notification logic

When the notification has never been issued, the time is invalid and
we'll want to send a notification.

Thanks Adam Williamson for pointing it out!