Avoid destroying yet alive color devices

This is a typo that I forgot to fix, therefore not requesting code
review. After QVector::erase() has been called, we cannot use the
iterator as it will lead to undefined behavior.