Clean up listener

If the style is unloaded for example by an application dynamically
switching styles, the listener was still around leading us to call
into destroyed objects when the color scheme was changed.
BUG:432660


(cherry picked from commit 5de36bd1)