Remove the filter colors functionality in in the color popup

This should fix a crash due to index not being correct

Also make sure we actually intialize with a palette

Okay to backport?

BUG:288071