use a set instead of vector for performance reasons

Lookup cost goes down from O(log n) to O(1). Not much gain in reality
though.