Go back to constructing two temporary sets

Better to keep the exact semantics when porting stuff, we can change it later
if we want. (Disregarding that I even got the boolean the wrong way around).