We only really need the luminance in the binary search, so skip constructing the color until just before returning.