record: Check VAAPI size constraints when initializing hardware encoding

If we exceed hardware constraints VAAPI will refuse to encode, so we
need to make sure we can fall back to software encoding in such a case.