Fix result of resolution if mixed protocols are needed but not allowed

If mixed protocols are not allowed, but for some recipients there are
only OpenPGP keys and for others there are only S/MIME certificates,
then the resolution should fail.

GnuPG-bug-id: 5283