Only load one plugin per type

If we're installed in multiple prefixes (ie self built vs system packages)
only load the proper one which is from the prefix of the self built one