Remove the vtable hierarchy in the header private classes.

This requires a bit of extra care in the dtors, but it saves 20-30% of
memory per instance.