backends: Drop waylandServer dependency

Backends dependending on the wayland server is not great, it makes them
less reusable and creates a strict instantion order that doesn't fully
make sense.