Keep strong references when using pointers

It doesn't make much sense to keep the raw pointer of a weak pointer
that could potentially disappear at any point.

This fixes warnings and just uses references like they're meant to be
used.