Fix navigation with side mouse buttons

!206 breaks Next/Previous navigation using side mouse buttons. It still
works as expected when the cursor is over the toolbar, but if it's over
the image display area, the clicks are ignored.

This fixes it, and preserves the improved middle click behavior.


(cherry picked from commit fcd3e8c7)