Always go back/forward in history when pressing the respective buttons

Before this patch, pressing one of these buttons while an item is
hovered selected this item. The motivation for this behavior was to
provide a fast way to select items. However, this was counter-intuitive
and confusing for many users.

BUG: 310288
FIXED-IN: 4.13.0
REVIEW: 116469