systemdialog: also consider buttonbox for dimensions

otherwise the dialog may be shrunk beyond the space that the buttonbox
desires resulting in cut off buttons (only when the mainItem is small
enough, naturally)


(cherry picked from commit 1faeaf2e)