dialog: Make sure that max and min sizes are always consistent

In Qt6 changing minimum and maximum sizes immediately changes
the window size. This can lead to situations where upon resize
the new minimum size is bigger than the current maximum size or
the new maximum size smaller than the current minimum one.
Going down this code path asserts.
To avoid this, first set the maximum to the maximum of all sizes
then the new minumum size and finally the new maximum.