Fixed a subpixel precision calculation of mask sizes

Now the size of the mask is calculated depending on the fractional offset.
This is the second version of a commit 2c30521e which was reverted due to
the problems with the autobrush. Now the auto brush calculates the center
of the  dab correctly using the hotspot information which gives the most
stable and correct results.

BUG:323200