encode: Only increase filtered frame count if frame was actually submitted

In some cases we can end up with a frame with no data. In that case, we
would try to submit the frame to the encoder, which would then reject
it. We'd still increase the pending filtered frame count however, but
never end up reducing the count. So the pending filtered frame count
would increase until we'd start dropping frames - including frames with
actual data.

To resolve this, make Encoder::filterFrame return false if we reject the
frame for any reason and only increase pending filtered frame count if
filterFrame returns true.


(cherry picked from commit 50fcd65a)