screencast: Fix format announced with SHM enum format

Kwin announces a format with alpha when Dma-Bufs are available, even
when the texture doesn't contain any. This results in clients segfault
when trying to access the buffer assuming a maxsize wrt. the announced
format by ways of dimensions and bpp.

This patch moves the format masking to affect the Dma-Buf transport
only.