backends/drm: don't cache formats

While it could be useful with tiled displays, the isFormatSupported and
supportedModifier functions can be called before prepareModeset, so where
m_formats is still empty. Additionally they're neither in a hot path nor
performance critical.