Qt Multimedia GStreamer backend | Qt Multimedia | Qt 6.12.0
Description: Platform notes for the GStreamer backend

- Qt 6.12
- Qt Multimedia
- Qt Multimedia GStreamer backend
- Qt 6.12.0 Reference Documentation

Platform Notes - Linux ◦ Advanced FFmpeg Configuration

# Qt Multimedia GStreamer backend

This page covers limitations and customization points of the GStreamer backend of Qt Multimedia, the default media backend on embedded Linux.

## Architectural Considerations

GStreamer is the default media backend on embedded Linux mainly because it is the framework most embedded board vendors prioritize for hardware accelerated media support. FFmpeg may also be suitable on some platforms, so users are encouraged to experiment with both media backends to find out which works best for them.

Qt Multimedia is not a general purpose streaming framework and not necessarily the architecturally best way to use GStreamer with Qt. Developers, who need a high degree of control over the GStreamer pipeline, but only want to show the video output Qt, may want to consider using GStreamer's qml6glsink.

## Limitations and Known Issues

GStreamer is not bundled with Qt, but it is typically deployed with the Linux distribution.

- Certain bugs may be due to the GStreamer version being used. We recommend to use the latest GStreamer bug fix release on your platform.
- Certain bugs may also be related to the libraries used by GStreamer, like Pulseaudio. Most notably Pulseaudio v16 has a known bug that causes the GStreamer pipeline to hang and requires backports of these two patches:
  - https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/745
  - https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/764

  This bug currently affects most mainstream Linux distributions, including Ubuntu 22.04, 23.10 and 24.04, Debian 11 and 12, as well as Fedora 39 and 40.

- Switching camera devices or cameras during recording is not supported
- Seeking, playback rates, loop, switching sinks have known bugs.
- Audio features requires PulseAudio. See Linux platform notes for details.
- Some embedded boards needs individual configurations to solve pipeline negotiation/linking issues and improve performance. We recommend developers to experiment with the environment variables listed under Customization points when facing such problems. Qt's Yocto meta layer files contain recommended configurations for certain combinations of boards, BSPs and Boot to Qt versions. Here are links to the dev branch versions:
  - NXP i.MX 6/8/9: https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/freescale-layer/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev
  - Raspberry Pi 4/5: https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/raspberrypi/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev

  Some NXP i.MX boards expose a v4l2 device named imx-capture, which can cause errors when used with v4l2src in GStreamer. To prevent Qt Multimedia from adding imx-capture to the list of available camera devices, set the environment variable QT_GSTREAMER_SKIP_IMXCAPTURE=1. Note that this is considered as private API which will be removed when NXP fixes the problems with imx-capture.

- Video playback or camera streaming using system memory video frames or software conversion can result in dropped frames and unresponsiveness due to heavy CPU load. If the GStreamer decoder or camera device outputs DMA-BUF- or GLMemory-backed video frames in one of our supported pixel formats, Qt can render them without additional conversion or copying. GStreamer selects a format and memory type based on the capabilities of its pipeline elements. Different ways of controlling which elements are added to the pipeline are described under Customization points. See also Specifying I/O mode for v4l2 cameras.

## Customization points

Qt Multimedia provides certain customization points to allow access to the underlying GStreamer pipeline. The entry point is `class QGStreamerPlatformSpecificInterface`. Additional customizations can be done by setting environment variables.

Warning: The customization points and Qt specific environment variables listed here, are considered as private APIs and may be subject to change. Be aware that environment variables are by default inherited by child processes and can have unintended consequences.

### External OpenGL texture rendering

Qt imports DMA-BUF-backed video frames into the OpenGL graphics pipeline by binding the underlying buffer as an EGL image and exposing it to shaders as an OpenGL texture. By default, Qt samples these textures using the GL_TEXTURE_2D texture target and standard shader code. Some embedded boards performs better with the GL_TEXTURE_EXTERNAL_OES texture target instead, together with a dedicated fragment shader. This behavior can be enabled by setting the following environment variable:

     QT_MULTIMEDIA_FORCE_GL_TEXTURE_EXTERNAL_OES=1

### Specifying I/O mode for v4l2 cameras

When using a v4l2 capture device with a QCamera or QML Camera, the io-mode property of the underlying v4l2src GStreamer element affects whether or not it outputs DMA-BUF-backed video frames that allows zero-copy rendering. In some cases the default auto mode chooses mmap mode even though the camera driver supports dmabuf mode. The io-mode property int value can be set explicitly via the environment variable below. Here's an example showing how to set io-mode to dmabuf with the enum GST_V4L2_IO_DMABUF:

     QT_GSTREAMER_V4L2SRC_IOMODE=4

### Selecting specific GStreamer elements

A GStreamer distribution can include multiple different plugins that can potentially perform the same task in the media pipeline. GStreamer automatically selects elements like decoders and demuxers based on their feature rank, which can be overridden via a GStreamer environment variable. Here's an example that ranks the v4l2h264dec hardware decoder higher than any H.264 software decoder, while also making sure the aiurdemux demuxer isn't selected:

     GST_PLUGIN_FEATURE_RANK=v4l2h264dec:MAX,aiurdemux:NONE

To inspect which elements are being created during runtime, enable debug logging level 4 for GstElementFactory:

     GST_DEBUG=GST_ELEMENT_FACTORY:4

Furthermore, you can visually inspect the whole GStreamer pipeline by specifying a folder for `.dot` graph files that will be dumped at various events, for example, changes in playback state:

     GST_DEBUG_DUMP_DOT_DIR=/root/graphs

Read more about these environment variables in the GStreamer documentation:

- Environment variables - GST_PLUGIN_FEATURE_RANK
- Basic tutorial - The debug log
- Basic tutorial - Getting pipeline graphs

#### Specifying hardware conversion elements

If the media source of a GStreamer pipeline cannot provide video buffers in a pixel format supported by Qt Multimedia, the pipeline will use software conversion unless it contains a suitable hardware accelerated video conversion element. To specify a hardware conversion element to be used, set it as a GStreamer pipeline description to the following environment variable:

     QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT

Some vendor specific conversion elements (imxvideoconvert_g2d, nvvidconv) are added to the pipeline by default if available. This can cause unnecessary conversion if they aren't needed, and can be disabled by specifying a GStreamer identity element using the same environment variable:

     QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=identity

GStreamer can also perform format conversion with OpenGL elements, providing Qt with GLMemory-backed video frames:

     QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=glupload ! glcolorconvert

### Raw pipeline access

The `GstPipeline` underlying the QMediaPlayer and QMediaCaptureSession can be accessed.

Warning: This is an unsafe API, as the pipeline is still managed by the Qt implementation. Great care is required when using this API.

     #include <QtMultimedia/private/qgstreamer_platformspecificinterface_p.h>

     [...]
     QMediaPlayer player;
     GstPipeline *pipeline = QGStreamerPlatformSpecificInterface::instance()->gstPipeline(&player);
     [...]
     QMediaCaptureSession session;
     GstPipeline *pipeline = QGStreamerPlatformSpecificInterface::instance()->gstPipeline(&session);

### Custom GStreamer elements as sinks and sources

It is possible to create GStreamer elements from a GStreamer pipeline description and wrap them inside a `QCamera` or `QAudioDevice`:

     #include <QtMultimedia/private/qgstreamer_platformspecificinterface_p.h>

     [...]
     QByteArray pipelineString = "videotestsrc is-live=true ! gamma gamma=2.0";

     QMediaCaptureSession session;
     session.setVideoSink(wid.videoSink());

     QCamera *cam = QGStreamerPlatformSpecificInterface::instance()->makeCustomGStreamerCamera(
              pipelineString, &session);
     session.setCamera(cam);

### QMediaPlayer: custom sources

The `QMediaPlayer` accepts a GStreamer pipeline description as source URI:

     QMediaPlayer player;
     player.setSource(u"gstreamer-pipeline: videotestsrc name=testsrc"_s);

This will try to compile the pipeline description to use as source in the QMediaPlayer and will be automatically connected to the sinks of the QMediaPlayer.

Warning: Hic sunt dracones! Custom pipelines are an experimental feature: the custom pipelines do not map well to QMediaPlayer APIs, most notably the media status, metadata APIs, and transport state. Most calls will directly map to the GStreamer pipeline, which can lead to undefined behavior depending on the pipeline. In most cases, the `gstreamer-pipeline:` may not be the right choice for application code: for arbitrary video sources, the `QMediaCaptureSession` with a custom camera (see above) is the preferred choice. For arbitrarily complex pipelines that only want to draw into a Qt/QML GUI, GStreamer's `qml6glsink` (see below) may be a more robust choice.

Platform Notes - Linux ◦ Advanced FFmpeg Configuration