Play Video Using Gstreamer

Introduction

Playback of compressed video can be much more efficient in both CPU and thus power consumption when using the on-board video decoder built into i.MX8M and i.MX8M Mini. This article shows how to take advantage of this capability.

We will assume that you are running one of our Yocto demo images available from our download server:

For i.MX8M:
PICO-IMX8MQ
EDM-IMX8MQ

For i.MX8M Mini:
PICO-IMX8M-MINI
AXON-IMX8M-MINI
FLEX-IMX8M-MINI
XORE-IMX8M-MINI

Download Example Movies

Movies are available in a variety of formats and resolutions are freely downloadable from the Peach Open Movie Project. The popular movie "Big Buck Bunny" has been around for quite a long time and is available for download here:

https://download.blender.org/demo/movies/BBB/

For the purposes of this article we are going to focus on playback of .mp4 files.

Using a USB Flash Drive

In most cases, our Yocto images come with about 10% extra space in the root filesystem, which is not enough if you want to download larger movies for playback. You can also download files to a USB flash drive and use that as extra storage on your target device. You can download them to the drive using your host computer and plug that into a USB Type A port on the development kit, or you can simply plug in a USB flash drive into the USB Type A port on the kit. In most images, this will be detected and automatically mount to the directory /run/media/sda1. You can then download a movie to the board using the wget command as follows:

For a full HD movie (1080P30):

    # cd /run/media/sda1
    # wget https://download.blender.org/demo/movies/BBB/bbb_sunflower_1080p_30fps_normal.mp4

For 4K video (2016P30): (For playback on i.MX8M at full resolution, if you have a 4K display attached to the HDMI port):

    # cd /run/media/sda1
    # wget https://download.blender.org/demo/movies/BBB/bbb_sunflower_2160p_30fps_normal.mp4

Playback on i.MX8M and i.MX8M Mini

Playback on i.MX8M using HDMI

Playback 1080P video using the playbin element

This element sets up a complete playback pipeline, if possible, to playback both the video and audio in the movie at the same time. When using an HDMI display, audio is routed over the HDMI connection to the display.

    $: gst-launch-1.0 playbin uri=file:/run/media/sda1/bbb_sunflower_1080p_30fps_normal.mp4
    Setting pipeline to PAUSED ...
    Pipeline is PREROLLING ...

    ====== AIUR: 4.4.5 build on Feb 13 2020 14:23:29. ======
        Core: MPEG4PARSER_06.16.01  build on Dec 11 2018 03:04:17
     file: /usr/lib/imx-mm/parser/lib_mp4_parser_arm_elinux.so.3.2
    ------------------------
        Track 00 [video_0] Enabled
        Duration: 0:10:34.533333000
        Language: und
        Mime:
        video/x-h264, parsed=(boolean)true, alignment=(string)au, stream-format=(string)avc, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, codec_data=(buffer)01640029ffe1001b67640029acca501e0089f970110000030001000003003c8f18319601000568e93b2c8b 
    ------------------------
    ====== VPUDEC: 4.4.5 build on Feb 13 2020 14:23:47. ======
        wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Dec  3 2019 23:12:15)
        vpulib: 1.1.1
        firmware: 1.1.1.65535
    ------------------------
        Track 01 [audio_0] Enabled
        Duration: 0:10:34.200000000
        Language: und
        Mime:
        audio/mpeg, mpegversion=(int)1, channels=(int)2, rate=(int)48000, bitrate=(int)160000 
    ------------------------
    ------------------------
        Track 02 [audio_1] Enabled
        Duration: 0:10:34.143333000
        Language: und
        Mime:
        audio/x-ac3, channels=(int)6, rate=(int)48000, bitrate=(int)0, framed=(boolean)true 
    ------------------------
    Missing element: AC-3 (ATSC A/52) decoder
    WARNING: from element /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0: No decoder available for type 'audio/x-ac3, framed=(boolean)true, rate=(int)48000, channels=(int)6, alignment=(string)frame'.
    Additional debug info:
    ../../../git/gst/playback/gsturidecodebin.c(921): unknown_type_cb (): /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0

    ====== BEEP: 4.4.5 build on Feb 13 2020 14:23:35. ======
        Core: MP3 decoder Wrapper  build on Jan 11 2018 10:20:25
     file: /usr/lib/imx-mm/audio-codec/wrap/lib_mp3d_wrap_arm_elinux.so.3
    CODEC: BLN_MAD-MMCODECS_MP3D_ARM_02.13.01_ARMV8  build on Jan 11 2018 10:05:45.

    ===!!! Current pulsesink device is alsa_output.platform-sound-hdmi.stereo-fallback !!!===

    Redistribute latency...
    Pipeline is PREROLLED ...
    Setting pipeline to PLAYING ...
    New clock: GstPulseSinkClock
    ^Chandling interrupt.
    Interrupt: Stopping pipeline ...
    Execution ended after 0:00:18.976329498
    Setting pipeline to PAUSED ...
    Setting pipeline to READY ...
    Total showed frames (572), playing for (0:00:18.984334000), fps (30.130).
    Setting pipeline to NULL ...
    Freeing pipeline ...

Playback 4K video using the playbin element

    $: gst-launch-1.0 playbin uri=file:/run/media/sda1/bbb_sunflower_2160p_30fps_normal.mp4
    Setting pipeline to PAUSED ...
    Pipeline is PREROLLING ...

    ====== AIUR: 4.4.5 build on Feb 13 2020 14:23:29. ======
        Core: MPEG4PARSER_06.16.01  build on Dec 11 2018 03:04:17
     file: /usr/lib/imx-mm/parser/lib_mp4_parser_arm_elinux.so.3.2
    ------------------------
        Track 00 [video_0] Enabled
        Duration: 0:10:34.533333000
        Language: und
        Mime:
        video/x-h264, parsed=(boolean)true, alignment=(string)au, stream-format=(string)avc, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, codec_data=(buffer)01640033ffe1001b67640033acca500f0010fb0110000003001000000303c8f183196001000568e93b2c8b 
    ------------------------
    ====== VPUDEC: 4.4.5 build on Feb 13 2020 14:23:47. ======
        wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Dec  3 2019 23:12:15)
        vpulib: 1.1.1
        firmware: 1.1.1.65535
    ------------------------
        Track 01 [audio_0] Enabled
        Duration: 0:10:34.200000000
        Language: und
        Mime:
        audio/mpeg, mpegversion=(int)1, channels=(int)2, rate=(int)48000, bitrate=(int)160000 
    ------------------------
    ------------------------
        Track 02 [audio_1] Enabled
        Duration: 0:10:34.143333000
        Language: und
        Mime:
        audio/x-ac3, channels=(int)6, rate=(int)48000, bitrate=(int)0, framed=(boolean)true 
    ------------------------
    Missing element: AC-3 (ATSC A/52) decoder
    WARNING: from element /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0: No decoder available for type 'audio/x-ac3, framed=(boolean)true, rate=(int)48000, channels=(int)6, alignment=(string)frame'.
    Additional debug info:
    ../../../git/gst/playback/gsturidecodebin.c(921): unknown_type_cb (): /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0

    ====== BEEP: 4.4.5 build on Feb 13 2020 14:23:35. ======
        Core: MP3 decoder Wrapper  build on Jan 11 2018 10:20:25
     file: /usr/lib/imx-mm/audio-codec/wrap/lib_mp3d_wrap_arm_elinux.so.3
    CODEC: BLN_MAD-MMCODECS_MP3D_ARM_02.13.01_ARMV8  build on Jan 11 2018 10:05:45.

    ===!!! Current pulsesink device is alsa_output.platform-sound-hdmi.stereo-fallback !!!===

    Redistribute latency...
    Pipeline is PREROLLED ...
    Setting pipeline to PLAYING ...
    New clock: GstPulseSinkClock
    ^Chandling interrupt.
    Interrupt: Stopping pipeline ...
    Execution ended after 0:00:31.908833738
    Setting pipeline to PAUSED ...
    Setting pipeline to READY ...
    Total showed frames (839), playing for (0:00:31.913177000), fps (26.290).
    Setting pipeline to NULL ...
    Freeing pipeline ...

Playback on i.MX8M and i.MX8M Mini Using 5" MIPI-DSI Panel

Adjust Screen Rotation to Landscape Mode

When using the 5" MIPI panel, you may want to change the screen orientation of the Weston display manager so that it is in landscape mode (1280x720). The default mode for the 5" MIPI display is portrait mode, as the panel itself is a portrait mode panel (720x1280).

Change Screen Rotation in Wayland/Weston

Playback video only in fullscreen

Then, you can run the following commands. Note that this will only play the video portion of the file. It will not play the audio portion.

# EXPORT TEST_FILE=/run/media/sda1/bbb_sunflower_1080p_30fps_normal.mp4
# gst-launch-1.0 filesrc location=$TEST_FILE typefind=true \
! video/quicktime ! aiurdemux ! queue max-size-time=0 \
! vpudec ! waylandsink window-width=1280 window-height=720 async=false enable-last-sample=false qos=false sync=true

The window parameters for the waylandsink element (window-width=1280 window-height=720) are to adjust the position of the video so that it corresponds to the display resolution.

Playback video only, omitting window size parameters of 'waylandsink'

If you omit the window size parameters, you will see video that is offset on the screen and scaled to the window size. The following pipeline:

    # EXPORT TEST_FILE=/run/media/sda1/bbb_sunflower_1080p_30fps_normal.mp4
    # gst-launch-1.0 filesrc location=$TEST_FILE typefind=true \
    ! video/quicktime ! aiurdemux ! queue max-size-time=0 \
    ! vpudec ! waylandsink async=false enable-last-sample=false qos=false sync=true

Will result in something that looks like this:

Playback in fullscreen with both audio and video using playbin element

If you want to playback using the full screen, and also have audio playback as well, you can use the playbin element but explicitly define its video sink as follows:

    # EXPORT TEST_FILE=/run/media/sda1/bbb_sunflower_1080p_30fps_normal.mp4
    # gst-launch-1.0 playbin video-sink="waylandsink window-width=1280 window-height=720" uri=file:$TEST_FILE