Using MIPI-CSI-2 Peripherals with Wandboard EDM-G

Introduction

It would be hard to name an industry that does not use vision in some way. We depend on embedded vision processing for a myriad of many tasks.

One of the challenges faced in developing vision products is the way that the video stream from the camera is transmitted to the SOC. For most SOCs, including those from NXP, that interface is MIPI-CSI-2.

Another step to take is software integration. In general, MIPI-CSI-2 devices are not automatically recognized and enumerated in embedded systems. The embedded software platform has to be informed that a specific device (e.g. Camera, HDMI receiver) might be present on a specific MIPI-CSI-2 port, what it's I2C address is, what GPIO should be used to control the device, and so on. We have simplified this task using Linux device tree overlays, but the task remains that the developer must configure the system correctly.

In this article, we go through the steps to connect and integate a MIPI-CSI-2 camera and an HDMI adapter with Linux running our Wandboard EDM-G-IMX8M-PLUS.

What is MIPI-CSI-2?

The MIPI-CSI-2 interface (MIPI stands for "Mobile Industry Processor Interface" and CSI stands for "Camera Serial Interface 2") is the most widely used camera interface for embedded processors. It is a multi-lane high-speed serial interface, consisting of 1 to 4 (lanes) for camera data, and a single high-speed (differential) clock.

In order to support the continued demand for higher resolution image sensors, the MIPI-Alliance has revised the MIPI-CSI-2 specification to improve data throughput. The EDM-G family, supporting NXP's latest 64-bit processors, supports up to 2 MIPI-CSI-2 interfaces, with 4 differential data lanes each, at speeds of up to 1.5 Gbps, for a total throughput of around 6 Gbps.

In addition to the MIPI-CSI-2 interface, other signals between the processor and the video peripheral include:

• A bidirectional communications interface (usually I2C), used to initialize internal registers of the sensor and provide real-time status feedback to the SOC
• Usually at least 2 control signals:
  • An active low RESET signal (RESET_N), which resets the peripheral when low
  • An active-high power control signal (PWDN), which can power the peripheral down when low

Because MIPI-CSI-2 connections directly connect processors to the video source (e.g. camera sensor), when compared to other common interfaces used to connect processors with video peripherals, such as USB and Ethernet, MIPI-CSI-2 provides the lowest latency and highest video throughput.

MIPI-CSI-2 Products

With Wandboard EDM-G, we have introduced a new way to connect peripherals into the MIPI-CSI-2 interfaces of the module. With this interface, you can work with several different types of video inputs including:

• Embedded Cameras (TEVI series)
• VizionLink Cameras (via a VizionLink Adapter)
• VizionLink HDMI adapter

TEVI-OV5640 5MP Color Image Sensor with integrated Image Signal Processor (ISP)	TEVI-AR0521 5MP Color Image Sensor Kit with integrated Image Signal Processor (ISP), including Ribbon Cable	VizionLink-HDMI Adapter - HDMI-to-MIPI Receiver

These products are easily purchased separately, or with the WB-EDM-G development kit via our web shop.

MIPI-CSI-2 Connectors on Wandboard EDM-G

On Wandboard EDM-G, there are two MIPI-CSI-2 connectors. These are 70-pin board-to-board connectors from Hirose. Below is a picture of the board with the connectors shown. We refer to them as CAMERA1 and CAMERA2 (also as "CAM1" and "CAM2" for short).

CAMERA1 and CAMERA2 Support on Various EDM-G Kits

When using the EDM-G-IMX8M-PLUS, you can plug up to 2 cameras or adapters into the Wandboard EDM-G baseboard (supporting both CAMERA1 and CAMERA2). With EDM-G-IMX8M-MINI, you can use only one adapter or camera, as the i.MX8M Mini supports only 1 MIPI-CSI2 interface (CAMERA1 only).

Basically, what this means is that with EDM-G-IMX8M-PLUS, you can plug cameras or HDMI receivers into both CAMERA connectors. With EDM-G-IMX8M-MINI and EDM-G-IMX8M-NANO, the camera or adapter must be plugged into CAMERA1 connector only.

In the following video, we have shown most of the steps we go through in this article, so it would a great reference in case you get stuck or are looking for more information.

Getting Started Using TEVI-OV5640

Using the TEVI-OV5640 camera is a pretty simple process. This is because we enable the TEVI-OV5640 camera in the device tree by default. The following steps assume that you are running either Yocto or Ubuntu Linux.

Required Hardware:

• WB-EDM-G kit with EDM-G-IMX8M-PLUS/MINI/NANO
• TEVI-OV5640 Camera Kit
• HDMI Monitor and HDMI Cable
• USB Keyboard/Mouse
• Ethernet Cable with Internet Connection
• USB Type C cable (for power)

Step 1: Load the kit with a Linux image

You can use the TechNexion Software Loader (TSL) to load either a Yocto or Ubuntu image onto the kit. The TSL is a graphical program, so a display and keyboard/mouse will be needed to interact with it. With a few mouse clicks, you can easily download and install an image. The TSL will download the image from TechNexion's servers and install that image into the e.MMC of the module (or a microSD card).

Installing an Image using the TSL

To install an image using the TSL, you can follow these steps:

1. Connect HDMI monitor to the HDMI connector of the WB-EDM-G baseboard
2. Connect Ethernet cable to the RJ45 connector of the WB-EDM-G baseboard
3. Plug in USB connector for the USB keyboard and mouse
4. Power the board on with the USB Type C cable
5. Allow the board to boot. You will be presented with a blue TSL screen.
6. Select the options for the image that you wish to download (Yocto or Ubuntu) and start to install the image

Step 2) Connect the Camera

Once you have a unit loaded with a Linux image, you can power the off by disconnecting power. Connect the camera to the appropriate camera connector. The camera connector you use will determine the Linux V4L2 (Video4Linux2) device used.

The table below shows the camera connector mapping to the cooresponding Video4Linux2 capture device.

Once the camera is connected, you can power the unit back up. Once booted up, you can check to see if a camera has been connected correctly and initialized. You can list the v4l2 devices as below.

# ls /dev/video*
/dev/video0  /dev/video1 /dev/video2

If you see /dev/video0 or /dev/video1, you likely have a valid camera device working. In order to check further, you can use the v4l2-ctl command to list the devices.

# v4l2-ctl --list-devices
ev ():
        /dev/v4l-subdev0

ev ():
        /dev/v4l-subdev1

mxc-isi-cap (platform:32e00000.isi:cap_devic):
        /dev/video0

mxc-isi-m2m (platform:32e00000.isi:m2m_devic):
        /dev/video1

mxc-isi-cap (platform:32e02000.isi:cap_devic):
        /dev/video1

Look for the mxc-isi-cap devices as above. These indicate the devices that are capture devices (e.g. cameras). Note that /dev/video2 is also present in this example because in this configuration there are to camera devices populated on an EDM-G-IMX8M-PLUS platform.

Step 3) Start a Gstreamer pipeline

We'll use Gstreamer to display the camera feed. Keep note of the v4l2 device (/dev/video0, if the capture device is in CAMERA1, or /dev/video1 if the device is in CAMERA2). In the case below, the camera was plugged into CAMERA2, so we'll use /dev/video1:

# gst-launch-1.0 v4l2src device=/dev/video1 ! autovideosink

This will start up a pipeline that takes its source video from the camera (in CAMERA2) and displays it to the display using the autovideosink element. This is a quick way to check to see if your camera works.

Changing Resolution

To change the resolution of the camera capture, you can add a capsfilter to the pipeline. This informs the upstream gstreamer elements of the required resolution and framerate.

For full HD (1920x1080) resolution at 30 frames per second:

# gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,width=1920,height=1080,framerate=30/1 ! autovideosink

For 1280x720 resolution at 30 frames per second:

# gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,width=1280,height=720,framerate=30/1 ! autovideosink

For 640x480 resolution at 30 frames per second:

# gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480,framerate=30/1 ! autovideosink

Improving framerate

If you notice that the framerate reported by the pipeline is lower than the target framerate, it is possible that this can be improved using the imxvideoconvert_g2d element in the pipeline. This element will utilize the 2D GPU within the SOC to perform colerspace conversions instead of performing them in software running on the Cortex A53 CPUs.

For example (without imxvideoconvert_g2d):

# gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480,framerate=30/1 ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 1461.171238] bypass csc
[ 1461.173612] input fmt YUV4
[ 1461.176333] output fmt YUYV
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:08.532416838
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (182), playing for (0:00:08.532669704), fps (21.330).
Freeing pipeline ...

The framerate 21.3 fps while the target is 30 fps.

Example (with imxvideoconvert_g2d):

# gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480,framerate=30/1 ! imxvideoconvert_g2d ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 1627.690285] bypass csc
[ 1627.692651] input fmt YUV4
[ 1627.695376] output fmt YUYV
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:14.990066600
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (435), playing for (0:00:14.990336591), fps (29.019).
Freeing pipeline ...

The frame rate is much closer to the desired frame rate of 30 fps.

Getting Started using the VizionLink-HDMI Adapter

The VizionLink-HDMI adapter is a small PCB module that plugs into the camera slot on the WB-EDM-G development board. It features a Toshiba TC358743 HDMI to MIPI-CSI-2 bridge IC. This converts received HDMI into MIPI-CSI-2 which can be captured by the SOC on the EDM-G module.

Step 1) Connect the VizionLink-HDMI card

Connect the card to either CAMERA1 or CAMERA2. Both The are accompanied by soldered threaded mounting posts. Be careful to press the card connector onto the camera connector only when aligned, and use the provided screws to firmly fix the card to the baseboard. The screws are important to prevent the accidental unmating of the card which can easily occur when an attached HDMI cable is manipulated.

VizionLink-HDMI Card mated to CAMERA1 connector

Step 2) Make sure that Linux Kernel Supports VizionLink HDMI

In order to use VizionLink HDMI, the Linux Kernel needs to have specific patches that support the Toshiba HDMI to MIPI bridge IC (TC358743). In addition, a device tree overlay must be imported by U-boot and applied at boot time. This device tree overlay both enables the VizionLink HDMI card (or cards, depending on the overlay) and disables the TEVI-OV5640.

Here are the necessary patches. Releases after 17 June 2021 should incorporate these by default.

The following steps assume your kernel supports VizionLink-HDMI already.

Check to see if the dtbo exists in the boot filesystem (check /run/media/mmcblk2p1 or /run/media/mmcblk0p1 if booting from SD card).

Step 3) Enable the Device Tree Overlay (dtbo) in U-boot

For this step, you will need to have the USB-Serial adapter connected to the WB-EDM-G baseboard into the CONSOLE1 connector because you need to modify the U-boot environment.

Connecting the USB-Serial Cable

The picture below illustrates how to connect the USB-Serial cable to the CONSOLE1 port.

Connect the other end of the USB-serial cable to your host computer and use Putty, Minicom, Teraterm, screen, or other preferred terminal emulator you prefer. The default baud rate is 115200 baud, 8 bits, 1 stop bit, no parity, no hardware flow control.

Here is an example of connecting to the serial console using the 'screen' utility, which is present on most Linux hosts and Mac OS.

Linux Hosts
In Linux hosts, the default USB serial device is usually /dev/ttyUSB0 (or /dev/ttyUSB1, /dev/ttyUSB2, and so on depending on how many USB serial devices you have connected). Assuming you have just the one USB serial cable connected, it should be /dev/ttyUSB0, so:

$ sudo screen /dev/ttyUSB0 115200

By default, in Ubuntu hosts, normal users do not have access to the serial devices, so you'll need to prepend sudo to the command to run it with root permissions.

Mac OS Hosts
In Mac OS hosts, you'll need to discover the USB serial device name. This is easily done by just:

% ls /dev/tty.usbserial*
/dev/tty.usbserial-AL035SJ2

then, start screen:

% screen /dev/tty.usbserial-AL035SJ2 115200

Modifying the U-boot environment to configure the dtoverlay environment variable

We will now go through the process of setting the dtoverlay environment variable in U-boot.

Reboot the board by pressing the reset button. This is located just below the CONSOLE1 port connector, and above the speaker connectors.

In the serial console window, watch for the messages after the reboot that prompt you to hit any key to stop autoboot, and press a key to halt the boot process and get a U-boot prompt:

U-Boot 2020.04-235350493ba888d2340e29b6ec942314f2792761+g235350493b (May 08 2021 - 15:33:16 +0000)

CPU:   i.MX8MP[8] rev1.1 1800 MHz (running at 1200 MHz)
CPU:   Commercial temperature grade (0C to 95C) at 44C
Reset cause: POR
Model: TechNexion EDM-G-IMX8MP
DRAM:  4 GiB
MMC:   FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
Can't find cec device id=0x3c
[*]-Video Link 1probe video device failed, ret -2

        [0] lcd-controller@32e80000, video
        [1] mipi_dsi@32e60000, video_bridge
        [2] rm67191_panel, panel
probe video device failed, ret -2
In:    serial
Out:   serial
Err:   serial

 BuildInfo:
  - ATF f1d7187
  - U-Boot 2020.04-235350493ba888d2340e29b6ec942314f2792761+g235350493b

switch to partitions #0, OK
mmc2(part 0) is current device
flash target is MMC:2
Net:   No ethernet found.
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot:  0 
u-boot=> 

Next, you need to get a listing of the .dtbo files in the boot filesystem. This is a small FAT (File Allocation Table) filesystem that is used to store the kernel image (Image), device tree blobs (.dtb), and other files needed by U-boot.

You can do this by first making sure of the correct mmc device:

u-boot=> mmc list
FSL_SDHC: 1
FSL_SDHC: 2 (eMMC)

In this case, we are using the onboard e.MMC and the correct mmc device is 2.

u-boot=> fatls mmc 2
 22374408   Image
     8208   hello_world.bin
     2155   imx8mp-edm-g-wb-lvds-vl10112880.dtbo
     2155   imx8mp-edm-g-wb-lvds-vl15613676.dtbo
     2184   imx8mp-edm-g-wb-lvds-vl215192108.dtbo
    79010   imx8mp-edm-g-wb-rpmsg.dtb
    81519   imx8mp-edm-g-wb.dtb
    19040   rpmsg_lite_pingpong_rtos_linux_remote.bin
    18528   rpmsg_lite_str_echo_rtos.bin
    40948   sai_low_power_audio.bin
   420310   splash.bmp
     4038   imx8mp-edm-g-wb-hdmi2mipi-tc358743-dual.dtbo
     2348   imx8mp-edm-g-wb-hdmi2mipi-tc358743-single-cam1.dtbo

13 file(s), 0 dir(s)

If you are using 2 VizionLink HDMI adapters, then the correct .dtbo file is imx8mp-edm-g-wb-hdmi2mipi-tc358743-dual.dtbo.

If you're using just one VizionLink-HDMI adapter, the correct file is imx8mp-edm-g-wb-hdmi2mipi-tc358743-single-cam1.dtbo. In that case, the VizionLink-HDMI adapter must be plugged into CAMERA1.

Now, let's take that information and use it to set the dtoverlay environment variable. When setting this variable, you don't need the prefix information that contains the name of the platform (this example, the platform is imx8mp-edm-g) or the baseboard (wb). You also don't need to add the suffix (.dtbo). There is a macro that constructs the entire filename so that it can be loaded. For your information, that macro is loadoverlay and you can see this by printing it out:

u-boot=> printenv loadoverlay
loadoverlay=fdt addr ${fdt_addr} && fdt resize ${fdt_buffer}; setexpr fdtovaddr ${fdt_addr} + 0xF0000; for ov in ${dtoverlay}; do echo Overlaying ${ov}...; fatload mmc ${mmcdev}:${mmcpart} ${fdtovaddr} imx8mp-edm-g-${baseboard}-${ov}.dtbo && fdt apply ${fdtovaddr}; done

The operative part of that macro is this: imx8mp-edm-g-${baseboard}-${ov}.dtbo where baseboard is wb and ov is the unique part of the .dtbo filename.

Thus, for a dual VizionLink-HDMI configuration, you set the dtoverlay variable as follows:

u-boot=> setenv dtoverlay hdmi2mipi-tc358743-dual
u-boot=> saveenv

And for a single VizionLink-HDMI configuration, where the adapter is plugged into CAMERA1:

u-boot=> setenv dtoverlay hdmi2mipi-tc358743-single-cam1
u-boot=> saveenv

Running saveenv will save the environment to e.MMC.

Now you can reboot the board using the reset button or simply boot using the U-boot boot command:

u-boot=> boot  
switch to partitions #0, OK
mmc2(part 0) is current device
22374408 bytes read in 518 ms (41.2 MiB/s)
Booting from mmc ...
Loading fdt_file imx8mp-edm-g-wb.dtb...
81519 bytes read in 16 ms (4.9 MiB/s)
Overlaying hdmi2mipi-tc358743-single-cam1...
2348 bytes read in 14 ms (163.1 KiB/s)
## Flattened Device Tree blob at 43000000
   Booting using the fdt blob at 0x43000000
   Using Device Tree in place at 0000000043000000, end 000000004301ffff
probe video device failed, ret -2

If you are successful in setting the dtoverlay variable, you will see:

...
Overlaying hdmi2mipi-tc358743-single-cam1...
...

In the single VizionLink-HDMI case, or

...
Overlaying hdmi2mipi-tc358743-dual...
...

in the dual VizionLink-HDMI camera case.

Allow the board to boot completely so that you see an HDMI monitor output.

Check to see if the VizionLink-HDMI device (or devices) were recognized and the drivers loaded

root@edm-g-imx8mp:~# dmesg | grep tc358743 
[    1.913139] imx-tc358743 sound-tc358743: clock not found.
[    1.918571] imx-tc358743 sound-tc358743: ASoC: failed to init link hdmi-in: -517
[    1.925975] imx-tc358743 sound-tc358743: snd_soc_register_card failed (-517)
[    2.475676] tc358743 1-000f: tc358743 found @ 0x1e (30a30000.i2c)
[    3.115374] mx8-img-md: Registered sensor subdevice: tc358743 1-000f (1)
[    3.148476] mx8-img-md: created link [tc358743 1-000f] => [mxc-mipi-csi2.0]
[    3.185977] imx-tc358743 sound-tc358743: clock not found.
[    3.199753] imx-tc358743 sound-tc358743: tc358743.1-000f <-> 30c20000.sai mapping ok

If you're working with two VizionLink-HDMI adapters, you should see another adapter registered like this:

...
mx8-img-md: Registered sensor subdevice: tc358743 4-000f (1)
...

Step 4) Connect a Source and Capture HDMI Video

Now that we have our HDMI adapters set up and the drivers loaded, it is time to connect up an HDMI source to the adapter and see if we can capture some video.

First, it should be noted that there are some limitations of the VizionLink-HDMI adapter in terms of the resolutions that it can manage. It can manage the following resolutions:

Make sure that the HDMI source is providing video at one of the above resolutions, otherwise the pipeline may fail.

Start a Gstreamer Pipeline to Capture and Display the Video

Make sure that you're using the right V4L2 device:

Assuming that the output HDMI video is a 1080P video stream, you can use the following gstreamer pipeline (adapter plugged into CAMERA1 slot):

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080 ! autovideosink

A note on the setting of the capsfilter (...! video/x-raw,width=1920,height=1080 !...). The framerate parameter is excluded. This is different than when running a pipeline capturing video from a camera. In that in the case of camera capture, the SOC can command a certain frame rate and set up the pipeline accordingly. When receiving HDMI data, the SOC cannot dictate the frame rate as this is left up to the transmitter, so this is left out. Assigning framerate=60/1, for example, will cause the pipeline to fail.

Possible Pitfalls: Resolution Matching

If the resolution of the HDMI source is different than that of the capsfilter, then you may run into the following error:

root@edm-g-imx8mp:~# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080  ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 2393.678369] isi-capture 32e00000.isi:cap_device: mxc_isi_source_fmt_init: src:(1280,720), dst:(1920,1080) Not support upscale
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.
Additional debug info:
../../../git/sys/v4l2/gstv4l2src.c(660): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Buffer pool activation failed
Execution ended after 0:00:00.053696843
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

This is a limitation of the SOC in that the Image Sensor Interface (ISI) driver does not support upscaling to a different target resolution.

The upshot, make sure that your capsfilter is set to the same resolution as the source.