Forums » Software Development »
Playing MP4 to Display Troubles
Added by Holden Wozniak 1 day ago
I have been trying to output MP4 video to the display using the hardware encoder/decoder. For the sake of consistency, I am using the 10.01 SDK prebuilt tisdk-default-image and making changes in ti-linux-6.6.y. This image provides the gstreamer plugin for hardware encoding/decoding:
gst-inspect-1.0 | grep v4l2
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2h264dec: V4L2 H264 Decoder
video4linux2: v4l2h264enc: V4L2 H.264 Encoder
video4linux2: v4l2h265dec: V4L2 H265 Decoder
video4linux2: v4l2h265enc: V4L2 H.265 Encoder
video4linux2: v4l2jpegenc: V4L2 JPEG Encoder
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2src: Video (video4linux2) Source
Right now I have a test mp4 (I am attaching to this post) with metadata:
gst-discoverer-1.0 bunny.mp4
Analyzing file:///root/bunny.mp4
[ 312.173582] vdec 30210000.video-codec: wave5_vpu_firmware_command_queue_error_check: result not ready: 0x800
[ 312.183539] vdec 30210000.video-codec: wave5_vpu_firmware_command_queue_error_check: result not ready: 0x800
[ 312.193636] vdec 30210000.video-codec: wave5_vpu_firmware_command_queue_error_check: result not ready: 0x800
Done discovering file:///root/bunny.mp4
Properties:
Duration: 0:00:10.000000000
Seekable: yes
Live: no
container #0: Quicktime
video #1: H.264 (High Profile)
Stream ID: 81fb72ed68904984fd3dc10b065ac836a5c1e6f9906631cfac417b26f230ed46/001
Width: 1280
Height: 720
Depth: 24
Frame rate: 30/1
Pixel aspect ratio: 1/1
Interlaced: false
Bitrate: 8171792
Max bitrate: 0
The problem I am running into is that any pipeline that outputs the video to a display is very slow and laggy. What I have noticed is that the video-codec driver gets brought up but falls back to polling. I tried running a pipeline to decode the frames and save them using the hardware encoder and that runs very slowly with high CPU usage (~90%). Here are the dmesg logs for the video codec:
[ 7.743861] vdec 30210000.video-codec: error -ENXIO: IRQ index 0 not found
[ 7.757086] vdec 30210000.video-codec: failed to get irq resource, falling back to polling
[ 7.780006] vdec 30210000.video-codec: OPP table not found in device tree
[ 7.841206] vdec 30210000.video-codec: Added wave5 driver with caps: 'ENCODE' 'DECODE'
[ 7.853338] vdec 30210000.video-codec: Product Code: 0x521c
[ 7.863120] vdec 30210000.video-codec: Firmware Revision: 334314
[ 312.173582] vdec 30210000.video-codec: wave5_vpu_firmware_command_queue_error_check: result not ready: 0x800
[ 312.183539] vdec 30210000.video-codec: wave5_vpu_firmware_command_queue_error_check: result not ready: 0x800
[ 312.193636] vdec 30210000.video-codec: wave5_vpu_firmware_command_queue_error_check: result not ready: 0x800
Pipeline: gst-launch-1.0 filesrc location=bunny.mp4 ! qtdemux ! h264parse ! queue ! v4l2h264dec ! videoconvert ! waylandsink
I noticed that k3-am62p-main.dsti has an interrupt defined for the video-codec that is not present in k3-am62a-main.dsti despite having the same driver, though I am unsure if defining an interrupt would fix it. I tried the interrupt used in k3-am62p-main.dsti which gets rid of the IRQ errors but causes the pipeline to hang on "redistributing latency" and I get stuck and can't quit out of the process without hard rebooting. I am not able to find anywhere that states the AM62A should be configured with the interrupt despite the codec driver throwing an error because it is not present. I'm guessing the error making it fall back to software encoding/decoding which would explain why it cannot keep up. Any guidance on this would be greatly appreciated.
Thanks,
Holden
Replies (4)
RE: Playing MP4 to Display Troubles - Added by Jake Pregitzer 1 day ago
Good afternoon Holden,
I am looking into your issue now and have replicated some of your failing behavior. I will update you when I have further info/findings.
Thank you,
Jake
RE: Playing MP4 to Display Troubles - Added by Michael Williamson 1 day ago
Hi Jake and Holden,
I think the issue may be related to using weston/wayland instead of drm/kms.
The AM62Ax does not have a GPU, so weston/wayland rendering is going to be
CPU bound.
Do you need weston/wayland?
-Mike
RE: Playing MP4 to Display Troubles - Added by Holden Wozniak 1 day ago
Hi Jake and Mike,
We do not need Weston/Wayland specifically, but we have a legacy app that heavily depends on X11 which might cause some difficulty using drm/kms. The same issue is present using X11 (using ximagesink instead of waylandsink). After some reading it seems like there might be a way to switch between virtual terminals to output to kmssink while the display server is still running though I have not been able to find a way to get that to work as of now.
Thanks,
Holden
RE: Playing MP4 to Display Troubles - Added by Jake Pregitzer about 9 hours ago
Hello Holden,
I have been looking into your issue with Jonathan Cormier, and we are able to get the video to playback at 30fps with around ~40% CPU utilization and no visible signs of frame loss or lag using the following pipeline:
$ gst-launch-1.0 filesrc location=bunny.mp4 ! qtdemux ! h264parse ! queue ! v4l2h264dec ! videocrop left=1 ! waylandsink
I tried the pipeline you provided initially, but when launching, the video never seems to render. I just see a black video feed on my display. In our testing, this black feed also occurs if no filter or property is placed after videocrop.
While examining performance by pipelining to fakesink and isolating various elements, we noticed a bottleneck specifically between the decoding stage and the sink. We began testing with elements like videocrop and videoscale, and found that when using an element property or filter with elements such as these, we are then able to see the video render completely on our display at the expected frame rate. Looking at the pipeline graphs from both the working and non-working pipeline and they are pretty identical, so something unknown to us currently must be happening when adding the element filter/property (left in this case above) to the element. We have verified this behavior on the AM62Ax EVM using TI's released SDK, and the results are consistent.
Since we do not have ximagesink supported on our current filesystem, would you mind trying the pipeline provided above and letting us know your results? We’ll be reporting this MP4 playback behavior to TI as well.
Thank you,
Jake
