Hi Marius,
Many thanks for the info.
Some notes/questions below:
Terry
On 22/02/2024 17:49, Marius Vlad wrote:
Hi,
On Thu, Feb 22, 2024 at 03:21:01PM +0000, Terry Barnaby wrote:
Hi,
We are developing a video processing system that runs on an NXP imx8
processor using a Yocto embedded Linux system that has Qt6, GStreamer,
Wayland and Weston.
We are having a problem displaying the video stream from GStreamer on a
QWidget. In the past we had this working with Qt5 and older GStreamer,
Wayland and Weston.
A simple test program also shows the issue on Fedora37 with QT6 and
KDE/Plasma/Wayland.
I'm tempted to say if this happens on a desktop with the same Qt version and
other compositors to be an issue with Qt rather than waylandsink or
the compositor. Note that on NXP they have their own modified Weston version.
That is my current feeling and is one reason why I tried it on Fedora
with whatever Wayland compositor KDE/Plasma is using.
The technique we are using is to get the Wayland surface from the QWidget is
using (It has been configured to use a Qt::WA_NativeWindow) and pass this to
the GStreamer's waylandsink which should then update this surface with video
frames (via hardware). This works when the QWidget is a top level Window
widget (QWidget(0)), but if this QWidget is below others in the hierarchy no
video is seen and the gstreamer pipeline line is stalled.
So the assumption is that aren't there other widgets which obscures this
one, when you move it below others?
My simple test example has two QWidgets with the one for video being
created as a child of the first so it should be above all others. I have
even tried drawing in it to make sure and it displays its Qt drawn
contents fine, just not the video stream.
It appears that waylandsink does:
Creates a surface callback:
callback = wl_surface_frame (surface);
wl_callback_add_listener (callback, &frame_callback_listener, self);
Then adds a buffer to a surface:
gst_wl_buffer_attach (buffer, priv->video_surface_wrapper);
wl_surface_set_buffer_scale (priv->video_surface_wrapper, priv->scale);
wl_surface_damage_buffer (priv->video_surface_wrapper, 0, 0, G_MAXINT32,
G_MAXINT32);
wl_surface_commit (priv->video_surface_wrapper);
But never gets a callback and just sits in a loop awaiting that callback.
I assume that the surface waylandsink is using, which is created using the
original QWidget surface (sub-surface ? with window ?) is not "active" for
some reason.
Possibly when QWidget is below in hierarcy to be a child of of a parent,
as described in
https://wayland.app/protocols/xdg-shell#xdg_toplevel:request:set_parent,
so I assume to have a different surface than the parent one. This would
be easy to determine with WAYLAND_DEBUG. Seems unlikely to a itself a
sub-surface of a surface.
I haven't really got the gist of whats going on, but waylandsink
certainly creates a subsurface from the QWidget surface, in fact it
seems to create a few things.
I assume a subsurface is used so the video can be displayed in that
subsurface separately from the parent (de synced from it).
I am trying to debug this, but this graphics stack is quite complicated with
waylandsink, qtwayland, wayland-lib and Weston not to mention the NXP
hardware levels. My thoughts are that it is something qtwayland is doing
with the surface stack or thread locking issues (gstreamer uses separate
threads). I also don't understand Wayland or Weston in detail. So some
questions:
1. Anyone seen something like this ?
Someone else reported something similar but that by causing damage,
or moving pointer to make the video sub-surface to show up:
https://gitlab.freedesktop.org/wayland/weston/-/issues/843.
Thanks, I will have a look. Moving the mouse cursor in my case (at least
with Weston) does not affect things.
2. Anyone any idea one where to look ?
3. Given the wl_surface in the Qt app or in waylandsink is there a way I can
print out its state and the surface hierarchy easily ?
In Weston there's something called scene-graph. You can grab it by
starting Weston with with the --debug argument, then you can print
with `weston-debug scene-graph` command. A more recent Weston version
would indent sub-surfaces by their (main) surface parent.
Thanks, that could be useful.
4. Any idea on any debug methods to use ?
WAYLAND_DEBUG=1 as env variable.
Any idea on how to get a surfaces ID from a C pointer so I can match up
the QtWidget/waylandsink surface with the Wayland debug output ?
Cheers
Terry
