Re: [Interest] Faster way of rendering QOpenGLFramebufferObject to image

[Nuno Santos]

Elvis,

Your sketch runs fine. 

Yesterday night I have been playing with calling glReadPixels directly rather 
than calling toImage from FBO object. Apparently, there wasn't much difference 
in the machine I was running this tests. I need to test on other machines.

But I also realised that the beforeRendering signal was by itself limited to 30 
fps. I was doing this tests on a Retina 5K 27” Intel iMac. I’ve been doing 
tests on other machines like a Windows laptop with a powerful NVIDIA gpu, an 
Intel i5 quad code machine and an Apple M1 iPad. I don’t have precise numbers 
right now, but I’ve had the frame rate above 60, and mostly on the Windows 
machine, I’ve seen frame rate of 150 fps.

But for some reason, in this machine, even with your simple example, 
afterRendering is not called more than 30 fps.

This raises the question: who is throttling the QQuickView? And why this value 
is not constant between machines? In the mean time I will run the same example 
on all other machines to get more precise numbers.

Best regards,

Nuno

> On 14 Jul 2021, at 23:48, Elvis Stansvik  wrote:
> 
> My idea would be something like (sketch):
> 
> main.cpp:
> 
> #include 
> #include 
> #include 
> #include 
> #include 
> 
> int main(int argc, char *argv[])
> {
>   QGuiApplication app(argc, argv);
> 
>   auto view = new QQuickView;
>   view->setSource(QUrl::fromLocalFile("test.qml"));
>   view->show();
> 
>   // I use a QVector here, but you would have your AVFrame and
>   // its allocated memory buffer..
>   QVector buffer;
>   QMutex mutex;
> 
>   QObject::connect(view, &QQuickView::afterRendering, view, [&]() {
>   mutex.lock();
>   buffer.resize(4 * view->width() * view->height());
>   glReadPixels(0, 0, view->width(), view->height(), GL_RGBA,
> GL_UNSIGNED_BYTE, buffer.data());
>   mutex.unlock();
>   }, Qt::DirectConnection);
> 
>   return app.exec();
> }
> 
> test.qml:
> 
> import QtQuick 2.2
> 
> Rectangle {
>   width: 1920
>   height: 1080
> 
>   Rectangle {
>   width: 500
>   height: 500
>   color: "green"
>   anchors.centerIn: parent
>   RotationAnimation on rotation {
>   from: 0
>   to: 360
>   duration: 2000
>   loops: Animation.Infinite
>   }
>   }
> }
> 
> I don't know whether this would work for you.
> 
> Elvis
> 
> Den tis 13 juli 2021 kl 15:31 skrev Nuno Santos :
>> 
>> Elvis,
>> 
>> Thanks for sharing your thoughts. It makes sense.
>> 
>> I need to dive into the toImage function, try to read directly the bytes 
>> from the FBO and see if that has any performance impact.
>> 
>> Best regards,
>> 
>> Nuno
>> 
>>> On 13 Jul 2021, at 14:10, Elvis Stansvik  wrote:
>>> 
>>> Hi Nuno,
>>> 
>>> I'm really out of my waters here, but, provided you don't need to hold
>>> on to each AVFrame after you are "done with it", you could perhaps
>>> avoid having to allocate a QImage for each frame (which toImage forces
>>> you to do) by just allocating a a single AVFrame and a single memory
>>> buffer for it, and then do what toImage does, which is make sure the
>>> FBO is bound and read the pixels off of it with glReadPixels. Then you
>>> could read the pixels straight into the memory buffer used by your
>>> AVFrame.
>>> 
>>> That way you would save the overhead of a new QImage being allocated
>>> each time, which might speed things up..?
>>> 
>>> Just ideas here. Have not worked with GL or ffmpeg before.
>>> 
>>> Elvis
>>> 
>>> Den tis 13 juli 2021 kl 11:22 skrev Nuno Santos :
 
 Hi,
 
 I’m trying to capture the content of an FBO to a video file. This video 
 file should contain the animations generated by a qml scene.
 
 To do this, I’m recurring to QOpenGLFramebufferObject class toImage() 
 method.
 
 My scene is being drawn at 1920x1080. Each call to toImage takes 30 ms! :(
 
 If I want to render to file at 60 fps, ideally, this call would need to 
 take less than 16 ms to give me room to do other operations, such as video 
 encoding and the actual render.
 
 As anyone been here before? What other strategies are available to copy 
 the FBO data to an image?
 
 I’m using libav to encode the video file, therefor I need to fill an 
 AVFrame. Right now I’m filling the AVFrame from the QImage generated by 
 the FBO toImage method.
 
 Does any one knows a method of filling an AVFrame directly from texture 
 data?
 
 Thanks
 
 Best regards,
 
 Nuno
 ___
 Interest mailing list
 Interest@qt-project.org
 https://lists.qt-project.org/listinfo/interest
>> 

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Faster way of rendering QOpenGLFramebufferObject to image

[Elvis Stansvik]

My idea would be something like (sketch):

main.cpp:

#include 
#include 
#include 
#include 
#include 

int main(int argc, char *argv[])
{
   QGuiApplication app(argc, argv);

   auto view = new QQuickView;
   view->setSource(QUrl::fromLocalFile("test.qml"));
   view->show();

   // I use a QVector here, but you would have your AVFrame and
   // its allocated memory buffer..
   QVector buffer;
   QMutex mutex;

   QObject::connect(view, &QQuickView::afterRendering, view, [&]() {
   mutex.lock();
   buffer.resize(4 * view->width() * view->height());
   glReadPixels(0, 0, view->width(), view->height(), GL_RGBA,
GL_UNSIGNED_BYTE, buffer.data());
   mutex.unlock();
   }, Qt::DirectConnection);

   return app.exec();
}

test.qml:

import QtQuick 2.2

Rectangle {
   width: 1920
   height: 1080

   Rectangle {
   width: 500
   height: 500
   color: "green"
   anchors.centerIn: parent
   RotationAnimation on rotation {
   from: 0
   to: 360
   duration: 2000
   loops: Animation.Infinite
   }
   }
}

I don't know whether this would work for you.

Elvis

Den tis 13 juli 2021 kl 15:31 skrev Nuno Santos :
>
> Elvis,
>
> Thanks for sharing your thoughts. It makes sense.
>
> I need to dive into the toImage function, try to read directly the bytes from 
> the FBO and see if that has any performance impact.
>
> Best regards,
>
> Nuno
>
> > On 13 Jul 2021, at 14:10, Elvis Stansvik  wrote:
> >
> > Hi Nuno,
> >
> > I'm really out of my waters here, but, provided you don't need to hold
> > on to each AVFrame after you are "done with it", you could perhaps
> > avoid having to allocate a QImage for each frame (which toImage forces
> > you to do) by just allocating a a single AVFrame and a single memory
> > buffer for it, and then do what toImage does, which is make sure the
> > FBO is bound and read the pixels off of it with glReadPixels. Then you
> > could read the pixels straight into the memory buffer used by your
> > AVFrame.
> >
> > That way you would save the overhead of a new QImage being allocated
> > each time, which might speed things up..?
> >
> > Just ideas here. Have not worked with GL or ffmpeg before.
> >
> > Elvis
> >
> > Den tis 13 juli 2021 kl 11:22 skrev Nuno Santos :
> >>
> >> Hi,
> >>
> >> I’m trying to capture the content of an FBO to a video file. This video 
> >> file should contain the animations generated by a qml scene.
> >>
> >> To do this, I’m recurring to QOpenGLFramebufferObject class toImage() 
> >> method.
> >>
> >> My scene is being drawn at 1920x1080. Each call to toImage takes 30 ms! :(
> >>
> >> If I want to render to file at 60 fps, ideally, this call would need to 
> >> take less than 16 ms to give me room to do other operations, such as video 
> >> encoding and the actual render.
> >>
> >> As anyone been here before? What other strategies are available to copy 
> >> the FBO data to an image?
> >>
> >> I’m using libav to encode the video file, therefor I need to fill an 
> >> AVFrame. Right now I’m filling the AVFrame from the QImage generated by 
> >> the FBO toImage method.
> >>
> >> Does any one knows a method of filling an AVFrame directly from texture 
> >> data?
> >>
> >> Thanks
> >>
> >> Best regards,
> >>
> >> Nuno
> >> ___
> >> Interest mailing list
> >> Interest@qt-project.org
> >> https://lists.qt-project.org/listinfo/interest
>
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Faster way of rendering QOpenGLFramebufferObject to image

[Nuno Santos]

Andy,

Thanks for your reply.

Are you able to capture at 60fps at 1920x1080?

Best,

Nuno

> On 14 Jul 2021, at 20:38, Andy  wrote:
> 
> Yet another "not sure if this applies, but..."
> 
> Maybe take a look at what QRenderCapture is doing in Qt3D? It allows you to 
> capture a QImage per frame, so it might have some of the bit 
> processing/manipulation you're looking for.
> 
> I'm using it to capture video by saving images to a temp dir and then 
> processing them using ffmpeg.
> 
> I use QtConcurrent::run() to do image processing (e.g. adding an overlay) & 
> writing so it doesn't affect the main thread doing the capturing.
> 
> ---
> Andy Maloney  //  https://asmaloney.com 
> twitter ~ @asmaloney 
> 
> 
> On Tue, Jul 13, 2021 at 9:32 AM Nuno Santos  > wrote:
> Elvis,
> 
> Thanks for sharing your thoughts. It makes sense.
> 
> I need to dive into the toImage function, try to read directly the bytes from 
> the FBO and see if that has any performance impact.
> 
> Best regards,
> 
> Nuno
> 
> > On 13 Jul 2021, at 14:10, Elvis Stansvik  > > wrote:
> > 
> > Hi Nuno,
> > 
> > I'm really out of my waters here, but, provided you don't need to hold
> > on to each AVFrame after you are "done with it", you could perhaps
> > avoid having to allocate a QImage for each frame (which toImage forces
> > you to do) by just allocating a a single AVFrame and a single memory
> > buffer for it, and then do what toImage does, which is make sure the
> > FBO is bound and read the pixels off of it with glReadPixels. Then you
> > could read the pixels straight into the memory buffer used by your
> > AVFrame.
> > 
> > That way you would save the overhead of a new QImage being allocated
> > each time, which might speed things up..?
> > 
> > Just ideas here. Have not worked with GL or ffmpeg before.
> > 
> > Elvis
> > 
> > Den tis 13 juli 2021 kl 11:22 skrev Nuno Santos  > >:
> >> 
> >> Hi,
> >> 
> >> I’m trying to capture the content of an FBO to a video file. This video 
> >> file should contain the animations generated by a qml scene.
> >> 
> >> To do this, I’m recurring to QOpenGLFramebufferObject class toImage() 
> >> method.
> >> 
> >> My scene is being drawn at 1920x1080. Each call to toImage takes 30 ms! :(
> >> 
> >> If I want to render to file at 60 fps, ideally, this call would need to 
> >> take less than 16 ms to give me room to do other operations, such as video 
> >> encoding and the actual render.
> >> 
> >> As anyone been here before? What other strategies are available to copy 
> >> the FBO data to an image?
> >> 
> >> I’m using libav to encode the video file, therefor I need to fill an 
> >> AVFrame. Right now I’m filling the AVFrame from the QImage generated by 
> >> the FBO toImage method.
> >> 
> >> Does any one knows a method of filling an AVFrame directly from texture 
> >> data?
> >> 
> >> Thanks
> >> 
> >> Best regards,
> >> 
> >> Nuno
> >> ___
> >> Interest mailing list
> >> Interest@qt-project.org 
> >> https://lists.qt-project.org/listinfo/interest 
> >> 
> 
> ___
> Interest mailing list
> Interest@qt-project.org 
> https://lists.qt-project.org/listinfo/interest 
> 

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Faster way of rendering QOpenGLFramebufferObject to image

[Nuno Santos]

João,

Thanks for the tip. Will search about it.

Best,

Nuno

> On 14 Jul 2021, at 20:22, joao morgado via Interest  
> wrote:
> 
> 
> Hello Nuno
> 
> Not sure if it helps, but the book Anton's OpenGL 4 Tutorials has a chapter 
> about recording a video from OpenGL. It's the same approach you are doing, 
> saving images from a frame buffer at certain fixed rate. I just dont know 
> what kinf of optimisations are done, I didnt read it with much attention.
> 
> Cheers
> João
> 
> 
> Em terça-feira, 13 de julho de 2021 14:33:42 GMT+1, Nuno Santos 
>  escreveu:
> 
> 
> Elvis,
> 
> Thanks for sharing your thoughts. It makes sense.
> 
> I need to dive into the toImage function, try to read directly the bytes from 
> the FBO and see if that has any performance impact.
> 
> Best regards,
> 
> Nuno
> 
> > On 13 Jul 2021, at 14:10, Elvis Stansvik  > > wrote:
> > 
> > Hi Nuno,
> > 
> > I'm really out of my waters here, but, provided you don't need to hold
> > on to each AVFrame after you are "done with it", you could perhaps
> > avoid having to allocate a QImage for each frame (which toImage forces
> > you to do) by just allocating a a single AVFrame and a single memory
> > buffer for it, and then do what toImage does, which is make sure the
> > FBO is bound and read the pixels off of it with glReadPixels. Then you
> > could read the pixels straight into the memory buffer used by your
> > AVFrame.
> > 
> > That way you would save the overhead of a new QImage being allocated
> > each time, which might speed things up..?
> > 
> > Just ideas here. Have not worked with GL or ffmpeg before.
> > 
> > Elvis
> > 
> > Den tis 13 juli 2021 kl 11:22 skrev Nuno Santos  > >:
> >> 
> >> Hi,
> >> 
> >> I’m trying to capture the content of an FBO to a video file. This video 
> >> file should contain the animations generated by a qml scene.
> >> 
> >> To do this, I’m recurring to QOpenGLFramebufferObject class toImage() 
> >> method.
> >> 
> >> My scene is being drawn at 1920x1080. Each call to toImage takes 30 ms! :(
> >> 
> >> If I want to render to file at 60 fps, ideally, this call would need to 
> >> take less than 16 ms to give me room to do other operations, such as video 
> >> encoding and the actual render.
> >> 
> >> As anyone been here before? What other strategies are available to copy 
> >> the FBO data to an image?
> >> 
> >> I’m using libav to encode the video file, therefor I need to fill an 
> >> AVFrame. Right now I’m filling the AVFrame from the QImage generated by 
> >> the FBO toImage method.
> >> 
> >> Does any one knows a method of filling an AVFrame directly from texture 
> >> data?
> >> 
> >> Thanks
> >> 
> >> Best regards,
> >> 
> >> Nuno
> >> ___
> >> Interest mailing list
> >> Interest@qt-project.org 
> >> https://lists.qt-project.org/listinfo/interest 
> >> 
> 
> ___
> Interest mailing list
> Interest@qt-project.org 
> https://lists.qt-project.org/listinfo/interest 
> 
> ___
> Interest mailing list
> Interest@qt-project.org
> https://lists.qt-project.org/listinfo/interest

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Faster way of rendering QOpenGLFramebufferObject to image

[Nuno Santos]

David,

The frames are not available forever. I’m using a double frame buffer object 
and one frame will only be available until the next one is ready. Then the 
frame buffer object is swapped. 

Adding concurrency to this problem might bring even more problems. It’s a 
guess, not sure.

Thanks!

Best,

Nuno

> On 13 Jul 2021, at 15:22, David M. Cotter  wrote:
> 
> I am also out of my league here because I’ve never done any of this stuff, 
> but is it possible to send these frames to a different thread to be saved? 
> That way your render thread and your export thread are on different cores?
> 
> Sent from my iPhone
> 
>> On Jul 13, 2021, at 6:33 AM, Nuno Santos  wrote:
>> 
>> Elvis,
>> 
>> Thanks for sharing your thoughts. It makes sense.
>> 
>> I need to dive into the toImage function, try to read directly the bytes 
>> from the FBO and see if that has any performance impact.
>> 
>> Best regards,
>> 
>> Nuno
>> 
>>> On 13 Jul 2021, at 14:10, Elvis Stansvik  wrote:
>>> 
>>> Hi Nuno,
>>> 
>>> I'm really out of my waters here, but, provided you don't need to hold
>>> on to each AVFrame after you are "done with it", you could perhaps
>>> avoid having to allocate a QImage for each frame (which toImage forces
>>> you to do) by just allocating a a single AVFrame and a single memory
>>> buffer for it, and then do what toImage does, which is make sure the
>>> FBO is bound and read the pixels off of it with glReadPixels. Then you
>>> could read the pixels straight into the memory buffer used by your
>>> AVFrame.
>>> 
>>> That way you would save the overhead of a new QImage being allocated
>>> each time, which might speed things up..?
>>> 
>>> Just ideas here. Have not worked with GL or ffmpeg before.
>>> 
>>> Elvis
>>> 
>>> Den tis 13 juli 2021 kl 11:22 skrev Nuno Santos :
 
 Hi,
 
 I’m trying to capture the content of an FBO to a video file. This video 
 file should contain the animations generated by a qml scene.
 
 To do this, I’m recurring to QOpenGLFramebufferObject class toImage() 
 method.
 
 My scene is being drawn at 1920x1080. Each call to toImage takes 30 ms! :(
 
 If I want to render to file at 60 fps, ideally, this call would need to 
 take less than 16 ms to give me room to do other operations, such as video 
 encoding and the actual render.
 
 As anyone been here before? What other strategies are available to copy 
 the FBO data to an image?
 
 I’m using libav to encode the video file, therefor I need to fill an 
 AVFrame. Right now I’m filling the AVFrame from the QImage generated by 
 the FBO toImage method.
 
 Does any one knows a method of filling an AVFrame directly from texture 
 data?
 
 Thanks
 
 Best regards,
 
 Nuno
 ___
 Interest mailing list
 Interest@qt-project.org
 https://lists.qt-project.org/listinfo/interest
>> 
>> ___
>> Interest mailing list
>> Interest@qt-project.org
>> https://lists.qt-project.org/listinfo/interest

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Faster way of rendering QOpenGLFramebufferObject to image

[Andy]

Yet another "not sure if this applies, but..."

Maybe take a look at what QRenderCapture is doing in Qt3D? It allows you to
capture a QImage per frame, so it might have some of the bit
processing/manipulation you're looking for.

I'm using it to capture video by saving images to a temp dir and then
processing them using ffmpeg.

I use QtConcurrent::run() to do image processing (e.g. adding an overlay) &
writing so it doesn't affect the main thread doing the capturing.

---
Andy Maloney  //  https://asmaloney.com
twitter ~ @asmaloney 



On Tue, Jul 13, 2021 at 9:32 AM Nuno Santos 
wrote:

> Elvis,
>
> Thanks for sharing your thoughts. It makes sense.
>
> I need to dive into the toImage function, try to read directly the bytes
> from the FBO and see if that has any performance impact.
>
> Best regards,
>
> Nuno
>
> > On 13 Jul 2021, at 14:10, Elvis Stansvik  wrote:
> >
> > Hi Nuno,
> >
> > I'm really out of my waters here, but, provided you don't need to hold
> > on to each AVFrame after you are "done with it", you could perhaps
> > avoid having to allocate a QImage for each frame (which toImage forces
> > you to do) by just allocating a a single AVFrame and a single memory
> > buffer for it, and then do what toImage does, which is make sure the
> > FBO is bound and read the pixels off of it with glReadPixels. Then you
> > could read the pixels straight into the memory buffer used by your
> > AVFrame.
> >
> > That way you would save the overhead of a new QImage being allocated
> > each time, which might speed things up..?
> >
> > Just ideas here. Have not worked with GL or ffmpeg before.
> >
> > Elvis
> >
> > Den tis 13 juli 2021 kl 11:22 skrev Nuno Santos <
> nuno.san...@imaginando.pt>:
> >>
> >> Hi,
> >>
> >> I’m trying to capture the content of an FBO to a video file. This video
> file should contain the animations generated by a qml scene.
> >>
> >> To do this, I’m recurring to QOpenGLFramebufferObject class toImage()
> method.
> >>
> >> My scene is being drawn at 1920x1080. Each call to toImage takes 30 ms!
> :(
> >>
> >> If I want to render to file at 60 fps, ideally, this call would need to
> take less than 16 ms to give me room to do other operations, such as video
> encoding and the actual render.
> >>
> >> As anyone been here before? What other strategies are available to copy
> the FBO data to an image?
> >>
> >> I’m using libav to encode the video file, therefor I need to fill an
> AVFrame. Right now I’m filling the AVFrame from the QImage generated by the
> FBO toImage method.
> >>
> >> Does any one knows a method of filling an AVFrame directly from texture
> data?
> >>
> >> Thanks
> >>
> >> Best regards,
> >>
> >> Nuno
> >> ___
> >> Interest mailing list
> >> Interest@qt-project.org
> >> https://lists.qt-project.org/listinfo/interest
>
> ___
> Interest mailing list
> Interest@qt-project.org
> https://lists.qt-project.org/listinfo/interest
>
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Faster way of rendering QOpenGLFramebufferObject to image

[joao morgado via Interest]

 
Hello Nuno
Not sure if it helps, but the book Anton's OpenGL 4 Tutorials has a chapter 
about recording a video from OpenGL. It's the same approach you are doing, 
saving images from a frame buffer at certain fixed rate. I just dont know what 
kinf of optimisations are done, I didnt read it with much attention.
CheersJoão

Em terça-feira, 13 de julho de 2021 14:33:42 GMT+1, Nuno Santos 
 escreveu:  
 
 Elvis,

Thanks for sharing your thoughts. It makes sense.

I need to dive into the toImage function, try to read directly the bytes from 
the FBO and see if that has any performance impact.

Best regards,

Nuno

> On 13 Jul 2021, at 14:10, Elvis Stansvik  wrote:
> 
> Hi Nuno,
> 
> I'm really out of my waters here, but, provided you don't need to hold
> on to each AVFrame after you are "done with it", you could perhaps
> avoid having to allocate a QImage for each frame (which toImage forces
> you to do) by just allocating a a single AVFrame and a single memory
> buffer for it, and then do what toImage does, which is make sure the
> FBO is bound and read the pixels off of it with glReadPixels. Then you
> could read the pixels straight into the memory buffer used by your
> AVFrame.
> 
> That way you would save the overhead of a new QImage being allocated
> each time, which might speed things up..?
> 
> Just ideas here. Have not worked with GL or ffmpeg before.
> 
> Elvis
> 
> Den tis 13 juli 2021 kl 11:22 skrev Nuno Santos :
>> 
>> Hi,
>> 
>> I’m trying to capture the content of an FBO to a video file. This video file 
>> should contain the animations generated by a qml scene.
>> 
>> To do this, I’m recurring to QOpenGLFramebufferObject class toImage() method.
>> 
>> My scene is being drawn at 1920x1080. Each call to toImage takes 30 ms! :(
>> 
>> If I want to render to file at 60 fps, ideally, this call would need to take 
>> less than 16 ms to give me room to do other operations, such as video 
>> encoding and the actual render.
>> 
>> As anyone been here before? What other strategies are available to copy the 
>> FBO data to an image?
>> 
>> I’m using libav to encode the video file, therefor I need to fill an 
>> AVFrame. Right now I’m filling the AVFrame from the QImage generated by the 
>> FBO toImage method.
>> 
>> Does any one knows a method of filling an AVFrame directly from texture data?
>> 
>> Thanks
>> 
>> Best regards,
>> 
>> Nuno
>> ___
>> Interest mailing list
>> Interest@qt-project.org
>> https://lists.qt-project.org/listinfo/interest

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest
  ___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest