And more:
Can you see if the slowdown is apparent in the following:
oiiotool -threads 1 -runstats bigfile.dpx -o out.dpx
Do you see a big slowdown there for 1.8.5 vs 2.0 -- for a simple full
read/write of a single large DPX file with one thread? What if you leave off
the "-threads 1" argument?
I would tend to suspect that it's related to operating within the QtThreadPool,
somehow interacting badly with OIIO's own thread pool. Except that in the
video, PNG, EXR, and JPG actually speed up a bit between 1.8.5 and 2.0, so that
feels more like a DPX specific issue.
I'm able to see a *small* slowdown in the oiiotool-based example, but not very
dramatic (well within a factor of 2, but it's only a few seconds to fully read
and write a 2k image) -- maybe that's the slowdown, or maybe not. You reported
>10x degradation.
I know it's a lot to impose on you, but is it possible for you to build a few
more versions of OIIO to see exactly where it broke? For starters, it would be
helpful to test the last 1.8 release (1.8.17, I think) to narrow down if it
truly happened for 2.0 and later, and not somewhere in the middle of 1.8.x. I'd
also be curious to see if a relatively modern master exhibits the same problem.
> On Sep 3, 2019, at 11:23 PM, Larry Gritz <[email protected]> wrote:
>
> A few more questions:
>
> How are you reading the files? I know that you saw a fix with implementing a
> DPXInput::read_image(), but are you indeed reading directly through an API
> call to ImageInput::read_image()? Or using an ImageBuf?
>
> From C++ or from Python?
>
> Do you see the slowdown if you are reading just one DPX file at a time, or do
> you only see it if you are reading a bunch of files simultaneously, as
> appeared in the demo video?
>
> -- lg
>
>
>> On Sep 3, 2019, at 11:09 PM, Larry Gritz <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> I still don't quite understand what would have caused the performance
>> degradation, so I'm going to compare the code to try to understand what
>> changed.
>>
>> Is there anything characteristic about the DPX files? Just ordinary RGB?
>> Anything unusual about them, especially high resolution, or other
>> attributes? Or did you see orders of magnitude slowdown with every file you
>> tried? Are you Windows only or did you get a chance to see if a slowdown was
>> apparent on Linux or OSX as well? Trying to understand what happened, as DPX
>> is pretty widely used in studios but nobody had previously noticed this.
>>
>> -- lg
>>
>>
>>> On Sep 3, 2019, at 10:37 PM, Renaud Talon <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> This use to work fine for me on 1.8.5 and previous. I honestly didn’t
>>> update OIIO before the first(ish) release of OIIO 2.0 and that’s when I
>>> started running into the performance issues with DPXs.
>>>
>>> At the time I was compiling with a certain set of thrid party libraries and
>>> Visual Studio 2015. Since then I re-compiled all my dependencies for Visual
>>> Studio 2017 and made a new attempt at compiling oiio 2.0 (in this case
>>> 2.0.10) but I ran into the same performance issue.
>>>
>>> My initial test code was using the “read_scanline()” method instead of
>>> “read_image()” and Michael’s patch only fixes the read_image() method of
>>> course so the problem remains with “read_scanline()”.
>>>
>>> I did compare the dpx plugin code between 1.8.5 and 2.0.10 and noticed that
>>> there were quite a few changes that were made but my C++ skills are not
>>> advanced enough for me to figure out what could cause this behavior.
>>>
>>> I created a PR but this is more to help keep track of the issue than a
>>> request to merge this code in as I don’t think this is a proper “long-term”
>>> fix.
>>>
>>> Thanks again,
>>>
>>> Renaud
>>>
>>> From: Larry Gritz <mailto:[email protected]>
>>> Sent: Tuesday, September 3, 2019 5:37 PM
>>> To: OpenImageIO developers <mailto:[email protected]>
>>> Subject: Re: [Oiio-dev] OIIO 2.x extremely slow DPX load time
>>>
>>> Definitely. Can you prepare a real PR with the patch that seems to work?
>>>
>>> I'll do some digging to try to figure out what broke. Did it used to be
>>> like this and we broke something?
>>>
>>>
>>>
>>>
>>> On Sep 3, 2019, at 5:32 PM, Renaud Talon <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Hi Michael,
>>>
>>> your patch absolutely works ! My benchmark goes from 60s+ to 700ms !!!
>>>
>>> I was getting corrupted images with my test code at first because I wasn't
>>> passing the image spec.format to the "read_image()" method. (which worked
>>> fine with OIIO 1.8.5 but is clearly needed to use the DPXInput::read_image
>>> patch you provided)
>>>
>>> I changed :
>>> pixels = srcFile.read_image()
>>>
>>> to:
>>> pixels = srcFile.read_image(spec.format)
>>>
>>> and it worked like a charm.
>>>
>>> Larry, if you're reading this. Could we get Michael's patch or a similar
>>> fix merged into OIIO code please ?
>>> Thanks,
>>>
>>> Renaud
>>> _______________________________________________
>>> Oiio-dev mailing list
>>> [email protected] <mailto:[email protected]>
>>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>>> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>>>
>>> --
>>> Larry Gritz
>>> [email protected] <mailto:[email protected]>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Oiio-dev mailing list
>>> [email protected] <mailto:[email protected]>
>>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>>> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>> --
>> Larry Gritz
>> [email protected] <mailto:[email protected]>
>>
>>
>>
>>
>> _______________________________________________
>> Oiio-dev mailing list
>> [email protected] <mailto:[email protected]>
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>
> --
> Larry Gritz
> [email protected] <mailto:[email protected]>
>
>
>
>
> _______________________________________________
> Oiio-dev mailing list
> [email protected]
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
[email protected]
_______________________________________________
Oiio-dev mailing list
[email protected]
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
