I'm not too well versed on Python internals, just enough to have made the
boost.python bindings work for OIIO. I do understand that the GIL bites a lot
of people, and we basically have ignored it up to now. A quick google search
turns up this: https://wiki.python.org/moin/boost.python/HowTo (search for
"GIL" on that page). This makes me think that maybe it's not hard to use that
ScopedGILRelease in our most expensive Python wrappers, to at least release the
GIL while they do their work. I can probably try that, or feel free to give a
go yourself, it looks fairly straightforward and you presumably already have a
good test case to know if it's helping.
Also, it's possible that you are using particular ImageBufAlgo functions that
do not multithread themselves. For completeness, I added the 'nthreads'
parameter to all IBA functions, but in fact there are probably a number of them
that ignore it and just operate single-threaded. Or, who knows, just have some
other big inefficiency that could be easily addressed. If you send me a sample
command line for how you are using oiiotool (or short Python snippet, if you
are primarily using those APIs), I'm happy to see which IBA functions are
sucking up the time and take a look to see if there are any obvious speedups. I
bet we use a lot of those same functions here, so even if there haven't been
complaints thus far, if I found a way to make those functions a lot faster
without too much hassle, people would be happy.
-- lg
On Aug 31, 2014, at 5:35 PM, Dave Lajoie <[email protected]>
wrote:
> Hello Guys!
>
> I have been searching the archive and I could not find anything on this.
> Basically I am using open image IO as slate generation tool (among other
> things of course). It works great! I am looking at speeding up the process,
> so I have attempted to call my python function using various threading
> modules, and I am not getting a lot of speed up. I am using nthread=0 to
> ensure the call uses all available threads/cores.
>
> I do know that Python GIL is well known to be great at certain type of
> multi-threading calls, especially anything IO bound offers best performance.
> However I was wondering if there are any know issues with python
> multithreading, and/or if there was any list of do's and don'ts.
>
> Currently the only way I could get all the cores to be utilized to 100% was
> to spawn a separate python process, which will process a single frame, so on
> my box I would have 24 concurrent python subprocesses invoking OIIO. The
> machine grinds almost to halt ( which is good in this case), but I was
> wondering what are the other things I don't know regarding this subject. :)
>
> Any help would be appreciated.
> Best
> Dave Lajoie
> R&D Director Digital District Montreal
>
>
> _______________________________________________
> Oiio-dev mailing list
> [email protected]
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
[email protected]
_______________________________________________
Oiio-dev mailing list
[email protected]
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
