Hmmm...
I think you've manipulated yourself into an interesting performance edge case.
OIIO holds its image as ImageBuf's internally, and read-only ImageBuf for very
large images get automatically backed by ImageCache. The logic for when to do
this and how (or if) to emulate tiling in the cache is making assumptions that
images are really big in memory by being *spatially* big, but with 50 channels
you are fat in a different way, and I think it's conspiring to have the
different threads block against each other as well as do redundant I/O.
I would recommend trying one or more of the following oiio command line flags
(see what each one does alone and in combination, as far as both time and
memory footprint and whether or not it scales well with threads):
--cache <something bigger>
This will increase the ImageCache size. The default is 4096 (MB). Maybe
bigger if you've got enough memory?
--autotile <SIZE>
Enables autotiling with the given size. Try 4096? 2048? 512?
--native
Forces a full read in native format (no conversion to float, no back by
imagecache)
> On Jul 30, 2019, at 10:34 AM, Carl Bérubé <[email protected]> wrote:
>
> Thank you! That's what I imagined.
>
> However, I'm finding something to be a bit unusual in terms of performance --
> passing 'nthreads=1' to the ImageBufAlgo copy function made the code run
> twice as fast as when I give it more than one (or when I don't pass the
> argument at all). However, I can't explain it... Would you have any idea
> what's going on? I'm reading the source but I'm not that familiar with CPP in
> general.
>
> Keep in mind that this is just strictly out of curiosity!
>
> Thanks again for your help
>
> On Tue, Jul 30, 2019 at 3:01 AM Larry Gritz <[email protected]
> <mailto:[email protected]>> wrote:
> um... it's not *quite* easy to do with a single oiiotool command line. But
> you sure could do it easily in a small python script, something like this:
>
> # nxtiles,nytiles is the number of x and y tiles
> # tilewidth, tilelength is the width and length of each tile
> base = ImageBuf(ImageSpec(nxtiles*tilewidth, nytiles*tileheight, nchans,
> TypeFloat)
> for y in range(nytiles) :
> for x in range(nxtiles) :
> img = ImageBuf("image.{:04}.{:04}.exr".format(x,y))
> roi = ROI(x*tilewidth, (x+1)*tilewidth, y*tileheight, (y+1)*tileheight)
> ImageBufAlgo.copy(base, img, roi=roi)
> base.write("out.exr", "half")
>
>
> Yes, I guess that's pretty much the same as what you proposed originally. So
> I think you were on the right track all along.
>
>
>
>> On Jul 29, 2019, at 5:47 PM, Carl Bérubé <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Yeah exactly, each file has the full data window, but only some part we care
>> about and there may be non-black pixel values outside of that!
>>
>> They're named like image.t###.####.exr
>>
>> Tiles are the same sizes but won't always be square. The users inputs the
>> amount of tiles they want in x and y. 10x10 for examples.
>>
>> On Mon, Jul 29, 2019 at 8:42 PM Larry Gritz <[email protected]
>> <mailto:[email protected]>> wrote:
>> So the data windows are not restricted to their tiled areas? Each file has a
>> full data window, but only part you care about, and there may be non-black
>> pixel values outside of that?
>>
>> Are the files/tiles at least all the same size (if offset)? And are they
>> named something sensible?
>>
>>
>>> On Jul 29, 2019, at 5:39 PM, Carl Bérubé <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> That's correct! Sorry if this wasn't clear!
>>>
>>> Each tile has the full data window, so in Nuke we currently just merge them
>>> on top of each other! Unfortunately we can't do that anymore as some
>>> garbage pixels just appeared outside the tile's rectangle and they get
>>> added to the final image!
>>>
>>> Does that make sense?
>>>
>>> On Mon, Jul 29, 2019 at 8:35 PM Larry Gritz <[email protected]
>>> <mailto:[email protected]>> wrote:
>>> I'm not quite sure what you're trying to do. Is this the gist...?
>>>
>>> You broke up a large frame into several overlapping pieces and you're
>>> trying to recombine again, is that it? Do they all have their data windows
>>> set correctly to their section, so that among them all they are all exactly
>>> abutting and non-overlapping? And do they all have the same display window
>>> that indicates the size of the full frame you are trying to assemble?
>>>
>>>
>>>
>>>
>>>> On Jul 29, 2019, at 5:13 PM, Carl Bérubé <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>
>>>
>>>> Hey!
>>>>
>>>> Sorry about the confusing title!
>>>>
>>>> I'm dealing with a small issue and I'm curious to know if either oiiotool
>>>> or the Python binding of OpenImageIO can help me do what I want...
>>>>
>>>> I've got EXR images with lots of channels (~50 -- cryptomatte and all)
>>>> rendered in regions, but the buffer is the full resolution.
>>>>
>>>> So you can split it in tiles, and they can be uneven -- 5 rows and 7
>>>> columns for example.
>>>>
>>>> I didn't think it was possible to do it through oiiotool, but I've managed
>>>> to do with I wanted with the Python Binding so far, but it hasn't been as
>>>> efficient as I thought it would be...
>>>>
>>>> My pseudo code would go as follow :
>>>>
>>>> final_image_buffer = OpenImageIO.ImageBuffer(full_frame_spec)
>>>>
>>>> for tile_file_path, rect in tile_file_paths:
>>>> tile_image_buffer = OpenImageIO.ImageBuf(tile_file_path)
>>>> roi_cropped = OpenImageIO.ROI(*rect) # it's like (xbegin, xend, ybeing,
>>>> yend)
>>>> OpenImageIO.ImageBufAlgo.copy(final_image_buffer, tile_image_buffer,
>>>> roi=roi_cropped)
>>>>
>>>> final_image_buffer.write(output_path)
>>>> Now that works pretty well, like I said, but it is very slow. Eventually
>>>> that might not be that big of a deal when it goes on better farm machines
>>>> than my development toaster oven, but is there anything obvious that I'm
>>>> missing? Would oiiotool be able to deal with this more efficiently?
>>>>
>>>> Thanks!
>>>
>>>> _______________________________________________
>>>> Oiio-dev mailing list
>>>> [email protected] <mailto:[email protected]>
>>>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>>>> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>>>
>>> --
>>> Larry Gritz
>>> [email protected] <mailto:[email protected]>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Oiio-dev mailing list
>>> [email protected] <mailto:[email protected]>
>>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>>> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>>> _______________________________________________
>>> Oiio-dev mailing list
>>> [email protected] <mailto:[email protected]>
>>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>>> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>>
>> --
>> Larry Gritz
>> [email protected] <mailto:[email protected]>
>>
>>
>>
>>
>> _______________________________________________
>> Oiio-dev mailing list
>> [email protected] <mailto:[email protected]>
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>> _______________________________________________
>> Oiio-dev mailing list
>> [email protected] <mailto:[email protected]>
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>
> --
> Larry Gritz
> [email protected] <mailto:[email protected]>
>
>
>
>
> _______________________________________________
> Oiio-dev mailing list
> [email protected] <mailto:[email protected]>
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
> _______________________________________________
> Oiio-dev mailing list
> [email protected]
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
[email protected]
_______________________________________________
Oiio-dev mailing list
[email protected]
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
