That's some interesting information you detail. Any chance I could have your recipe for the timing work?
Many thanks -Piotr On 22 December 2015 at 17:30, Larry Gritz <[email protected]> wrote: > OK, let's get this straight once and for all: there is no such thing as a > ".tx" file. I'm pretty sure that NONE of the couple dozen image file > formats that OIIO supports actually consider the file name extension > semantically meaningful. The files are always identified by their internal > layout, and at most the file extension is just used to help guess which > ImageInput type to try first, but if that fails it will simply try every > one it knows about until one successfully reads the file (or none of them > do). > > By convention -- but in no way required or aided by OIIO -- it is common > to name tiled & MIP-mapped TIFF files ".tx", I suppose just to distinguish > them from the ordinary TIFF files floating around (but which, usually > scanline and almost never MIPmapped, are unsuitable as efficient texture > files). I guess this dates from PRMan's "txmake" program, though IIRC it > was at least as common in the olden days to name its proprietary files > ".tex", but since I use LaTeX a lot myself, I never liked that so tended to > use ".tx." > > Anyway. > > OpenEXR versus TIFF for textures. > > It's funny you should ask, because I recently did some extensive > benchmarks to shed light on this very issue. > > The first thing you need to know is that OpenEXR only supports 'half' (16 > bit float), float (32 bit), and unsigned 32 bit int (the latter only used > for object IDs as far as I know, never for color values). Really, half is > what you want, it seems stupidly wasteful to use 32 bit float for most > possible uses of texture. > > But TIFF *doesn't* support half. So it's hard to do an "apples-to-apples" > comparison. > > So the decision has traditionally come down to: if you want uint8 or > uint16 textures, use TIFF; if you need extended range (such as for IBL > environment maps), use half in OpenEXR. > > OK, but it turns out I sort of lied. There is this > http://chriscox.org/TIFFTN3d1.pdf which is an Adobe tech report > describing 16 bit float extension to TIFF. OIIO has for some time been > rigged to *read* this properly, but since 'half TIFF' does not read > properly by either PhotoShop or Nuke (and worse, Nuke doesn't even notice > it's broken, it just badly misinterprets it as uint16), it seemed too > dangerous to enable it for writing, for fear of infesting the world with > files that will not be read correctly. (Aside: PhotoShop and Nuke, if you > used OIIO, it would read just fine. So there.) > > But I had to know. So I finally enabled this in OIIO just a couple weeks > ago (https://github.com/OpenImageIO/oiio/pull/1283). For safety, it ONLY > will write 'half' TIFF files if you set the global > OIIO::attribute("tiff:half", 1). You have to really want it, and you have > to be ok knowing that only OIIO-based apps will read it correctly. > > But once you have it, you can finally compare TIFF vs OpenEXR directly, > with the same data format. > > I took a whole bunch (many GBs) of uint16 textures randomly selected from > a certain large VFX show we're working on at the moment, and converted them > to half TIFF, and also to half OpenEXR, and then ran a bunch of synthetic > benchmarks on them. (Certain modes of OIIO's 'testtex' unit test are really > helpful for timing tests.) > > I'm on vacation for the next couple weeks, so if you really care I can dig > up the numbers when I get back, but the bottom line is that file access is > significantly faster for TIFF! Like, it's 1.5x faster or more to read the > same tile pattern from a half TIFF file as from the equivalent half OpenEXR > file. > > I'm still in the middle of trying to understand why. But it seems pretty > incontrovertible. I can give a detailed recipe if you want for how to > reproduce these findings (with your own textures, of course). > > Ironically, the whole purpose of my going down that road was that I was > interested in trying the DWA lossy compression for texture files -- for the > sake of reducing disk storage and network bandwidth for texture reads. It's > great for that, these textures look visually identical while achieving a > 2-3x reduction of space and bandwidth, but to make a complete evaluation I > did time trials as well, and then it got really confusing because the > format with the better compression has such massively worse read > performance. > > The perf difference is not because of compression/decompression. I > replicated the experiment for zip compression, and for no compression. The > ratio by which TIFF seems faster than OpenEXR seems fairly stable no matter > what else I vary. When I use the DWA lossy compression for EXR, it speeds > up a bit (fewer bytes read/transmitted!) but still never catches up to > TIFF, not even close. > > Honestly, I'm shocked. I truly expected TIFF (which has no shortage of > design issues) to be slower, and thus clear the path to using lossily > compressed textures (for some purposes, and subject to aesthetic approval, > of course). But no, it's determined not to be a clean win, but a very > complex tradeoff. > > So, since I have you on the line, Bruce, is there anything you can tell us > about DWA compression as used by DWA? How do you use it? For what > purposes/uses? What compression levels? Have you ever benchmarked it versus > other formats? > > Has anybody at any time tried to validate OpenEXR read/write performance > versus any other file format? If so, did they also find that it's a little > lacking? > > -- lg > > > On Dec 22, 2015, at 4:48 PM, Thiago Ize <[email protected]> wrote: > > .tx files are just .tiff or .exr files that have been renamed .tx. I > think as long as you generated the .tif/.exr file with maketx, it'll be the > exact same thing. You can confirm by looking at your files with the "iinfo > -v " oiio tool. Calling it .tx just helps in letting people know that the > texture has already been optimized without forcing them to parse the file > and look at the file attributes. > > Just to be clear, a .exr/.tif that was not made with maketx will not > perform as well since it doesn't have the AverageColor or SAH-1 oiio > attributes. > > Finally, don't take a .tx file, rename it to .exr, edit it in photoshop, > and rename to .tx. Do that and now it's no longer an optimized file. In > fact, you might even get rendering artifacts since the AverageColor and > SHA-1 attributes could be wrong. > > On Tue, Dec 22, 2015 at 5:36 PM, Bruce Tartaglia < > [email protected]> wrote: > >> Hi all, >> >> I am using OIIO to manage mipmapped texture lookups, >> and I have a question about the exr vs the tx format. >> >> In brief, is there a preferred / optimized format for the OIIO library? >> Both seem to work quite well, both support mip mapping, half >> data formats, maketx --oiio flag, and tiling. So is one superior to >> the other? >> >> I suspect that tx is likely closer to an internal, native OIIO format, >> so perhaps exr incurs a data conversion cost, but I have not proof of >> that. >> I am starting to read the OIIO code to best understand this, but I >> thought >> I could simply ask here as well. >> >> Any information is appreciated, and thank you for such >> useful software. >> >> Best, >> >> Bruce >> >> _______________________________________________ >> Oiio-dev mailing list >> [email protected] >> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org >> >> > _______________________________________________ > Oiio-dev mailing list > [email protected] > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org > > > -- > Larry Gritz > [email protected] > > > > _______________________________________________ > Oiio-dev mailing list > [email protected] > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org > >
_______________________________________________ Oiio-dev mailing list [email protected] http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
