Have you gotten a chance to try this on a different file type? It's possible (I haven't audited the code yet) that the DDS reader has some kind of persistent state that is growing for each file it opens?
This is all just a guess. I'll try to recreate on my end with a script, and I'll be curious to hear if you find anything looking at the cache statistics. > On Apr 11, 2016, at 7:57 PM, Jonathan Tilden (2K) <[email protected]> > wrote: > > Yeah I thought so too - I wrote the test in a huge hurry on the way out the > door. It's possible I did something very embarrassingly dumb in the test :) > > I intend to revisit in a bit here just to make sure... > > > > -----Original Message----- > From: Larry Gritz [[email protected] <mailto:[email protected]>] > Sent: Monday, April 11, 2016 10:48 PM Eastern Standard Time > To: OpenImageIO developers > Subject: Re: [Oiio-dev] decrease performance when opening 1000s of files with > python > > So the cost of each additional one is proportional to the number or previous > images opened? That's weird. > > > > On Apr 11, 2016, at 6:51 PM, Jonathan Tilden (2K) <[email protected]> > > wrote: > > > >> My suspicion is that this is related to a growing number of files held > >> open by the underlying ImageCache, which will not be immediately closed or > >> free just because you clear() or even destroy the ImageBuf. Or possibly by > >> the overhead of what happens when the maximum number of open files is > >> reached? > > > > I was coming to a similar conclusion when I was trying to modify files in > > my test bed, and the OS was complaining that I still had open filehandles. > > > >> The default is 100, I'm curious what happens if you just raise that. On > >> Linux/OSX it should be safe to go into thousands, but Windows usually has > >> some lower limit, to let's try 500 just to see what happens to your > >> timings. In particular, does the slowdown seem to come later, i.e. after > >> more files have been touched? > > > > I ran some more tests locally and, unfortunately, I actually didn't see any > > improvement using the cache. Nor do I see much improvement using the > > invalidate method you described. > > > > I've attached a script and some graphs if you wanted to take a look. The > > only real requirement is that you populate a local directory with the same > > image 2500x (if you can't get the attachments through here, let me know and > > I'll email them direct). > > > > The interesting bit, is that I always get a linear-ish increase in time > > with the imagebuf, but if I graph the time it takes to open an ImageInput > > and query the spec, its pretty constant. > > > > Curious to know if you have any other thoughts! Thanks again! > > -J > > > > > > <oiiotest_imgbuf_perf.py><oiiotest_2500_cache.png><oiiotest_2500_cache_invalidate.png><oiiotest_2500_nocache.png>_______________________________________________ > > Oiio-dev mailing list > > [email protected] > > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org > > <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org> > > -- > Larry Gritz > [email protected] > > > _______________________________________________ > Oiio-dev mailing list > [email protected] > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org > <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org> > _______________________________________________ > Oiio-dev mailing list > [email protected] > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org -- Larry Gritz [email protected]
_______________________________________________ Oiio-dev mailing list [email protected] http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
