Hey, Thanks Steve and Nathan for all the suggestions.
I was considering just filling my own buffer and then accessing it when needed. However, not having any previous experience with this, I had 2 concerns: 1. Is it efficient to copy that much data around? My understanding is that Tiles and Rows are well tied into Nuke's cache system. If I copy their data to a new buffer, would I be loosing the benefits of fast returns when the original data is cached? 2. I considered requesting a whole tile for each input, but thought that getting one row at a time would be more efficient. In this particular case, each pixel on the output will only ever require that same pixel from one of the inputs (or from more than one of the inputs). Processing of that data is very minimal. The main objective is to copy data from the right place as fast as possible. If I make one big Tile for each input, wouldn't I be forcing Nuke to read all that data at once, even if I'm only interested in a few lines? Thanks again for the advice. Cheers, Ivan On Tue, Nov 5, 2013 at 10:05 AM, Steve Booth <st...@boothsoft.com> wrote: > I believe in 7, it's possible and reliable to execute the Tiles in _open, > instead of the engine (though, you might want to check on that.) If so, do > the Tiles once in _open, establish your own frame store, and access that in > the engine pixel-parse loop. Open is only ever called by the UI thread, as > opposed to engine which is multi-threaded. Thus, the threads only access > common memory, which should be safe. > > > > Steve > > > > > > *From:* nuke-dev-boun...@support.thefoundry.co.uk [mailto: > nuke-dev-boun...@support.thefoundry.co.uk] *On Behalf Of *Nathan Rusch > *Sent:* Tuesday, November 05, 2013 9:21 AM > > *To:* Nuke plug-in development discussion > *Subject:* Re: [Nuke-dev] Best way to keep Rows from multiple inputs > around? > > > > Ah, the fact that Deep samples within a given pixel can change the input > source definitely complicates any row-wise optimization. With that in mind, > I would agree with Steve that a Tile makes the most sense, as long as > asking for a Tile over the same image area from multiple worker threads > results in buffer sharing between them (which is my somewhat-limited > understanding). > > > > -Nathan > > > > *From:* Steve Booth <st...@boothsoft.com> > > *Sent:* Monday, November 04, 2013 9:57 PM > > *To:* 'Nuke plug-in development discussion'<nuke-dev@support.thefoundry.co.uk> > > *Subject:* RE: [Nuke-dev] Best way to keep Rows from multiple inputs > around? > > > > Ivan, > > > > I just saw this. Let me make another suggestion. Issuing millions of > interest/tile requests is going to be really inefficient, as you point > out. I would do one giant Tile request and get the entire frame for each > channel you need (if there is a high probability you will need them all) > into a set of memory-resident buffers, then simply pick what you need, when > you need it, directly from memory, without ever doing another interest call. > > > > Also... this seems like a perfect application for GPU/CUDA processing. > > > > Steve > > > > > > *From:* nuke-dev-boun...@support.thefoundry.co.uk [ > mailto:nuke-dev-boun...@support.thefoundry.co.uk<nuke-dev-boun...@support.thefoundry.co.uk>] > *On Behalf Of *Ivan Busquets > *Sent:* Monday, November 04, 2013 9:11 PM > *To:* Nuke plug-in development discussion > *Subject:* Re: [Nuke-dev] Best way to keep Rows from multiple inputs > around? > > > > Hey, > > > > Thanks Nathan. That certainly gives me some ideas to think about. > > > > In hindsight, though, I see I have presented an overly-simplified case as > an example. > > > > To expand a little bit more, there is one additional loop per pixel (think > of a DeepPixel, for example, and looping through all samples in that > DeepPixel) > > > > So, the flow of loops would look more like this: > > > > for each y: > > for each x: > > for each sample: > > inputNumber = figureOutInput(); > > sampleout = input(inputNumber)->at(x,y); > > > > That's why, considering how deeply buried those at() calls are, I was > thinking that it might be more optimal to cull full Rows for each input. > > > > Some of the other optimizations you suggest are great ideas, but in this > case I don't think I'll get much of a benefit from a) knowing whichs inputs > are really needed beforehand; and b) optimizing the X and R bounds of each > row. > > In this case scenario, I would say ALL inputs will be used most of the > time, and almost ALL pixels from each input will be needed as well. > > > > Thanks for pointing me towards Interests, though. Looking a bit closer, I > see there's also a InterestRatchet class which might be exactly what I > need. >From the class declaration in Interest.h: > > > > /** InterestRatchet > > > > ** If you create one of these, and pass it to 'Interest' then it will > remember > > ** which Iops it has previously called addInterest on, and not do > re-do these, > > ** thus saving time as addInterest involves contention > > ** > > ** Interests are removed again when the InterestRatchet is destroyed. > > **/ > > > > > > Thanks again, > > Ivan > > > > > > > > On Mon, Nov 4, 2013 at 7:24 PM, Nathan Rusch <nathan_ru...@hotmail.com> > wrote: > > Hey Ivan, > > > > > How would I go about keeping multiple Rows around, specially when the > number of inputs is not predefined? > > > > I'm certainly not the most qualified person to answer all of your > questions (especially about relative performance), but I'm going to take a > crack at this one anyway for fun. > > > > To keep things really simple, you could just loop through and create > threaded Interests on all input rows, store them in a vector, and then as > you loop through your mask row, use Interest::at() to get data from the one > you need based on the required input index. The header documentation seems > a little undecided on whether this would actually be faster than Iop::at(), > though. > > > > > > A slightly more involved approach would be to pre-determine which inputs > you actually need data from. Loop through your "mask" row first, determine > which input each pixel indicates, and if you haven't already added a row > for that input, add one. Then go through your array of rows in a second > loop and pass each one to a call to get() on its corresponding input. > > > > For some reason this still feels like a somewhat primitive way of doing > things, but it would at least prevent you from pulling in data from inputs > that you don't need. You could also use threaded Interests instead of Rows, > which may be more efficient for concurrently fetching data from multiple > inputs in preparation for generating your output (I'll leave that to > someone else to answer). > > > > > > Finally, another variation of this that seems like it could be more > efficient would be to loop through your "mask" row as before, but instead > of creating Rows or Interests outright, just store X and R bounding > coordinates for each input you're going to need first. In other words, > start with some sort of signal value for each input saying "I don't need > this", but as you encounter mask pixels that say otherwise, remember the > first X position for each input, and then keep a running max R for each as > you encounter subsequent required positions. Then, in a second loop, you > can create your Rows or Interests with better bounds to save > memory/calculation time. > > > > I have the same hunch you do about fetching whole rows being more > efficient than piece-wise calls to Iop::at(), but if you can predetermine > the row positions of the pixels you need from a given input, you could > potentially make some sort of decision about how to get them, and in some > cases, it may actually be faster to use Iop::at() than to ask for a whole > row (consider a case where you only need 2 pixels from input N: the first > and last in the row). > > > > > > Anyway, sorry for the brain dump. Looking forward to what others have to > say as well. > > > > > -Nathan > ------------------------------ > > Date: Mon, 4 Nov 2013 18:13:09 -0800 > From: ivanbusqu...@gmail.com > To: nuke-dev@support.thefoundry.co.uk > Subject: [Nuke-dev] Best way to keep Rows from multiple inputs around? > > > > Hi, > > I'm hoping someone with more experience with the Nuke API can help me out > with the following: > > - I have an algorithm that wants to use data from multiple inputs. > > - I only know what input I need to pull data from once I'm in a per-pixel > loop. (aka, for each pixel, the data from one of the inputs drives which > one of the other inputs is needed) > > - To avoid having to do Iop::at() or Iop::sample() calls, I would rather > fill a Row for each one of the inputs BEFORE the per-pixel loop, and then > access that Row later on. > > My assumption is that this will be more memory hungry, but faster than > calling at() or sample(). > However, I have questions: > > - Is this assumption correct? Does this kind of optimization make sense? > > - How would I go about keeping multiple Rows around, specially when the > number of inputs is not predefined? > > - I've thought of creating my own "uber" float array and just keep > appending all rows to it. Then figure out the offset within that array > during the per-pixel loop. Is there a more straightforward approach that > someone could recommend? > > For the sake of clarity, I'm looking to turn something like this > (pseudocode) > > for each y: > > for each x: > > inputNumber = figureOutInput(); > > out = input(inputNumber)->at(x,y); > > > > Into something like this: > > for each y: > > for each inputNumber in inputs: > > // need to keep a row for each input > > row(n) = input(inputNumber)->get(row); > > for each x: > inputNumber = figureOutInput(); > out = row(inputNumber)[x] > > > > > Hope that makes sense? Help? > > Thanks, > > Ivan > > > > > > _______________________________________________ Nuke-dev mailing list > Nuke-dev@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/ > http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-dev > > > _______________________________________________ > Nuke-dev mailing list > Nuke-dev@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/ > http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-dev > > > ------------------------------ > > _______________________________________________ > Nuke-dev mailing list > Nuke-dev@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/ > http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-dev > > _______________________________________________ > Nuke-dev mailing list > Nuke-dev@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/ > http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-dev > >
_______________________________________________ Nuke-dev mailing list Nuke-dev@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-dev
