Re: [Bf-committers] Proposal: Blender OpenCL compositor

[François T .]

I guess he is talking of algorithm where the entire buffer needs to be
evaluated before processing the pixels. Like knowing which is going to be
the Max Luma value in your buffer to divide all your pixels by this value.
(very simple example, but just for the case) or something like Retinex
algorithm, which recontrast localy based on value from the entire buffer, if
you only have tiles and some part of the image at the process time, that
would be an issue.
But even with that I don't understand why it would be a problem. as I
understand it, only the node itself does the tile, but the I/O of each nodes
is full buffers right ?
I don't know how this works exactly, but I can understand his fear about it,
but again, I'm pretty sure we are not the first compositor node doing tile
base right :) ?



2011/1/22 Jeroen Bakker 

> On 01/21/2011 04:14 PM, Aurel W. wrote:
> > You are talking about things such as convolution with a defined kernel
> > size. There are other operations and a compositor truly transforms an
> > image to another image and not pixels to pixels etc. If it's
> > implemented in such a naive way, the compositor will be very limited.
> > I got a very bad feeling about this
> >
> > Ok, let's normalize an image with a tile based approach,... uh damn
> it
> Aurel, don't worry on that. Tile based is that the output is part of a
> tile. But the input data can be the whole or a part of the image. On the
> technical side there will be some issues to overcome (mostly device
> memory related). Btw there are possibilities when you need every image
> pixel as input to use a intermediate to reduce memory need. I did this
> already in the defocus node.
>
> Please help me to determine the case when a whole output image is
> needed. IMO input is readonly and output is writeonly. I don't see the
> need atm to support whole output images in a 'per output pixel'
> approach. And every 'per input pixel' approach can be written by a 'per
> output pixel' approach. In the current nodes the two approaches are mixed.
>
> Jeroen
>
> ___
> Bf-committers mailing list
> Bf-committers@blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>



-- 

François Tarlier
www.francois-tarlier.com
www.linkedin.com/in/francoistarlier
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Proposal: Blender OpenCL compositor

[Jeroen Bakker]

Hi,

I missed some IRC chats on this subject and there are some definition 
problems. Like what are tiles in the perspective of a user/artist and 
what are tiles in perspective of parallelization.
Both definitions are right, but developers and users mix the definition 
and the meaning. Sorry for that.
With tile I meant a part of a larger entity, to be used as Ton mentioned.

On th IRC there was a sample of 'normalize' an image. My question back, 
how will this work in an movie pipeline. Where every frame can have 
different high and low value? Don't you want/need to control the min and 
max values? Please clarify.

If we need a normalize node, this node has two kernels. The first kernel 
will find the lowest and highest value of a complete image. This is 
passed to the second kernel and here is a pixel processor where the 
colors are changed based on values of kernel 1 and the colors in input 
image. The highest/lowest value is calculated once (not parallelized) 
the pixel processor is parallelized.

Jeroen



On 01/22/2011 08:50 AM, Jeroen Bakker wrote:
> On 01/21/2011 04:14 PM, Aurel W. wrote:
>> You are talking about things such as convolution with a defined kernel
>> size. There are other operations and a compositor truly transforms an
>> image to another image and not pixels to pixels etc. If it's
>> implemented in such a naive way, the compositor will be very limited.
>> I got a very bad feeling about this
>>
>> Ok, let's normalize an image with a tile based approach,... uh damn it
> Aurel, don't worry on that. Tile based is that the output is part of a
> tile. But the input data can be the whole or a part of the image. On the
> technical side there will be some issues to overcome (mostly device
> memory related). Btw there are possibilities when you need every image
> pixel as input to use a intermediate to reduce memory need. I did this
> already in the defocus node.
>
> Please help me to determine the case when a whole output image is
> needed. IMO input is readonly and output is writeonly. I don't see the
> need atm to support whole output images in a 'per output pixel'
> approach. And every 'per input pixel' approach can be written by a 'per
> output pixel' approach. In the current nodes the two approaches are mixed.
>
> Jeroen
>
> ___
> Bf-committers mailing list
> Bf-committers@blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>


-- 

Met vriendelijke groet,

Jeroen Bakker

*At Mind BV
*

Telefoon: 06 50 611 262
E-mail: j.bak...@atmind.nl 


___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Proposal: Blender OpenCL compositor

[Aurel W.]

Hi Jeron,

> Please help me to determine the case when a whole output image is
> needed. IMO input is readonly and output is writeonly. I don't see the
> need atm to support whole output images in a 'per output pixel'
> approach. And every 'per input pixel' approach can be written by a 'per
> output pixel' approach. In the current nodes the two approaches are mixed.
The problem of the concept of pixel to pixel operations is also that
this tends to be implemented with a lot of overhead. Like having 3
frames on the call stack for adding two pixels and this for every
pixel in the buffer, it is really nasty. This is why even adding
buffers together is rather inefficient at the moment. Another example
would be the filter node, with these pixel_processors for convolution.
If you really think about low level efficiency, down to the level of
single instructions, a lot could be done better at the moment.

I also realize that the argument "it would work with the current
compositor" is a strong argument. But I got some problems with that.
First of all I think that a compositor should be in principal be able
to support all image processing operations. I think it's a rather bad
idea to be stuck with a very limited architecture, which already
requires a bunch of hacks to implement the functionality of current
nodes as those doing convolution.

Another problem I see with tiling is, that you are doing spacial
partitioning and are therefore stuck in the spatial domain. But there
are a lot of possibilities of working in gradient and frequency
domain, also including speedups. But you won't be able to convert a
tile to gradient domain, because you can't determine the correct
gradient on the borders. When you want to work in frequency domain you
also run into issues with tiling, because of your spacial
partitioning.

But back to the simple issue with operations, which need full buffer
access. I agree that this could be still done with tiling, because you
can simply compute all input tiles and just access those when
computing one single output tile. So this is sort of how this should
work? At least your diagram in your document looks like this. Any
other workarounds like using overlapping tiles for the very special
case of a 3*3 kernel convolution are just hacks, but will prevent the
implementation of any future nodes, which have other non pixel->pixel
operations.

Such future node for instance could be tone-mapping. This is for e.g.
a standard feature in lux, so I guess it's not that absurd to include
such features in blenders compositor. And some tone mapping algorithms
need to operate on the entire image.

In terms of memory usage, caching, etc. if we assume that only
reasonable sized buffers are used, let's say up to 64MB, I also don't
see the strong benefits in using tiles rather than buffers, which hold
the entire image. But maybe you have to be more specific about the
caching scheme you want to use here.

aurel
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Proposal: Blender OpenCL compositor

[Matt Ebb]

On Sat, Jan 22, 2011 at 8:11 PM, Aurel W.  wrote:
> I also realize that the argument "it would work with the current
> compositor" is a strong argument. But I got some problems with that.
> First of all I think that a compositor should be in principal be able
> to support all image processing operations.

On the contrary, I think the compositor should be designed and
optimised for its purpose, compositing CGI/vfx imagery. It doesn't
need to be a completely generalised image processing system, it just
needs to do what it's intended for, well. So far I've seen a mostly
theoretical objections here, but I think it's important to keep
focused on enabling people to produce shots.

> But back to the simple issue with operations, which need full buffer
> access. I agree that this could be still done with tiling, because you
> can simply compute all input tiles and just access those when
> computing one single output tile.

Or rather the tiles that are necessary at any given time. In the case
of the Normalize node for example (which is mostly useless for
animated sequences, as are any tone mapping operators that work in a
similar way), it would be possible to retrieve each tile one by one in
a pre-process, read and store the statistical information, and then
apply that per tile or even per pixel.

> In terms of memory usage, caching, etc. if we assume that only
> reasonable sized buffers are used, let's say up to 64MB, I also don't
> see the strong benefits in using tiles rather than buffers, which hold
> the entire image.

The benefits are lower memory usage, and better/easier parallelisation.
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Proposal: Blender OpenCL compositor

[Aurel W.]

> On the contrary, I think the compositor should be designed and
> optimised for its purpose, compositing CGI/vfx imagery. It doesn't
> need to be a completely generalised image processing system, it just
> needs to do what it's intended for, well. So far I've seen a mostly
> theoretical objections here, but I think it's important to keep
> focused on enabling people to produce shots.
Even if we just assume the existing nodes, there are lots of issues
with tiles. To be future-proof, also other nodes and operations have
to be considered. I mentioned some, they might not be the best
examples, but they demonstrate issues with the design. Strong limits
in this tile based design are a con. Sorry if this seems to be just
"theoretical objections", but just comparing a design to the existing
nodes won't do it.

> Or rather the tiles that are necessary at any given time. In the case
> of the Normalize node for example (which is mostly useless for
> animated sequences, as are any tone mapping operators that work in a
> similar way), it would be possible to retrieve each tile one by one in
> a pre-process, read and store the statistical information, and then
> apply that per tile or even per pixel.
Again, just examples for operations on images,... and I guess
tone-mapping isn't such a bad one, especially if you consider
compositing of single images, not animations.

> The benefits are lower memory usage, and better/easier parallelisation.
In practice, if you assume that your memory can hold multiple buffers
anyway, I can't significant improvements in memory usage. We also have
to distinguish between two use cases here, the one where compositing
graph is just executed once and the one where a user interactively
adjusts settings and wants to keep intermediate results in memory.
Again, there is no proposal for a caching scheme for the tiled based
solution in the interactive case yet and I can't think of anything
that would have large benefits compared to work on full buffers and
also cache those.

I also highly doubt that this will lead to better/easier
parallelisation. I still think that more fine grained parallelisation
in each individual node, operating on the entire buffer, would turn
out better in practice.

At least I want to have a discussion on this. Just to assume
prematurely, that tiles will give better performance is not a good
idea.

aurel
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Proposal: Blender OpenCL compositor

[François T .]

>
> Jeron : Like what are tiles in the perspective of a user/artist and

what are tiles in perspective of parallelization.

Both definitions are right, but developers and users mix the definition

and the meaning. Sorry for that.


Thats what I thought :)



Aurel : Again, just examples for operations on images,... and I guess

tone-mapping isn't such a bad one, especially if you consider

compositing of single images, not animations.


the goal here is to build a compositor and as Matt says, it should do what
it's supposed too. I know that as for now the compositor had a design to
enhanced render and mostly still (and IMO that the reason it got some
limitation today too). As xsi had its fxtree or whatever.
Thinking about using it for still is like if you were telling me you would
use After Effects to do Photoshop stuff. It is possible and yes they could
make one tool of both. But there is a good reason it is two different
software even if at the end they do really similar things.
Just to say, I believe the design should concentrate on large image memory
(4k & higher is coming in future for sure) and animation above all.

>


some gradiant base algorithm & very fast blur are in needs of full buffer
for sure, but I don't understand why some nodes cannot says "I need full
buffer, so I'll wait all my parents to compute and give me a FB as input"
and other nodes (by default) based on tiles. So only a few  would be slower
than other, but still everybody would happy ?

Actually I wonder if Nuke is not doing some kind of similar thing. Matt ?
The reason it makes me think of that, is on some Nukes scripts, I have seen
some nodes updating all together, and then one of them updating kind of
seperatly, like if it was waiting for something. But again, maybe i don't
really understand the issue here, my apology :(

F


2011/1/22 Aurel W. 

> > On the contrary, I think the compositor should be designed and
> > optimised for its purpose, compositing CGI/vfx imagery. It doesn't
> > need to be a completely generalised image processing system, it just
> > needs to do what it's intended for, well. So far I've seen a mostly
> > theoretical objections here, but I think it's important to keep
> > focused on enabling people to produce shots.
> Even if we just assume the existing nodes, there are lots of issues
> with tiles. To be future-proof, also other nodes and operations have
> to be considered. I mentioned some, they might not be the best
> examples, but they demonstrate issues with the design. Strong limits
> in this tile based design are a con. Sorry if this seems to be just
> "theoretical objections", but just comparing a design to the existing
> nodes won't do it.
>
> > Or rather the tiles that are necessary at any given time. In the case
> > of the Normalize node for example (which is mostly useless for
> > animated sequences, as are any tone mapping operators that work in a
> > similar way), it would be possible to retrieve each tile one by one in
> > a pre-process, read and store the statistical information, and then
> > apply that per tile or even per pixel.
> Again, just examples for operations on images,... and I guess
> tone-mapping isn't such a bad one, especially if you consider
> compositing of single images, not animations.
>
> > The benefits are lower memory usage, and better/easier parallelisation.
> In practice, if you assume that your memory can hold multiple buffers
> anyway, I can't significant improvements in memory usage. We also have
> to distinguish between two use cases here, the one where compositing
> graph is just executed once and the one where a user interactively
> adjusts settings and wants to keep intermediate results in memory.
> Again, there is no proposal for a caching scheme for the tiled based
> solution in the interactive case yet and I can't think of anything
> that would have large benefits compared to work on full buffers and
> also cache those.
>
> I also highly doubt that this will lead to better/easier
> parallelisation. I still think that more fine grained parallelisation
> in each individual node, operating on the entire buffer, would turn
> out better in practice.
>
> At least I want to have a discussion on this. Just to assume
> prematurely, that tiles will give better performance is not a good
> idea.
>
> aurel
> ___
> Bf-committers mailing list
> Bf-committers@blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>



-- 

François Tarlier
www.francois-tarlier.com
www.linkedin.com/in/francoistarlier
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Proposal: Blender OpenCL compositor

[Aurel W.]

> some gradiant base algorithm & very fast blur are in needs of full buffer
> for sure, but I don't understand why some nodes cannot says "I need full
> buffer, so I'll wait all my parents to compute and give me a FB as input"
> and other nodes (by default) based on tiles. So only a few  would be slower
> than other, but still everybody would happy ?

Yes something like that would be necessary. I guess in practice it
will be very hard to determine the required tiles, so maybe there will
be only two cases, one where only one tile is needed and the one where
simply all tiles are needed.

I am also worried about the memory layout of this. Single tiles would
be computed to separate data structures, maybe just a single array.
All tiles are computed like this for an entire image. The next node,
which needs to operate on the entire image now has to access
individually pixels in all tiles. So you have two options, introduce
some sort of abstraction to access these pixels, or copy all tiles to
a single buffer, which then gets processed. The first one adds a lot
of overhead and cache unfriendliness. The second one also adds
overhead and memory usage by copying. Of course this would need
testing and better analysis, but it can tremendously slow things down.

> the goal here is to build a compositor and as Matt says, it should do what
it's supposed too.
well, of course,...

aurel
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Proposal: Blender OpenCL compositor

[Ton Roosendaal]

Hi,

To be honest, long winded discussions on ways how to implement stuff  
should not take away the freedom for a developer to find out him/ 
herself the optimal cases. I'm confident that Jeroen is aware of  
boundary cases here, and he will try to find a good balance for  
practical usage.

For as long we agree on existing and future demands on compositing in  
Blender, we should give him our blessings :)

Relevant specs are for example:
- desired input methods, like storage, types, UI workflow,  
colorspaces, alpha, (plugins?)
- desired output specifications, like memory/cpu/gpu performance and  
visual feedback

-Ton-


Ton Roosendaal  Blender Foundation   t...@blender.orgwww.blender.org
Blender Institute   Entrepotdok 57A  1018AD Amsterdam   The Netherlands

On 22 Jan, 2011, at 13:28, Aurel W. wrote:

>> some gradiant base algorithm & very fast blur are in needs of full  
>> buffer
>> for sure, but I don't understand why some nodes cannot says "I need  
>> full
>> buffer, so I'll wait all my parents to compute and give me a FB as  
>> input"
>> and other nodes (by default) based on tiles. So only a few  would  
>> be slower
>> than other, but still everybody would happy ?
>
> Yes something like that would be necessary. I guess in practice it
> will be very hard to determine the required tiles, so maybe there will
> be only two cases, one where only one tile is needed and the one where
> simply all tiles are needed.
>
> I am also worried about the memory layout of this. Single tiles would
> be computed to separate data structures, maybe just a single array.
> All tiles are computed like this for an entire image. The next node,
> which needs to operate on the entire image now has to access
> individually pixels in all tiles. So you have two options, introduce
> some sort of abstraction to access these pixels, or copy all tiles to
> a single buffer, which then gets processed. The first one adds a lot
> of overhead and cache unfriendliness. The second one also adds
> overhead and memory usage by copying. Of course this would need
> testing and better analysis, but it can tremendously slow things down.
>
>> the goal here is to build a compositor and as Matt says, it should  
>> do what
> it's supposed too.
> well, of course,...
>
> aurel
> ___
> Bf-committers mailing list
> Bf-committers@blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>

___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Removing auto registration

[Ton Roosendaal]

Hi Martin,

Not capable of grasping your discussions with Campbell on this topic,  
I proposed to ask arbitration by a third person who knows this well.  
We're all equally stubborn fallible humans you know, and who's "right"  
or "wrong" then is less relevant than just moving forward. :)

If you think Brecht has all information you want him to know, let's go  
for his advice.

-Ton-


Ton Roosendaal  Blender Foundation   t...@blender.orgwww.blender.org
Blender Institute   Entrepotdok 57A  1018AD Amsterdam   The Netherlands

On 17 Jan, 2011, at 14:20, Martin Poirier wrote:

>
>
> --- On Mon, 1/17/11, Campbell Barton  wrote:
>
>> Agree panel order shouldn't be a
>> factor in this discussion, it should
>> be solved irrespective of registration so addons panels can
>> be added in a logical order.
>
> Ideally, this should be done before going out of beta.
>
>> Though I'm still for auto-registration removal even if its
>> bug free,
>> most likely re-iterating from previous mails.
>
> It's not so much the reiteration of your same arguments that bugs me  
> but the fact that you've completely ignored the problems with the  
> registration order and properties registration that I've highlighted  
> many times and that has to be dealt with regardless of the  
> registration method if we want to correctly and cleanly handle  
> enabling and disabling addons (which you seem to think is an  
> argument for manual registration but is completely irrelevant).
>
>> - to me it feels mysterious that blender is operating on
>> classes
>> without being asked & errors don't trace back to
>> authors code.
>
> The traceback issue can be fixed quite simply. The Metaclass has  
> more info than manual registration on the class definition itself  
> (you can have the file and line where the class is defined if that's  
> what you want).
>
> As for blender operating on classes without being asked, that's the  
> whole point of inheritance.
>
>> - in simple cases where all classes are registered its not
>> a big win
>> to have it automatic, in complicated cases of dynamic
>> runtime registration this gets in the way.
>
> Yes, you did raise that point last time, it was bunk back then too.
>
> Can you show an example of dynamic runtime registration that deals  
> correctly with registration order and unregistration without making  
> a truckload of assumptions or not working at all?
>
> Registration order is tightly coupled with internal workings of  
> Blender, exposing that to python is completely ridiculous.
>
>> - it makes internals more complicated we need to support -
>> 3 cases:
>> direct execution, modules and addons.
>
> There's only two cases, runtime execution and delayed execution.
>
> Modules are addons that are registered automatically after definition.
>
>> - Matt Ebb and Nathan Vegdahl have complained about
>> auto-registration
>> in its current state fir renderman support which does
>> dynamic
>> generated classes from shaders, and rigify for rig types.
>
> Didn't I addressed Nathan's issues last time?
>
> There's nothing about autoregistration that prevents runtime  
> definition of classes.
>
>> It is regrettable that I accepted this patch in the first
>> place, but I
>> felt some obligation to do so since Martin addressed the
>> issues that
>> concerned me, also because Brecht and Andria also approved
>> of this
>> functionality at the time.
>
> It's regrettable that I tried to address the real problems two  
> months ago, after which you said you'd have to look into it yourself  
> and never came back until now, trying to force a decision again.
>
> It's regrettable that you think autoregistration is an opaque black  
> box just because there's the word metaclass in there.
>
> But most of all, it's regrettable that you think shoving that back  
> in the ands of python developers will solve all problems magically  
> when in fact it means having to write more documentation with  
> warnings all over the place about proper registration order.
>
> Martin
>
>
> ___
> Bf-committers mailing list
> Bf-committers@blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers

___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Removing auto registration

[Martin Poirier]

Hi Ton,

I'll try to explain this as simply as possible.

Registering python classes with the RNA system has to be done in a specific 
order to solve dependency issues.

Unregistering (when unloading) also has to be done in a specific order (not the 
same, maybe not just reverse order).

There are situations for which a good order isn't currently known (theorically, 
on paper).

This is the registration order problem.


If we choose to go with manual registration, we have to solve that problem and 
have very special guidelines for python programmers, otherwise stuff will break.

If we choose to go with automatic registration, we have to solve that problem 
and then write code that applies that solution (if that can't be done, at last 
resort, there could be a way to turn it off in specific cases and have to do it 
manually).

In either case, not solving the registration problem is a bad thing and things 
will break.

In my wiki pages, I've proposed a solution to the problem that seems simple and 
can easily be applied to both automatic or manual registration (it's a simple 
set of rules like: UI first, then operators, then ...), all that's needed is 
for other people to validate this.

Again, regardless of what registration method we use, the order problem has to 
be fixed (before we switched to automatic registration, people use to shuffle 
stuff around until it appeared to work, that's not a solution).

Is that clear?

Martin

--- On Sat, 1/22/11, Ton Roosendaal  wrote:

> From: Ton Roosendaal 
> Subject: Re: [Bf-committers] Removing auto registration
> To: "bf-blender developers" 
> Received: Saturday, January 22, 2011, 9:57 AM
> Hi Martin,
> 
> Not capable of grasping your discussions with Campbell on
> this topic,  
> I proposed to ask arbitration by a third person who knows
> this well.  
> We're all equally stubborn fallible humans you know, and
> who's "right"  
> or "wrong" then is less relevant than just moving forward.
> :)
> 
> If you think Brecht has all information you want him to
> know, let's go  
> for his advice.
> 
> -Ton-
> 
> 
> Ton Roosendaal  Blender Foundation   t...@blender.org 
>   www.blender.org
> Blender Institute   Entrepotdok 57A 
> 1018AD Amsterdam   The Netherlands
> 
> On 17 Jan, 2011, at 14:20, Martin Poirier wrote:
> 
> >
> >
> > --- On Mon, 1/17/11, Campbell Barton 
> wrote:
> >
> >> Agree panel order shouldn't be a
> >> factor in this discussion, it should
> >> be solved irrespective of registration so addons
> panels can
> >> be added in a logical order.
> >
> > Ideally, this should be done before going out of
> beta.
> >
> >> Though I'm still for auto-registration removal
> even if its
> >> bug free,
> >> most likely re-iterating from previous mails.
> >
> > It's not so much the reiteration of your same
> arguments that bugs me  
> > but the fact that you've completely ignored the
> problems with the  
> > registration order and properties registration that
> I've highlighted  
> > many times and that has to be dealt with regardless of
> the  
> > registration method if we want to correctly and
> cleanly handle  
> > enabling and disabling addons (which you seem to think
> is an  
> > argument for manual registration but is completely
> irrelevant).
> >
> >> - to me it feels mysterious that blender is
> operating on
> >> classes
> >> without being asked & errors don't trace back
> to
> >> authors code.
> >
> > The traceback issue can be fixed quite simply. The
> Metaclass has  
> > more info than manual registration on the class
> definition itself  
> > (you can have the file and line where the class is
> defined if that's  
> > what you want).
> >
> > As for blender operating on classes without being
> asked, that's the  
> > whole point of inheritance.
> >
> >> - in simple cases where all classes are registered
> its not
> >> a big win
> >> to have it automatic, in complicated cases of
> dynamic
> >> runtime registration this gets in the way.
> >
> > Yes, you did raise that point last time, it was bunk
> back then too.
> >
> > Can you show an example of dynamic runtime
> registration that deals  
> > correctly with registration order and unregistration
> without making  
> > a truckload of assumptions or not working at all?
> >
> > Registration order is tightly coupled with internal
> workings of  
> > Blender, exposing that to python is completely
> ridiculous.
> >
> >> - it makes internals more complicated we need to
> support -
> >> 3 cases:
> >> direct execution, modules and addons.
> >
> > There's only two cases, runtime execution and delayed
> execution.
> >
> > Modules are addons that are registered automatically
> after definition.
> >
> >> - Matt Ebb and Nathan Vegdahl have complained
> about
> >> auto-registration
> >> in its current state fir renderman support which
> does
> >> dynamic
> >> generated classes from shaders, and rigify for rig
> types.
> >
> > Didn't I addressed Nathan

Re: [Bf-committers] Removing auto registration

[Ton Roosendaal]

Hi,

Yes that sounds clean and clear. But apparently there's more ways to  
solve the problem, and either way has pros and cons. When visions on  
such problems don't align, then how to decide? The maintainers and  
main contributors to the code then can have a final word.

-Ton-


Ton Roosendaal  Blender Foundation   t...@blender.orgwww.blender.org
Blender Institute   Entrepotdok 57A  1018AD Amsterdam   The Netherlands

On 22 Jan, 2011, at 16:24, Martin Poirier wrote:

> Hi Ton,
>
> I'll try to explain this as simply as possible.
>
> Registering python classes with the RNA system has to be done in a  
> specific order to solve dependency issues.
>
> Unregistering (when unloading) also has to be done in a specific  
> order (not the same, maybe not just reverse order).
>
> There are situations for which a good order isn't currently known  
> (theorically, on paper).
>
> This is the registration order problem.
>
>
> If we choose to go with manual registration, we have to solve that  
> problem and have very special guidelines for python programmers,  
> otherwise stuff will break.
>
> If we choose to go with automatic registration, we have to solve  
> that problem and then write code that applies that solution (if that  
> can't be done, at last resort, there could be a way to turn it off  
> in specific cases and have to do it manually).
>
> In either case, not solving the registration problem is a bad thing  
> and things will break.
>
> In my wiki pages, I've proposed a solution to the problem that seems  
> simple and can easily be applied to both automatic or manual  
> registration (it's a simple set of rules like: UI first, then  
> operators, then ...), all that's needed is for other people to  
> validate this.
>
> Again, regardless of what registration method we use, the order  
> problem has to be fixed (before we switched to automatic  
> registration, people use to shuffle stuff around until it appeared  
> to work, that's not a solution).
>
> Is that clear?
>
> Martin
>
> --- On Sat, 1/22/11, Ton Roosendaal  wrote:
>
>> From: Ton Roosendaal 
>> Subject: Re: [Bf-committers] Removing auto registration
>> To: "bf-blender developers" 
>> Received: Saturday, January 22, 2011, 9:57 AM
>> Hi Martin,
>>
>> Not capable of grasping your discussions with Campbell on
>> this topic,
>> I proposed to ask arbitration by a third person who knows
>> this well.
>> We're all equally stubborn fallible humans you know, and
>> who's "right"
>> or "wrong" then is less relevant than just moving forward.
>> :)
>>
>> If you think Brecht has all information you want him to
>> know, let's go
>> for his advice.
>>
>> -Ton-
>>
>> 
>> Ton Roosendaal  Blender Foundation   t...@blender.org
>>   www.blender.org
>> Blender Institute   Entrepotdok 57A
>> 1018AD Amsterdam   The Netherlands
>>
>> On 17 Jan, 2011, at 14:20, Martin Poirier wrote:
>>
>>>
>>>
>>> --- On Mon, 1/17/11, Campbell Barton 
>> wrote:
>>>
 Agree panel order shouldn't be a
 factor in this discussion, it should
 be solved irrespective of registration so addons
>> panels can
 be added in a logical order.
>>>
>>> Ideally, this should be done before going out of
>> beta.
>>>
 Though I'm still for auto-registration removal
>> even if its
 bug free,
 most likely re-iterating from previous mails.
>>>
>>> It's not so much the reiteration of your same
>> arguments that bugs me
>>> but the fact that you've completely ignored the
>> problems with the
>>> registration order and properties registration that
>> I've highlighted
>>> many times and that has to be dealt with regardless of
>> the
>>> registration method if we want to correctly and
>> cleanly handle
>>> enabling and disabling addons (which you seem to think
>> is an
>>> argument for manual registration but is completely
>> irrelevant).
>>>
 - to me it feels mysterious that blender is
>> operating on
 classes
 without being asked & errors don't trace back
>> to
 authors code.
>>>
>>> The traceback issue can be fixed quite simply. The
>> Metaclass has
>>> more info than manual registration on the class
>> definition itself
>>> (you can have the file and line where the class is
>> defined if that's
>>> what you want).
>>>
>>> As for blender operating on classes without being
>> asked, that's the
>>> whole point of inheritance.
>>>
 - in simple cases where all classes are registered
>> its not
 a big win
 to have it automatic, in complicated cases of
>> dynamic
 runtime registration this gets in the way.
>>>
>>> Yes, you did raise that point last time, it was bunk
>> back then too.
>>>
>>> Can you show an example of dynamic runtime
>> registration that deals
>>> correctly with registration order and unregistration
>> without making
>>> a truckload of assumptions or not working at all?
>>>
>>> Registration order

Re: [Bf-committers] Removing auto registration

[Martin Poirier]

Hi,

The problem I have with taking a decision now is that we haven't solved the 
order problem.

If it turns out to require very complex rules, it might be easier to solve 
automatically than manually (or vice versa).

What I mean is that I don't think we have all the data to take a good decision 
at this point. Especially if correctly solving the problem means changes in the 
RNA api layer (it might), it would be better to start early (before the API is 
stabilized too much).

Martin

--- On Sat, 1/22/11, Ton Roosendaal  wrote:

> From: Ton Roosendaal 
> Subject: Re: [Bf-committers] Removing auto registration
> To: "bf-blender developers" 
> Received: Saturday, January 22, 2011, 11:24 AM
> Hi,
> 
> Yes that sounds clean and clear. But apparently there's
> more ways to  
> solve the problem, and either way has pros and cons. When
> visions on  
> such problems don't align, then how to decide? The
> maintainers and  
> main contributors to the code then can have a final word.
> 
> -Ton-
> 
> 
> Ton Roosendaal  Blender Foundation   t...@blender.org 
>   www.blender.org
> Blender Institute   Entrepotdok 57A 
> 1018AD Amsterdam   The Netherlands
> 
> On 22 Jan, 2011, at 16:24, Martin Poirier wrote:
> 
> > Hi Ton,
> >
> > I'll try to explain this as simply as possible.
> >
> > Registering python classes with the RNA system has to
> be done in a  
> > specific order to solve dependency issues.
> >
> > Unregistering (when unloading) also has to be done in
> a specific  
> > order (not the same, maybe not just reverse order).
> >
> > There are situations for which a good order isn't
> currently known  
> > (theorically, on paper).
> >
> > This is the registration order problem.
> >
> >
> > If we choose to go with manual registration, we have
> to solve that  
> > problem and have very special guidelines for python
> programmers,  
> > otherwise stuff will break.
> >
> > If we choose to go with automatic registration, we
> have to solve  
> > that problem and then write code that applies that
> solution (if that  
> > can't be done, at last resort, there could be a way to
> turn it off  
> > in specific cases and have to do it manually).
> >
> > In either case, not solving the registration problem
> is a bad thing  
> > and things will break.
> >
> > In my wiki pages, I've proposed a solution to the
> problem that seems  
> > simple and can easily be applied to both automatic or
> manual  
> > registration (it's a simple set of rules like: UI
> first, then  
> > operators, then ...), all that's needed is for other
> people to  
> > validate this.
> >
> > Again, regardless of what registration method we use,
> the order  
> > problem has to be fixed (before we switched to
> automatic  
> > registration, people use to shuffle stuff around until
> it appeared  
> > to work, that's not a solution).
> >
> > Is that clear?
> >
> > Martin
> >
> > --- On Sat, 1/22/11, Ton Roosendaal 
> wrote:
> >
> >> From: Ton Roosendaal 
> >> Subject: Re: [Bf-committers] Removing auto
> registration
> >> To: "bf-blender developers" 
> >> Received: Saturday, January 22, 2011, 9:57 AM
> >> Hi Martin,
> >>
> >> Not capable of grasping your discussions with
> Campbell on
> >> this topic,
> >> I proposed to ask arbitration by a third person
> who knows
> >> this well.
> >> We're all equally stubborn fallible humans you
> know, and
> >> who's "right"
> >> or "wrong" then is less relevant than just moving
> forward.
> >> :)
> >>
> >> If you think Brecht has all information you want
> him to
> >> know, let's go
> >> for his advice.
> >>
> >> -Ton-
> >>
> >>
> 
> >> Ton Roosendaal  Blender
> Foundation   t...@blender.org
> >>   www.blender.org
> >> Blender Institute   Entrepotdok
> 57A
> >> 1018AD Amsterdam   The Netherlands
> >>
> >> On 17 Jan, 2011, at 14:20, Martin Poirier wrote:
> >>
> >>>
> >>>
> >>> --- On Mon, 1/17/11, Campbell Barton 
> >> wrote:
> >>>
>  Agree panel order shouldn't be a
>  factor in this discussion, it should
>  be solved irrespective of registration so
> addons
> >> panels can
>  be added in a logical order.
> >>>
> >>> Ideally, this should be done before going out
> of
> >> beta.
> >>>
>  Though I'm still for auto-registration
> removal
> >> even if its
>  bug free,
>  most likely re-iterating from previous
> mails.
> >>>
> >>> It's not so much the reiteration of your same
> >> arguments that bugs me
> >>> but the fact that you've completely ignored
> the
> >> problems with the
> >>> registration order and properties registration
> that
> >> I've highlighted
> >>> many times and that has to be dealt with
> regardless of
> >> the
> >>> registration method if we want to correctly
> and
> >> cleanly handle
> >>> enabling and disabling addons (which you seem
> to think
> >> is an
> >>> argument for manual registration but is
> completely
> >> irrelevant

Re: [Bf-committers] Removing auto registration

[Brecht Van Lommel]

Hi,

I don't really see the order problem as a problem we have to solve. If
you're registering multiple things, then it seems logical to me that
if A references B, you register B first and then A. It's not clear to
me where these potential complex rules would show up?

Brecht.

On Sat, Jan 22, 2011 at 5:33 PM, Martin Poirier  wrote:
> Hi,
>
> The problem I have with taking a decision now is that we haven't solved the 
> order problem.
>
> If it turns out to require very complex rules, it might be easier to solve 
> automatically than manually (or vice versa).
>
> What I mean is that I don't think we have all the data to take a good 
> decision at this point. Especially if correctly solving the problem means 
> changes in the RNA api layer (it might), it would be better to start early 
> (before the API is stabilized too much).
>
> Martin
>
> --- On Sat, 1/22/11, Ton Roosendaal  wrote:
>
>> From: Ton Roosendaal 
>> Subject: Re: [Bf-committers] Removing auto registration
>> To: "bf-blender developers" 
>> Received: Saturday, January 22, 2011, 11:24 AM
>> Hi,
>>
>> Yes that sounds clean and clear. But apparently there's
>> more ways to
>> solve the problem, and either way has pros and cons. When
>> visions on
>> such problems don't align, then how to decide? The
>> maintainers and
>> main contributors to the code then can have a final word.
>>
>> -Ton-
>>
>> 
>> Ton Roosendaal  Blender Foundation   t...@blender.org
>>   www.blender.org
>> Blender Institute   Entrepotdok 57A
>> 1018AD Amsterdam   The Netherlands
>>
>> On 22 Jan, 2011, at 16:24, Martin Poirier wrote:
>>
>> > Hi Ton,
>> >
>> > I'll try to explain this as simply as possible.
>> >
>> > Registering python classes with the RNA system has to
>> be done in a
>> > specific order to solve dependency issues.
>> >
>> > Unregistering (when unloading) also has to be done in
>> a specific
>> > order (not the same, maybe not just reverse order).
>> >
>> > There are situations for which a good order isn't
>> currently known
>> > (theorically, on paper).
>> >
>> > This is the registration order problem.
>> >
>> >
>> > If we choose to go with manual registration, we have
>> to solve that
>> > problem and have very special guidelines for python
>> programmers,
>> > otherwise stuff will break.
>> >
>> > If we choose to go with automatic registration, we
>> have to solve
>> > that problem and then write code that applies that
>> solution (if that
>> > can't be done, at last resort, there could be a way to
>> turn it off
>> > in specific cases and have to do it manually).
>> >
>> > In either case, not solving the registration problem
>> is a bad thing
>> > and things will break.
>> >
>> > In my wiki pages, I've proposed a solution to the
>> problem that seems
>> > simple and can easily be applied to both automatic or
>> manual
>> > registration (it's a simple set of rules like: UI
>> first, then
>> > operators, then ...), all that's needed is for other
>> people to
>> > validate this.
>> >
>> > Again, regardless of what registration method we use,
>> the order
>> > problem has to be fixed (before we switched to
>> automatic
>> > registration, people use to shuffle stuff around until
>> it appeared
>> > to work, that's not a solution).
>> >
>> > Is that clear?
>> >
>> > Martin
>> >
>> > --- On Sat, 1/22/11, Ton Roosendaal 
>> wrote:
>> >
>> >> From: Ton Roosendaal 
>> >> Subject: Re: [Bf-committers] Removing auto
>> registration
>> >> To: "bf-blender developers" 
>> >> Received: Saturday, January 22, 2011, 9:57 AM
>> >> Hi Martin,
>> >>
>> >> Not capable of grasping your discussions with
>> Campbell on
>> >> this topic,
>> >> I proposed to ask arbitration by a third person
>> who knows
>> >> this well.
>> >> We're all equally stubborn fallible humans you
>> know, and
>> >> who's "right"
>> >> or "wrong" then is less relevant than just moving
>> forward.
>> >> :)
>> >>
>> >> If you think Brecht has all information you want
>> him to
>> >> know, let's go
>> >> for his advice.
>> >>
>> >> -Ton-
>> >>
>> >>
>> 
>> >> Ton Roosendaal  Blender
>> Foundation   t...@blender.org
>> >>   www.blender.org
>> >> Blender Institute   Entrepotdok
>> 57A
>> >> 1018AD Amsterdam   The Netherlands
>> >>
>> >> On 17 Jan, 2011, at 14:20, Martin Poirier wrote:
>> >>
>> >>>
>> >>>
>> >>> --- On Mon, 1/17/11, Campbell Barton 
>> >> wrote:
>> >>>
>>  Agree panel order shouldn't be a
>>  factor in this discussion, it should
>>  be solved irrespective of registration so
>> addons
>> >> panels can
>>  be added in a logical order.
>> >>>
>> >>> Ideally, this should be done before going out
>> of
>> >> beta.
>> >>>
>>  Though I'm still for auto-registration
>> removal
>> >> even if its
>>  bug free,
>>  most likely re-iterating from previous
>> mails.
>> >>>
>> >>> It's not so much the reiteration of your same
>> >> arguments tha

Re: [Bf-committers] Removing auto registration

[Bassam Kurdali]

a small comment if the core developers/ designers of the api can't
decide which way is 'right':
Don't change what's there. If there is controversy, it probably means
there are pros and cons both ways; any change comes with a cost- every
script has to be modified to work. In he case of stuff that ships with
blender, the py is compatible with the api, but it is a problem for
external developers; because now you have an incompatiblity between
betas/svn.
Even a simple change such as the change from bl_addon_info to bl_info
was a bit painful here, since we had to figure a way to keep addons
working for animators using betas, and for those of us who build svn (so
that we can get bug fixes early). It turned out to be a bit non trivial.
When we went to autoregistration , it was a couple days work cleaning
scripts, and get rid of setting attributes in autogenerated classes
(because they were now getting autoregistered) so code would work. We
still run into the occasional code path that has a 'register()' left
there that causes a script to raise an exception. Going back to
registration would cause similar havoc.

I'm not saying, don't do this if it's the right thing, but I am saying
it's useful to continue the discussion , and get a real consensus if the
problems can be solved. Cutting the gordions knot seems
counterproductive *if* things can be made to work cleanly.

to sum up my long winded argument :
"won't someone please think of the children" ;)
On Sat, 2011-01-22 at 17:24 +0100, Ton Roosendaal wrote:
> Hi,
> 
> Yes that sounds clean and clear. But apparently there's more ways to  
> solve the problem, and either way has pros and cons. When visions on  
> such problems don't align, then how to decide? The maintainers and  
> main contributors to the code then can have a final word.
> 
> -Ton-


___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Removing auto registration

[Martin Poirier]

--- On Sat, 1/22/11, Brecht Van Lommel  wrote:

> I don't really see the order problem as a problem we have
> to solve. If
> you're registering multiple things, then it seems logical
> to me that
> if A references B, you register B first and then A. It's
> not clear to
> me where these potential complex rules would show up?

I don't think it's terribly complex either. The fixed type order I propose on 
the wiki page seems like it should work (quite easy with automatic register and 
simply enough to define for manual). Campbell supposedly has all kinds of wild 
cases under hands that fail automatic register, so that's why he was supposed 
to look into it and come back with definite cases of failure.

Martin


___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Removing auto registration

[Knapp]

On Sat, Jan 22, 2011 at 9:28 PM, Martin Poirier  wrote:
>
>
> --- On Sat, 1/22/11, Brecht Van Lommel  wrote:
>
>> I don't really see the order problem as a problem we have
>> to solve. If
>> you're registering multiple things, then it seems logical
>> to me that
>> if A references B, you register B first and then A. It's
>> not clear to
>> me where these potential complex rules would show up?
>
> I don't think it's terribly complex either. The fixed type order I propose on 
> the wiki page seems like it should work (quite easy with automatic register 
> and simply enough to define for manual). Campbell supposedly has all kinds of 
> wild cases under hands that fail automatic register, so that's why he was 
> supposed to look into it and come back with definite cases of failure.
>
> Martin

I don't know the ins and outs of this code but is really looks like
someone needs to write up some test cases and do some verification and
validation of the registration. If the code passes all the wild cases'
tests then we say it is good. This looks like a huge trap for the end
programmers and it should not be. If the system does not end up being
simple, you guys will end up spending the next few years answering the
same questions over and over again.

As was pointed out, this is the 'new' blender and it should have a
stable api by the time it releases, so now is the time to make
changes, if they need to be made. Forcing people to rewrite scrips now
is much better than making them do it next year when everyone thought
the new system was stable and they have forgotten how to work with the
new system because they thought they had all the rewrites done a long
time ago. Do what needs do be done now and get it over with.
Best,


-- 
Douglas E Knapp

Creative Commons Film Group, Helping people make open source movies
with open source software!
http://douglas.bespin.org/CommonsFilmGroup/phpBB3/index.php

Massage in Gelsenkirchen-Buer:
http://douglas.bespin.org/tcm/ztab1.htm
Please link to me and trade links with me!

Open Source Sci-Fi mmoRPG Game project.
http://sf-journey-creations.wikispot.org/Front_Page
http://code.google.com/p/perspectiveproject/
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

[Bf-committers] Removing auto registration

[Sergey Kurdakov]

Hi,

I'm sorry, as actually I would need to dig into details to understand all
the subtleties,
but few sparse thoughts

order:
is it possible to have a two step registration? such that first - there  map
is created ( or graph
http://www.boost.org/doc/libs/1_45_0/libs/graph/doc/index.html )  like it is
done in boost.python (starting files  to look at
are \boost\python\converter\registry.hpp
| boost\python\converter\registrations.hpp  )  of types ,
where all requested types are registered upon class creation and then
then the whole system is glued together ( registered ) looking for
unavailable parts and the next step?

will it solve
> Matt Ebb and Nathan Vegdahl have complained about auto-registration
>in its current state fir renderman support which does dynamic
>generated classes from shaders, and rigify for rig types.

if yes. then why two step registration is not an option ( or will it add too
much complexity)?

Regards
Sergey
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

Re: [Bf-committers] Proposal: Blender OpenCL compositor

[GSR]

Hi,
j.bak...@atmind.nl (2011-01-22 at 0952.44 +0100):
> image. The highest/lowest value is calculated once (not parallelized) 
> the pixel processor is parallelized.

Not very good example ;] as this searching problem is near as much
parallelizable as the pixel processor would be. Split the work into N
workers, every one gets total_pixels/N (or tiles or whatever), looking
for the local max and min. Then scan the N maxes to get the final max,
and the same with the N mins. Even if you have a system with 1024
"workers", that is only an extra non parallel pass of 1024 checks
(assuming you do not parallelize it again, having four workers doing
256 each and finally compare four results, for example).

So the questions if you want to process in pixel stacks (what is the
final result for X,Y pixel before X+1,Y is known) or buffers (work in
one set of tiles and never look at them except if something down the
node tree changes). If you want the final full image, you will do the
full work in both cases anyway. Exceptions aside, you probably want
buffers approach (with tiled internal organization, that is fine),
because that way the code cache gets lots and lots of hits, and data
one probably too. The other way you are trashing all caches.

GSR
 
___
Bf-committers mailing list
Bf-committers@blender.org
http://lists.blender.org/mailman/listinfo/bf-committers