Re: [darktable-dev] implementation question: remove all unused modules

[Aurélien Pierre]

There is no image if the white balance has been disabled, just random
bits for debugging purposes. The input color profile expects
D50-balanced input, and this one is always on. Same with the highlight
clipping module, most software have it built in the input profile and
don't expose it. Disable these 2 and you mess up your whole module stack.

So, again, what is the purpose of disabling these 2 modules, except for
debugging, and why can they be disabled at all but are still
auto-applied at opening ?

The stack of exceptions handling in the SQL instructions smells bad.
Things should stay general.

Le 13/06/2019 à 08:36, para...@paraf.in a écrit :
>
>
> On 12 Jun 2019, at 20:21, Aurélien Pierre  > wrote:
>
>> Hi,
>>
>> I'm the author of the lighttable's compress history button. White
>> balance and highlight reconstruction should never be turned off, so
>> what would be the purpose of having them disabled in the first place ?
>>
>
> Yet they can be disabled, some other modules can’t.
>
>> Then, we can discuss what the history compression should
>> do/avoid/prevent.
>>
>
> History compression shouldn’t modify the image.
>
>> Aurélien.
>>
>> Le 12/06/2019 à 11:40, dt-l...@stefan-klinger.de a écrit :
>>> thokster (2019-Jun-11, excerpt):
 Hi,

 Am 11.06.19 um 15:32 schrieb dt-l...@stefan-klinger.de:

> This option is not about saving disk space, but rather about cleaning
> up.
 Is the "compress history" button in lighttable view doing anything else?
>>> Woha.  I have been talking about "compress history" in darkroom, not
>>> lighttable.
>>>
>>> They seem not to use the same code internally.
>>>
>>> I have to admit that I have not thought about the "compress history"
>>> button in lighttable.  That one already seems to remove switched-off
>>> modules in current master [1].  And it seems to do this incorrectly
>>> wrt. parafin's email:
>>>
>>> * In darkroom, disable "white balance" and/or "highlight
>>>   reconstruction".
>>>
>>> * Go to lighttable, select that image, and apply "history stack →
>>>   compress history"
>>>
>>> * When you open the image again, "white balance" and "highlight
>>>   reconstruction" will be enabled again.
>>>
>>> So if compressing should not change the image, then my current
>>> implementation [2] is even more correct, although it only is
>>> applicable from darkroom, not lighttable.
>>>
>>> Cheers
>>> Stefan
>>>
>>> 
>>> [1] 2.7.0+1443~g9bfbb225e
>>> [2] 
>>> https://github.com/darktable-org/darktable/compare/master...s5k6:compressHistory
>>>
>>>
>>> --
>>> http://stefan-klinger.deo/X
>>> I prefer receiving plain text messages, not exceeding 32kB. /\/
>>>   \
>>> ___
>>> darktable developer mailing list
>>> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
>>>
>>
>> ___
>> darktable developer mailing list to unsubscribe send a mail to
>> darktable-dev+unsubscr...@lists.darktable.org
>> 
>
> ___
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] implementation question: remove all unused modules

[parafin]

Just because you don’t agree with some design decisions made previously (I seem 
to remember that white balance was always on at some point), doesn’t mean you 
should introduce broken code. Correct fix is to add these 2 modules to the 
history, as I previously mentioned. Reverting them back to always-on I think is 
wrong, because technically they are not required (I think that was the reason 
to make them optional).
This bug should be fixed somehow.

> On 13 Jun 2019, at 09:39, Aurélien Pierre  wrote:
> 
> There is no image if the white balance has been disabled, just random bits 
> for debugging purposes. The input color profile expects D50-balanced input, 
> and this one is always on. Same with the highlight clipping module, most 
> software have it built in the input profile and don't expose it. Disable 
> these 2 and you mess up your whole module stack.
> 
> So, again, what is the purpose of disabling these 2 modules, except for 
> debugging, and why can they be disabled at all but are still auto-applied at 
> opening ? 
> 
> The stack of exceptions handling in the SQL instructions smells bad. Things 
> should stay general.
> 
>> Le 13/06/2019 à 08:36, para...@paraf.in a écrit :
>> 
>> 
>> On 12 Jun 2019, at 20:21, Aurélien Pierre  wrote:
>> 
>>> Hi,
>>> 
>>> I'm the author of the lighttable's compress history button. White balance 
>>> and highlight reconstruction should never be turned off, so what would be 
>>> the purpose of having them disabled in the first place ?
>>> 
>> 
>> Yet they can be disabled, some other modules can’t.
>> 
>>> Then, we can discuss what the history compression should do/avoid/prevent. 
>>> 
>> 
>> History compression shouldn’t modify the image.
>> 
>>> Aurélien.
>>> 
 Le 12/06/2019 à 11:40, dt-l...@stefan-klinger.de a écrit :
 thokster (2019-Jun-11, excerpt):
> Hi,
> 
> Am 11.06.19 um 15:32 schrieb dt-l...@stefan-klinger.de:
> 
>> This option is not about saving disk space, but rather about cleaning
>> up.
> Is the "compress history" button in lighttable view doing anything else?
 Woha.  I have been talking about "compress history" in darkroom, not
 lighttable.
 
 They seem not to use the same code internally.
 
 I have to admit that I have not thought about the "compress history"
 button in lighttable.  That one already seems to remove switched-off
 modules in current master [1].  And it seems to do this incorrectly
 wrt. parafin's email:
 
 * In darkroom, disable "white balance" and/or "highlight
   reconstruction".
 
 * Go to lighttable, select that image, and apply "history stack →
   compress history"
 
 * When you open the image again, "white balance" and "highlight
   reconstruction" will be enabled again.
 
 So if compressing should not change the image, then my current
 implementation [2] is even more correct, although it only is
 applicable from darkroom, not lighttable.
 
 Cheers
 Stefan
 
 
 [1] 2.7.0+1443~g9bfbb225e
 [2] 
 https://github.com/darktable-org/darktable/compare/master...s5k6:compressHistory
 
 
 --
 http://stefan-klinger.deo/X
 I prefer receiving plain text messages, not exceeding 32kB. /\/
   \
 ___
 darktable developer mailing list
 to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
 
>>> 
>>> ___ 
>>> darktable developer mailing list to unsubscribe send a mail to 
>>> darktable-dev+unsubscr...@lists.darktable.org 
>> 
>> ___ 
>> darktable developer mailing list to unsubscribe send a mail to 
>> darktable-dev+unsubscr...@lists.darktable.org 
> 
> ___ 
> darktable developer mailing list to unsubscribe send a mail to 
> darktable-dev+unsubscr...@lists.darktable.org 

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] implementation question: remove all unused modules

[dt-list]

Aurélien Pierre (2019-Jun-13, excerpt):
> There is no image if the white balance has been disabled, just random
> bits for debugging purposes. The input color profile expects
> D50-balanced input, and this one is always on. Same with the highlight
> clipping module, most software have it built in the input profile and
> don't expose it. Disable these 2 and you mess up your whole module stack.
> 
> So, again, what is the purpose of disabling these 2 modules, except for
> debugging, and why can they be disabled at all but are still
> auto-applied at opening ?

It might well be a crap idea to do this, but it's possible, so someone
will do it.

I see two possible outcomes to the current situation:

  1. Some modules must not be disabled.

Then the possibility to disable them is a bug that must be
resolved.

This will "break" all existing XMPs that disable one of these
modules.

  2. All modules can be disabled.

Then they all must be reported to the history stack, and
compression must not modify their topmost recorded state.

This will "break" all existing XMPs that lack a mentioning of
these modules in their stack.

Either way, it's a messy situation that needs to be resolved.

> The stack of exceptions handling in the SQL instructions smells bad.

I agree.  My first implementation was just like that.

> Things should stay general.

General would mean: No exception for "white balance" and others.
Later modules should simply refuse to work if no appropriate input is
available (message: “need white-balanced image”).  To implement this,
something like the 'type' of an image could be propagated upwards
along the stack of IOPs (is this really not happening right now?).
Just as an oversimplified example, "white balance" might require 'RAW'
and provide 'D50-balanced', while "color profile" would require
'D50-balanced'.  This would allow to implement other modules providing
'D50-balanced' without technical debt hindering their adoption later
on.

Regarding the length of this thread, I assume that it will take quite
some time to reach consensus here.

Clearly the current situation (having two different implementations of
"compress history", one more complete but changing images in
pathological cases) is unsatisfactory.

So where do you want to go?

Why I started this: I wanted to learn DTs internal data structures, in
order to implement better snapshots and history.  The idea is to
replace the history stack with a rooted DAG, with the RAW at the root,
and all leaves snapshots (but not necessarily all snapshots at
leaves).  Current histories would be straight lists.  Creating a
snapshot, going back in history, and then doing a different edit,
would create a tree.  This would pave the way to (in the future)
create IOPs that use more than one input, including a means to combine
multiple RAWs.  This approach would clearly benefit from a 'typing'
system outlined above.


-- 
http://stefan-klinger.deo/X
I prefer receiving plain text messages, not exceeding 32kB. /\/
  \
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

[darktable-dev] X-Trans Interpolation code

[David C. Partridge]

Hi folks

I've read that the implementation of Frank Markesteijn's X-Trans
interpolation code in darktable is much faster than that in LibRaw or dcraw.

Is that correct?

For better or worse I'm using LibRaw to extract regular raw Bayer image data
and processing that myself, but for Fuji X-Trans and Foveon images I'm
letting LibRaw::process() convert the image to a 3 colour RGB image and
catching the output.   Sad to say that the interpolation time for 3 passes
of xtrans_interpolate() in LibRaw is might slow (40 seconds or so on my i7
lappie).

So, if it is practical and you folks are OK with it,  I'd like to
"borrow/reuse" that code in DeepSkyStacker.

Please could someone who knows point me to the relevant source code file on
Github so I can take a look to see if I could adapt it in a subclass of
LibRaw?

If someone could indicate how this speed up was achieved I'd be most
interested to know and also whether it might be OpenMP compatible?

PS does Frank Markesteijn contribute here?

Many thanks
David

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] implementation question: remove all unused modules

[Julian Rickards]

Not to throw gas on the fire but it seems to me that some of this is
already done.

In Lighttable, when you wish to copy the history from one image to another,
there are some modules missing (input and output) and also, the "off"
modules aren't included in the possible modules to copy over.

On Thu, Jun 13, 2019 at 5:30 AM  wrote:

> Aurélien Pierre (2019-Jun-13, excerpt):
> > There is no image if the white balance has been disabled, just random
> > bits for debugging purposes. The input color profile expects
> > D50-balanced input, and this one is always on. Same with the highlight
> > clipping module, most software have it built in the input profile and
> > don't expose it. Disable these 2 and you mess up your whole module stack.
> >
> > So, again, what is the purpose of disabling these 2 modules, except for
> > debugging, and why can they be disabled at all but are still
> > auto-applied at opening ?
>
> It might well be a crap idea to do this, but it's possible, so someone
> will do it.
>
> I see two possible outcomes to the current situation:
>
>   1. Some modules must not be disabled.
>
> Then the possibility to disable them is a bug that must be
> resolved.
>
> This will "break" all existing XMPs that disable one of these
> modules.
>
>   2. All modules can be disabled.
>
> Then they all must be reported to the history stack, and
> compression must not modify their topmost recorded state.
>
> This will "break" all existing XMPs that lack a mentioning of
> these modules in their stack.
>
> Either way, it's a messy situation that needs to be resolved.
>
> > The stack of exceptions handling in the SQL instructions smells bad.
>
> I agree.  My first implementation was just like that.
>
> > Things should stay general.
>
> General would mean: No exception for "white balance" and others.
> Later modules should simply refuse to work if no appropriate input is
> available (message: “need white-balanced image”).  To implement this,
> something like the 'type' of an image could be propagated upwards
> along the stack of IOPs (is this really not happening right now?).
> Just as an oversimplified example, "white balance" might require 'RAW'
> and provide 'D50-balanced', while "color profile" would require
> 'D50-balanced'.  This would allow to implement other modules providing
> 'D50-balanced' without technical debt hindering their adoption later
> on.
>
> Regarding the length of this thread, I assume that it will take quite
> some time to reach consensus here.
>
> Clearly the current situation (having two different implementations of
> "compress history", one more complete but changing images in
> pathological cases) is unsatisfactory.
>
> So where do you want to go?
>
> Why I started this: I wanted to learn DTs internal data structures, in
> order to implement better snapshots and history.  The idea is to
> replace the history stack with a rooted DAG, with the RAW at the root,
> and all leaves snapshots (but not necessarily all snapshots at
> leaves).  Current histories would be straight lists.  Creating a
> snapshot, going back in history, and then doing a different edit,
> would create a tree.  This would pave the way to (in the future)
> create IOPs that use more than one input, including a means to combine
> multiple RAWs.  This approach would clearly benefit from a 'typing'
> system outlined above.
>
>
> --
> http://stefan-klinger.deo/X
> I prefer receiving plain text messages, not exceeding 32kB. /\/
>   \
> ___
> darktable developer mailing list
> to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
>
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] implementation question: remove all unused modules

[Aurélien Pierre]

General means every module should behave the same and avoid the burden
of having to manage them locally and separately, and forget later that
some of them need special care because *there is no dev documentation to
write that on *(that's what happened with the lighttable button).

Now, some modules need to be always enabled because a raw photograph is
not an image without them. What remains to decide is the scope of these
modules : are we only speaking of basic "codecs" modules (demosaicing,
input & output colour profiles), or are we integrating less basic ones
although very much needed ?

The fact that highlights reconstruction and white balance have been
moved from the "always on" to the "off upon user request but still on by
default" bin with no further handing/checking is a mistake, and possibly
bad design until someone is able to tell me what the use case for having
them disabled is. Because, without them, your clipped highlights are
magenta, your input color profile is useless and your whole image is
possibly green or else, so why anyone would like to have that ? Infrared
photographers ?

These 2 modules need to be synchronized with the default behaviour
handled by the SQL queries, instead of having the SQL queries patch
every possible combination of "user on/off" + "default on/off" +
"good/bad assumptions if no data". Because the list of modules is ever
growing, that will soon be a nightmare (already is).

Otherwise, having an "enabled by default if not actively disabled by
user" IOP state means all disabled modules will need to stay in the
history no matter what, because it's a fallacy to think that the ones
maintaining modules will think of updating well-hidden
sub-functionnalities like that when they rework the pixelpipe.
Especially in the absence of a documentation.

Le 13/06/2019 à 14:56, Julian Rickards a écrit :
> Not to throw gas on the fire but it seems to me that some of this is
> already done.
>
> In Lighttable, when you wish to copy the history from one image to
> another, there are some modules missing (input and output) and also,
> the "off" modules aren't included in the possible modules to copy over.
>
> On Thu, Jun 13, 2019 at 5:30 AM  > wrote:
>
> Aurélien Pierre (2019-Jun-13, excerpt):
> > There is no image if the white balance has been disabled, just
> random
> > bits for debugging purposes. The input color profile expects
> > D50-balanced input, and this one is always on. Same with the
> highlight
> > clipping module, most software have it built in the input
> profile and
> > don't expose it. Disable these 2 and you mess up your whole
> module stack.
> >
> > So, again, what is the purpose of disabling these 2 modules,
> except for
> > debugging, and why can they be disabled at all but are still
> > auto-applied at opening ?
>
> It might well be a crap idea to do this, but it's possible, so someone
> will do it.
>
> I see two possible outcomes to the current situation:
>
>   1. Some modules must not be disabled.
>
>     Then the possibility to disable them is a bug that must be
>     resolved.
>
>     This will "break" all existing XMPs that disable one of these
>     modules.
>
>   2. All modules can be disabled.
>
>     Then they all must be reported to the history stack, and
>     compression must not modify their topmost recorded state.
>
>     This will "break" all existing XMPs that lack a mentioning of
>     these modules in their stack.
>
> Either way, it's a messy situation that needs to be resolved.
>
> > The stack of exceptions handling in the SQL instructions smells bad.
>
> I agree.  My first implementation was just like that.
>
> > Things should stay general.
>
> General would mean: No exception for "white balance" and others.
> Later modules should simply refuse to work if no appropriate input is
> available (message: “need white-balanced image”).  To implement this,
> something like the 'type' of an image could be propagated upwards
> along the stack of IOPs (is this really not happening right now?).
> Just as an oversimplified example, "white balance" might require 'RAW'
> and provide 'D50-balanced', while "color profile" would require
> 'D50-balanced'.  This would allow to implement other modules providing
> 'D50-balanced' without technical debt hindering their adoption later
> on.
>
> Regarding the length of this thread, I assume that it will take quite
> some time to reach consensus here.
>
> Clearly the current situation (having two different implementations of
> "compress history", one more complete but changing images in
> pathological cases) is unsatisfactory.
>
> So where do you want to go?
>
> Why I started this: I wanted to learn DTs internal data structures, in
> order to implement better snapshots and history.  The idea i

Re: [darktable-dev] implementation question: remove all unused modules

[Julian Rickards]

>
> Do people modify demosaic, input and output on a regular basis? If not,
> could the options for each be moved to "Settings" so they aren't listed as
> a module but modifiable when needed?
>
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] implementation question: remove all unused modules

[William Ferguson]

I modify demosiac on a fairly regular basis.  I use PPG as the default, but
on noisy files I use AMAZE.  I've created several denoise styles that
include demosiac changes.

On Thu, Jun 13, 2019 at 11:55 AM Julian Rickards 
wrote:

> Do people modify demosaic, input and output on a regular basis? If not,
>> could the options for each be moved to "Settings" so they aren't listed as
>> a module but modifiable when needed?
>>
>>
> ___
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] implementation question: remove all unused modules

[Julian Rickards]

OK, thanks, that answers that question.


On Thu, Jun 13, 2019 at 12:00 PM William Ferguson 
wrote:

> I modify demosiac on a fairly regular basis.  I use PPG as the default,
> but on noisy files I use AMAZE.  I've created several denoise styles that
> include demosiac changes.
>
> On Thu, Jun 13, 2019 at 11:55 AM Julian Rickards <
> julian.ricka...@gmail.com> wrote:
>
>> Do people modify demosaic, input and output on a regular basis? If not,
>>> could the options for each be moved to "Settings" so they aren't listed as
>>> a module but modifiable when needed?
>>>
>>>
>> ___
>> darktable developer mailing list to unsubscribe send a mail to
>> darktable-dev+unsubscr...@lists.darktable.org
>>
>
> ___
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] implementation question: remove all unused modules

[KOVÁCS István]

On Thu, 13 Jun 2019 at 16:12, Aurélien Pierre
 wrote:
> [of highlights reconstruction and white balance] until someone is able to 
> tell me what the use case for having them disabled is

One use case is the workaround for the issue discussed here:
https://www.mail-archive.com/darktable-dev@lists.darktable.org/msg03734.html
It may have been fixed, but I don't know about it. Until it's gone,
the workaround is useful, even though it's not the correct thing to do
from the theoretical point of view.

Kofa
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] implementation question: remove all unused modules

[jys]

On Thu, Jun 13, 2019, at 12:43, KOVÁCS István wrote:
> On Thu, 13 Jun 2019 at 16:12, Aurélien Pierre
>  wrote:
> > [of highlights reconstruction and white balance] until someone is able to 
> > tell me what the use case for having them disabled is
> 
> One use case is the workaround for the issue discussed here:
> https://www.mail-archive.com/darktable-dev@lists.darktable.org/msg03734.html
> It may have been fixed, but I don't know about it.

https://github.com/darktable-org/darktable/pull/2127

It's an interesting example case, though.

-- 
jys
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org