Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[KOVÁCS István]

On Thu, 13 Aug 2020 at 08:34, Andreas Herold  wrote:
> I decided to summarize everything from now on in every mail.
> Additonally I have constructed an example, that shows my problem in a more 
> dramatic way.

I think it'd be great if you opened a github issue with the same
content and examples:
https://github.com/darktable-org/darktable/issues/

Kofa
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Andreas Herold]

Hi,

thank you for all the hints I got. Since there are always questions, I have 
already answered in previous mails, I decided to summarize everything from now 
on in every mail.
Additonally I have constructed an example, that shows my problem in a more 
dramatic way.

My workflow in 3.0.2

I'm using parametric masks very early in my workflow. This way the definition 
of the mask does not depend on global changes I am applying later. If I would 
do it in opposite order, I would constantly need to adjust the parametric mask, 
which is very time consuming and inconvinient. I'm very happy with my workflow, 
and I'm glad that I can do this with darktable.
The following screenshot shows the (constructed) example. The module "tone 
curve parametric mask" is below "shadow and highlights“: 
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-13+um+07.11.05.png
And here is the same situation with highlighted mask: 
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-13+um+07.11.28.png

Upgrade to 3.2.1

After upgrade to 3.2.1 the order of modules was very different from what I saw 
in 3.0.2. For me that means, that the parametric masks of hundreds of images 
are not the same as in 3.0.2. Finally that means, that the processing of some 
hundred images is destroyed now.
Here is the screenshot of the example image after upgrade to 3.2.1. Module 
"shadow and highlights" is now below "tone curve parametric mask", which 
destroys the parametric mask of this module:
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-13+um+07.16.33.png
The same with highlighted mask: 
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-13+um+07.16.52.png
After changing the module order to "legacy" it became worse. Now are two other 
modules before "tone curve parametric mask“: 
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-13+um+07.17.26.png
And again, the same with highlighted mask: 
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-13+um+07.17.47.png

These are the hints I got and I have already tested:

* changed setting of Preferences/processing/auto apply pixel workflow
* changed module order from custom to legacy

If you are curious, that is the image how it should look like: 
https://www.flickr.com/photos/14952@N07/50217006456/in/dateposted/

Best regards
Andreas


> Am 13.08.2020 um 00:26 schrieb Pascal Obry :
> 
> Le jeudi 13 août 2020 à 00:21 +0200, Andreas Herold a écrit :
>> Screenshot 3.0.2: 
>> https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-12+um+23.02.54+Kopie.jpg
>> Screenshot 3.2.1 custom: 
>> https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-12+um+23.24.53+Kopie.jpg
> 
> Those two screenshots are looking very similar. What is the issue?
> 
> -- 
>  Pascal Obry /  Magny Les Hameaux (78)
> 
>  The best way to travel is by means of imagination
> 
>  http://www.obry.net
> 
>  gpg --keyserver keys.gnupg.net --recv-key F949BD3B
> 

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Philippe Weyland]

Hi Andreas,
If I can give my cent...
1. keep your 3.0.2 database backup safe
2. do not use „load sidecar file …“ which is buggy
3. ... try the different configuration options after replaying the 
migration ... but here I cannot help much. Using "legacy module order" 
you should be able to keep your module order safe. You may try to set 
"legacy order" on a 3.0.2 database, at least on some images as a test, 
before migration to 3.2.1... (and disable write xmp file to not to lose 
them)


Hoping that will help.
Philippe

On 12/08/2020 16:42, Andreas Herold wrote:

Hi Richard,

thank you. I tried It, but applied afterwards the order has not changed back.

I have enabled writing of xmp files and I’m seeing that the timestamps of xmp 
files of images I had not opened yet, are still before the upgrade to 3.2.1.
Most xmp files should still contain the right information. Do I have a chance 
to load this information from xmp files? The „load sidecar file …“ button is 
not an option. It’s too much work to do it for so many pictures.

I also have a backup of the old library and could switch back to version 3.0.2. 
I would then disable auto apply of pixel workflow before doing the upgrade.

Do you have any idea, what’s the preferred/better way?

Andreas


___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Pascal Obry]

Le jeudi 13 août 2020 à 00:21 +0200, Andreas Herold a écrit :
> Screenshot 3.0.2: 
> https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-12+um+23.02.54+Kopie.jpg
> Screenshot 3.2.1 custom: 
> https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-12+um+23.24.53+Kopie.jpg

Those two screenshots are looking very similar. What is the issue?
 
-- 
  Pascal Obry /  Magny Les Hameaux (78)

  The best way to travel is by means of imagination

  http://www.obry.net

  gpg --keyserver keys.gnupg.net --recv-key F949BD3B

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Pascal Obry]

Le mercredi 12 août 2020 à 22:03 +0200, Andreas Herold a écrit :
> Thank you. I didn't notice that before. Currently it's showing
> "custom" (which is the wrong order) and I can change it to "legacy",
> which results in another wrong order.

"custom" is not the wrong order (expect if there is a bug) but the
order that is neither legacy or 3.0 because some modules where moved
around in the pixelpipe. Wasn't this the case?

-- 
  Pascal Obry /  Magny Les Hameaux (78)

  The best way to travel is by means of imagination

  http://www.obry.net

  gpg --keyserver keys.gnupg.net --recv-key F949BD3B

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Pascal Obry]

Le mercredi 12 août 2020 à 21:32 +0200, KOVÁCS István a écrit :
> I thought that setting changes which modules get auto-applied (base
> curve or filmic+exposure or none). There's a 'module order' setting
> in
> the darkroom (below the modules, above 'more modules') that allows
> you
> to switch between 3.0 and legacy order -- however, in that respect,
> 3.2.1 did not bring any changes, did it (that is, the order of
> modules
> between 3.0 and 3.2 is the same, is it not)?

Yes, 3.0 and 3.2 are using the same module order.

-- 
  Pascal Obry /  Magny Les Hameaux (78)

  The best way to travel is by means of imagination

  http://www.obry.net

  gpg --keyserver keys.gnupg.net --recv-key F949BD3B

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Andreas Herold]

Hi,

here is what I did in the meantime:

* 3.2.1.: Changed Preferences/processing/auto apply pixel workflow to none 
(without success)
* switched back to 3.0.2. Did not found any option to configure processing 
order of modules
* 3.0.2: created screenshot
* 3.2.1: upgraded DB
* 3.2.1: checked Preferences/processing/auto apply pixel workflow. Was still 
set to none (I guess it's stored in darktablerc, which I did not restored from 
backup)
* 3.2.1: created screenshot (current order: custom)
* 3.2.1: changed order to legacy
* 3.2.1: created screenshot (current order: legacy)

Now I have no further idea, how to go further.

Andreas

Screenshot 3.0.2: 
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-12+um+23.02.54+Kopie.jpg
 

Screenshot 3.2.1 custom: 
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-12+um+23.24.53+Kopie.jpg
 

Screenshot 3.2.1 legacy: 
https://www.dropbox.com/sh/orbifgwd23sgzw8/AAC3SjlwjRe-LvLhX7U5UZBaa?dl=0=Bildschirmfoto+2020-08-12+um+23.27.43+Kopie.jpg
 



> Am 12.08.2020 um 22:11 schrieb Philippe Weyland 
> :
> 
> 
> Hi Andreas,
> If I can give my cent...
> 1. keep your 3.0.2 database backup safe
> 2. do not use „load sidecar file …“ which is buggy
> 3. ... try the different configuration options after replaying the migration 
> ... but here I cannot help much. Using "legacy module order" you should be 
> able to keep your module order safe. You may try to set "legacy order" on a 
> 3.0.2 database, at least on some images as a test, before migration to 
> 3.2.1... (and disable write xmp file to not to lose them)
> 
> Hoping that will help.
> Philippe
> 
> On 12/08/2020 16:42, Andreas Herold wrote:
>> Hi Richard,
>> thank you. I tried It, but applied afterwards the order has not changed back.
>> I have enabled writing of xmp files and I’m seeing that the timestamps of 
>> xmp files of images I had not opened yet, are still before the upgrade to 
>> 3.2.1.
>> Most xmp files should still contain the right information. Do I have a 
>> chance to load this information from xmp files? The „load sidecar file …“ 
>> button is not an option. It’s too much work to do it for so many pictures.
>> I also have a backup of the old library and could switch back to version 
>> 3.0.2. I would then disable auto apply of pixel workflow before doing the 
>> upgrade.
>> Do you have any idea, what’s the preferred/better way?
>> Andreas
> ___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
> 


___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Andreas Herold]

> Am 12.08.2020 um 21:09 schrieb Mica Semrick :
> 
> Hi, 
> 
> You can still use the old, ordered processing pipeline with base curve and 
> all that. You just need to select it from the modules menu.

Thank you. I didn't notice that before. Currently it's showing "custom" (which 
is the wrong order) and I can change it to "legacy", which results in another 
wrong order.

Andreas

> 
> Your workflow sounds very nonstandard though, I'd think most start with 
> global adjustments and work toward local adjustments at the end of the edit.
> 
> - m
> 
> On August 12, 2020 11:15:32 AM PDT, Andreas Herold  
> wrote:
> Hi,
> 
> some feedback from a User who made currently the upgrade from 3.0.2 to 3.2.1.
> 
> In my workflow I’m using very often modules, which are using parametric 
> masks. I’m applying moduIes with parametric masks very early in processing. I 
> do this, to have stable masks, that are not depend on other modules. E.g. if 
> I want to change brightness and contrast of the sky, I’m doing this before 
> changing overall brightness. This way the mask of the sky does not depend on 
> overall brightness.
> After upgrade from 3.0.2 to 3.2.1 I now see, that the order of modules has 
> changed. For example, now the overall brightness is applied before the 
> module, that was changing the sky. Since I had used a parametric mask, this 
> mask became invalid. It is not the same selection I originally chosed. In the 
> result, the processing is destroyed for hundreds of images.
> 
> Best regards
> Andreas 
> Herold___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
> 
> 
> ___ 
> darktable developer mailing list to unsubscribe send a mail to 
> darktable-dev+unsubscr...@lists.darktable.org 


___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Andreas Herold]

> Am 12.08.2020 um 21:32 schrieb KOVÁCS István :
> 
> On Wed, 12 Aug 2020 at 21:09, Richard Hobday  wrote:
>> You might benefit from changing the settings in:
>> Preferences/processing/auto apply pixel workflow defaults
>> this should allow you to the revert to the legacy pipe order.
> 
> I thought that setting changes which modules get auto-applied (base
> curve or filmic+exposure or none). There's a 'module order' setting in
> the darkroom (below the modules, above 'more modules') that allows you
> to switch between 3.0 and legacy order -- however, in that respect,
> 3.2.1 did not bring any changes, did it (that is, the order of modules
> between 3.0 and 3.2 is the same, is it not)?
> 
> However, wouldn't raster masks solve the original poster's problem?
> E.g. one could create a 'dummy' module instance early in the pipe
> (like exposure with no actual change of the exposure), create a mask,
> and re-use that mask as a raster mask for any number of modules that
> come later in the pipe. Any modifications (e.g. changes to exposure)
> done after the 'dummy' exposure module would leave the raster mask
> unchanged.
> 
> https://www.darktable.org/2019/12/darktable-30/
> 'Raster masks add a new mode that allows you to reuse a mask from one
> module to another module. This is particularly interesting with
> parametric masks: copying the parameters from one module to another
> would not have been enough because the same parameters do not select
> the same pixels depending on where you are in the pipe. The raster
> mask reuses the mask directly, so it does not have this problem.
> Warning: The raster mask can only take a mask from a module preceding
> it in the processing chain. [...]'

Thank you. This sounds very promising and I will try it. But nevertheless I 
need to find a way to rescue my already processed images.

Andreas

> 
> Kofa
> ___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
> 

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Andreas Herold]

Hi Richard,

thank you. I tried It, but applied afterwards the order has not changed back.

I have enabled writing of xmp files and I’m seeing that the timestamps of xmp 
files of images I had not opened yet, are still before the upgrade to 3.2.1.
Most xmp files should still contain the right information. Do I have a chance 
to load this information from xmp files? The „load sidecar file …“ button is 
not an option. It’s too much work to do it for so many pictures.

I also have a backup of the old library and could switch back to version 3.0.2. 
I would then disable auto apply of pixel workflow before doing the upgrade.

Do you have any idea, what’s the preferred/better way?

Andreas




> Am 12.08.2020 um 21:03 schrieb Richard Hobday :
> 
> You might benefit from changing the settings in:
> Preferences/processing/auto apply pixel workflow defaults
> this should allow you to the revert to the legacy pipe order.
> 
> R.
> 
> On 12/08/2020 19:15, Andreas Herold wrote:
>> Hi,
>> 
>> some feedback from a User who made currently the upgrade from 3.0.2 to 3.2.1.
>> 
>> In my workflow I’m using very often modules, which are using parametric 
>> masks. I’m applying moduIes with parametric masks very early in processing. 
>> I do this, to have stable masks, that are not depend on other modules. E.g. 
>> if I want to change brightness and contrast of the sky, I’m doing this 
>> before changing overall brightness. This way the mask of the sky does not 
>> depend on overall brightness.
>> After upgrade from 3.0.2 to 3.2.1 I now see, that the order of modules has 
>> changed. For example, now the overall brightness is applied before the 
>> module, that was changing the sky. Since I had used a parametric mask, this 
>> mask became invalid. It is not the same selection I originally chosed. In 
>> the result, the processing is destroyed for hundreds of images.
>> 
>> Best regards
>> Andreas 
>> Herold___
>> darktable developer mailing list
>> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
>> 
> 
> --
> 
> ___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
> 

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[KOVÁCS István]

On Wed, 12 Aug 2020 at 21:09, Richard Hobday  wrote:
> You might benefit from changing the settings in:
> Preferences/processing/auto apply pixel workflow defaults
> this should allow you to the revert to the legacy pipe order.

I thought that setting changes which modules get auto-applied (base
curve or filmic+exposure or none). There's a 'module order' setting in
the darkroom (below the modules, above 'more modules') that allows you
to switch between 3.0 and legacy order -- however, in that respect,
3.2.1 did not bring any changes, did it (that is, the order of modules
between 3.0 and 3.2 is the same, is it not)?

However, wouldn't raster masks solve the original poster's problem?
E.g. one could create a 'dummy' module instance early in the pipe
(like exposure with no actual change of the exposure), create a mask,
and re-use that mask as a raster mask for any number of modules that
come later in the pipe. Any modifications (e.g. changes to exposure)
done after the 'dummy' exposure module would leave the raster mask
unchanged.

https://www.darktable.org/2019/12/darktable-30/
'Raster masks add a new mode that allows you to reuse a mask from one
module to another module. This is particularly interesting with
parametric masks: copying the parameters from one module to another
would not have been enough because the same parameters do not select
the same pixels depending on where you are in the pipe. The raster
mask reuses the mask directly, so it does not have this problem.
Warning: The raster mask can only take a mask from a module preceding
it in the processing chain. [...]'

Kofa
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Mica Semrick]

Hi, 

You can still use the old, ordered processing pipeline with base curve and all 
that. You just need to select it from the modules menu.

Your workflow sounds very nonstandard though, I'd think most start with global 
adjustments and work toward local adjustments at the end of the edit.

- m

On August 12, 2020 11:15:32 AM PDT, Andreas Herold  
wrote:
>Hi,
>
>some feedback from a User who made currently the upgrade from 3.0.2 to
>3.2.1.
>
>In my workflow I’m using very often modules, which are using parametric
>masks. I’m applying moduIes with parametric masks very early in
>processing. I do this, to have stable masks, that are not depend on
>other modules. E.g. if I want to change brightness and contrast of the
>sky, I’m doing this before changing overall brightness. This way the
>mask of the sky does not depend on overall brightness.
>After upgrade from 3.0.2 to 3.2.1 I now see, that the order of modules
>has changed. For example, now the overall brightness is applied before
>the module, that was changing the sky. Since I had used a parametric
>mask, this mask became invalid. It is not the same selection I
>originally chosed. In the result, the processing is destroyed for
>hundreds of images.
>
>Best regards
>Andreas
>Herold___
>darktable developer mailing list
>to unsubscribe send a mail to
>darktable-dev+unsubscr...@lists.darktable.org

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Upgrade from 3.0.2 to 3.2.1 destroys order of modules

[Richard Hobday]

You might benefit from changing the settings in:
Preferences/processing/auto apply pixel workflow defaults
this should allow you to the revert to the legacy pipe order.

R.

On 12/08/2020 19:15, Andreas Herold wrote:

Hi,

some feedback from a User who made currently the upgrade from 3.0.2 to 3.2.1.

In my workflow I’m using very often modules, which are using parametric masks. 
I’m applying moduIes with parametric masks very early in processing. I do this, 
to have stable masks, that are not depend on other modules. E.g. if I want to 
change brightness and contrast of the sky, I’m doing this before changing 
overall brightness. This way the mask of the sky does not depend on overall 
brightness.
After upgrade from 3.0.2 to 3.2.1 I now see, that the order of modules has 
changed. For example, now the overall brightness is applied before the module, 
that was changing the sky. Since I had used a parametric mask, this mask became 
invalid. It is not the same selection I originally chosed. In the result, the 
processing is destroyed for hundreds of images.

Best regards
Andreas 
Herold___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org



--

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org