[darktable-dev] Re: clang vs. gcc: dramatic performance difference

[Matthias Bodenbinder]

Am 13.09.18 um 07:38 schrieb Matthias Bodenbinder:
> I assume I am missing some significant compiler options for clang. But dont 
> know what that is.
> Can somebody help me with that and tell me what I need to do to make the 
> clang binary faster?

I found these messages with clang enabled:

-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES) 
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES) 
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND) 
CMake Warning at src/external/rawspeed/cmake/OpenMP.cmake:18 (message):
  Looking for OpenMP - failed.  utilities will not use openmp-based
  parallelization
Call Stack (most recent call first):
  src/external/rawspeed/CMakeLists.txt:186 (include)


What is missing? I installed openmp.


PS
I am using Manjaro


___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

[darktable-dev] clang vs. gcc: dramatic performance difference

[Matthias Bodenbinder]

Hi,

now that google announced that they are using clang for compiling chrome I 
wanted to give it a try with DT. I was mainly interested to see what the 
performance impact is for the DT binary when using clang. 

I simply edited build.sh like this to use clang:

export CC=/usr/bin/clang
export CXX=/usr/bin/clang++
INSTALL_PREFIX_DEFAULT="/opt/darktable-clang"

Result: Without opencl the clang binary of DT is three(!) times slower than the 
gcc binary. And with opencl enabled it is two times slower. 

19# ./bench-script-clang-vs-gcc.sh
3 runs no opencl
run clang 1: 48,421557 [dev_process_export] pixel pipeline processing took 
48,120 secs (47,050 CPU)
run gcc   1: 16,068927 [dev_process_export] pixel pipeline processing took 
15,733 secs (121,951 CPU)
run clang 2: 48,346575 [dev_process_export] pixel pipeline processing took 
48,054 secs (47,014 CPU)
run gcc   2: 16,240577 [dev_process_export] pixel pipeline processing took 
15,914 secs (123,015 CPU)
run clang 3: 48,445406 [dev_process_export] pixel pipeline processing took 
48,151 secs (47,046 CPU)
run gcc   3: 16,402528 [dev_process_export] pixel pipeline processing took 
16,073 secs (122,371 CPU)

3 runs with opencl
run clang 1: 17,418728 [dev_process_export] pixel pipeline processing took 
14,533 secs (12,262 CPU)
run gcc   1: 9,936274 [dev_process_export] pixel pipeline processing took 7,124 
secs (17,697 CPU)
run clang 2: 14,816236 [dev_process_export] pixel pipeline processing took 
14,448 secs (12,028 CPU)
run gcc   2: 7,755369 [dev_process_export] pixel pipeline processing took 7,342 
secs (17,904 CPU)
run clang 3: 14,897759 [dev_process_export] pixel pipeline processing took 
14,528 secs (12,253 CPU)
run gcc   3: 7,838780 [dev_process_export] pixel pipeline processing took 7,437 
secs (17,954 CPU)

I assume I am missing some significant compiler options for clang. But dont 
know what that is.
Can somebody help me with that and tell me what I need to do to make the clang 
binary faster?

Thanks
Matthias



___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Code reviews requested : Profile gamma correction

[Aurélien Pierre]

Le 12/09/2018 à 12:59, rawfiner a écrit :

> Hi Aurélien
>
> Fist, thank you for showing me this interesting video.
> I just compiled your branch.
>
> My first question is, is it possible to find shift power slope values
> that reproduce the result we had before with linear and gamma?
> If yes, I think you should compute the new parameters values from the
> old ones.
> You can take a look at function "legacy_param" in denoiseprofile.c to
> see an example.
> If that is not possible, we could imagine to have a "mode" selector in
> the GUI to switch between "linear and gamma" and "shift power slope".
It is not and I will re-integrate the legacy code as soon as I fix some
problems. I will definitely add the legacy_param().
>
> Considering opencl, I cannot help you here as I have never coded in
> opencl and I do not have a GPU.
> Yet, even without opencl, code seems already quite fast.
Actually, I even wonder if OpenCL is relevant for this as it's a linear
operation performed on only one pixel at the time over the flatten
array. I wouldn't be surprised if the OpenCL version were slower on some
systems than a good SSE2 version.
>
> Considering the code itself, my only remarks are for this line:
>   for(size_t k = 1; k < (size_t)ch * roi_out->width *
> roi_out->height; k++)
> First, is there a reason why you are using a size_t type? int or
> unsigned would be fine I think, and you wouldn't need a cast.
I took that line from the exposure.c IOP. I think size_t is an explicit
array index type so the compiler can do cleverer things with it than
just a regular int.
> Second, in C, array indexes start at 0, so the red value of the pixel
> at the top left corner is not processed by your loop (you can see it
> on exported image)
> So I guess you want the for loop to be:
>  for(unsigned k = 0; k < ch * roi_out->width * roi_out->height; k++)
That's a copy/paste error from another loop I removed since then.
>
> I know that C is hard to learn, so congratulations Aurélien! :-)
Thanks ! I have posted the first results :
https://discuss.pixls.us/t/solving-dynamic-range-problems-in-a-linear-way/9006

>
> rawfiner
>
>
> Le mer. 12 sept. 2018 à 14:46, Aurélien Pierre
> mailto:rese...@aurelienpierre.com>> a écrit :
>
> Hi everyone,
>
> when working with color profiles, the main historic issue was the
> non-linearity of the sensors/films. Now, it is rather that the
> color profile is performed on a chart having 6-7 EV of dynamic
> range while modern cameras have 12-15 EV. Simple gamma corrections
> (invented for CRT screens) don't work anymore, and video editors
> have invented a new standard able to remap the dynamic range and
> to fix the mid-tones at once :
> 
> https://www.youtube.com/watch?v=kVKnhJN-BrQ&index=7&list=PLa1F2ddGya_9XER0wnFS6Mgnp3T-hgSZO
>
> I have embedded the formula used in Blender
> (https://en.wikipedia.org/wiki/ASC_CDL) into the profile
> correction module of dt (using the same parameters for each RGB
> channel). The result looks more natural than the current version,
> without gamut or saturation issues in the highlights. It also
> speeds-up the worflow, since all is needed is this module to
> adjust the dynamic range, then a tone curve in auto RGB mode
> shaped as a stiff S to bring back the contrast. The result is much
> better than with the tonemapping modules, with less color fixes.
>
> I'm a newbie at C and it's the first time I achieve something
> inside dt, so I could use some reviews on my code and also some
> help on the OpenCL part (the kernel does not load, I don't know
> why) : https://github.com/aurelienpierre/darktable/tree/color-grading
>
> Thanks a lot !
>
> Aurélien.
>
>
> 
> ___
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
> 
>
>
> ___
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org


___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Code reviews requested : Profile gamma correction

[rawfiner]

Sorry for the spelling mistake, didn't mean "Fist" but "First"
My apologies
cheers,
rawfiner

Le mer. 12 sept. 2018 à 18:59, rawfiner  a écrit :

> Hi Aurélien
>
> Fist, thank you for showing me this interesting video.
> I just compiled your branch.
>
> My first question is, is it possible to find shift power slope values that
> reproduce the result we had before with linear and gamma?
> If yes, I think you should compute the new parameters values from the old
> ones.
> You can take a look at function "legacy_param" in denoiseprofile.c to see
> an example.
> If that is not possible, we could imagine to have a "mode" selector in the
> GUI to switch between "linear and gamma" and "shift power slope".
>
> Considering opencl, I cannot help you here as I have never coded in opencl
> and I do not have a GPU.
> Yet, even without opencl, code seems already quite fast.
>
> Considering the code itself, my only remarks are for this line:
>   for(size_t k = 1; k < (size_t)ch * roi_out->width * roi_out->height;
> k++)
> First, is there a reason why you are using a size_t type? int or unsigned
> would be fine I think, and you wouldn't need a cast.
> Second, in C, array indexes start at 0, so the red value of the pixel at
> the top left corner is not processed by your loop (you can see it on
> exported image)
>
> Sso I guess you want the for loop to be:
>  for(unsigned k = 0; k < ch * roi_out->width * roi_out->height; k++)
>
> I know that C is hard to learn, so congratulations Aurélien! :-)
>
> rawfiner
>
>
> Le mer. 12 sept. 2018 à 14:46, Aurélien Pierre 
> a écrit :
>
>> Hi everyone,
>>
>> when working with color profiles, the main historic issue was the
>> non-linearity of the sensors/films. Now, it is rather that the color
>> profile is performed on a chart having 6-7 EV of dynamic range while modern
>> cameras have 12-15 EV. Simple gamma corrections (invented for CRT screens)
>> don't work anymore, and video editors have invented a new standard able to
>> remap the dynamic range and to fix the mid-tones at once :
>> https://www.youtube.com/watch?v=kVKnhJN-BrQ&index=7&list=PLa1F2ddGya_9XER0wnFS6Mgnp3T-hgSZO
>>
>> I have embedded the formula used in Blender (
>> https://en.wikipedia.org/wiki/ASC_CDL) into the profile correction
>> module of dt (using the same parameters for each RGB channel). The result
>> looks more natural than the current version, without gamut or saturation
>> issues in the highlights. It also speeds-up the worflow, since all is
>> needed is this module to adjust the dynamic range, then a tone curve in
>> auto RGB mode shaped as a stiff S to bring back the contrast. The result is
>> much better than with the tonemapping modules, with less color fixes.
>>
>> I'm a newbie at C and it's the first time I achieve something inside dt,
>> so I could use some reviews on my code and also some help on the OpenCL
>> part (the kernel does not load, I don't know why) :
>> https://github.com/aurelienpierre/darktable/tree/color-grading
>>
>> Thanks a lot !
>>
>> Aurélien.
>>
>> ___
>> darktable developer mailing list to unsubscribe send a mail to
>> darktable-dev+unsubscr...@lists.darktable.org
>>
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Code reviews requested : Profile gamma correction

[rawfiner]

Hi Aurélien

Fist, thank you for showing me this interesting video.
I just compiled your branch.

My first question is, is it possible to find shift power slope values that
reproduce the result we had before with linear and gamma?
If yes, I think you should compute the new parameters values from the old
ones.
You can take a look at function "legacy_param" in denoiseprofile.c to see
an example.
If that is not possible, we could imagine to have a "mode" selector in the
GUI to switch between "linear and gamma" and "shift power slope".

Considering opencl, I cannot help you here as I have never coded in opencl
and I do not have a GPU.
Yet, even without opencl, code seems already quite fast.

Considering the code itself, my only remarks are for this line:
  for(size_t k = 1; k < (size_t)ch * roi_out->width * roi_out->height;
k++)
First, is there a reason why you are using a size_t type? int or unsigned
would be fine I think, and you wouldn't need a cast.
Second, in C, array indexes start at 0, so the red value of the pixel at
the top left corner is not processed by your loop (you can see it on
exported image)

Sso I guess you want the for loop to be:
 for(unsigned k = 0; k < ch * roi_out->width * roi_out->height; k++)

I know that C is hard to learn, so congratulations Aurélien! :-)

rawfiner


Le mer. 12 sept. 2018 à 14:46, Aurélien Pierre 
a écrit :

> Hi everyone,
>
> when working with color profiles, the main historic issue was the
> non-linearity of the sensors/films. Now, it is rather that the color
> profile is performed on a chart having 6-7 EV of dynamic range while modern
> cameras have 12-15 EV. Simple gamma corrections (invented for CRT screens)
> don't work anymore, and video editors have invented a new standard able to
> remap the dynamic range and to fix the mid-tones at once :
> https://www.youtube.com/watch?v=kVKnhJN-BrQ&index=7&list=PLa1F2ddGya_9XER0wnFS6Mgnp3T-hgSZO
>
> I have embedded the formula used in Blender (
> https://en.wikipedia.org/wiki/ASC_CDL) into the profile correction module
> of dt (using the same parameters for each RGB channel). The result looks
> more natural than the current version, without gamut or saturation issues
> in the highlights. It also speeds-up the worflow, since all is needed is
> this module to adjust the dynamic range, then a tone curve in auto RGB mode
> shaped as a stiff S to bring back the contrast. The result is much better
> than with the tonemapping modules, with less color fixes.
>
> I'm a newbie at C and it's the first time I achieve something inside dt,
> so I could use some reviews on my code and also some help on the OpenCL
> part (the kernel does not load, I don't know why) :
> https://github.com/aurelienpierre/darktable/tree/color-grading
>
> Thanks a lot !
>
> Aurélien.
>
> ___
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Add more tags on Redmine

[Pascal Obry]

Hi Aurélien,

> it makes sense, since pull requests and commits are on Github anyway.
> But as it has been bought by  Microsoft, I would rather have a
> duplicate elsewhere as safety measure.

Sure I understand, we can always move to GitLab if necessary. But this
is a separate issue and a trivial one since GitLab can be given a
GitHub project and migrate it automagically.

Best,

-- 
  Pascal Obry /  Magny Les Hameaux (78)

  The best way to travel is by means of imagination

  http://www.obry.net

  gpg --keyserver keys.gnupg.net --recv-key F949BD3B

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

[darktable-dev] Code reviews requested : Profile gamma correction

[Aurélien Pierre]

Hi everyone,

when working with color profiles, the main historic issue was the
non-linearity of the sensors/films. Now, it is rather that the color
profile is performed on a chart having 6-7 EV of dynamic range while
modern cameras have 12-15 EV. Simple gamma corrections (invented for CRT
screens) don't work anymore, and video editors have invented a new
standard able to remap the dynamic range and to fix the mid-tones at
once :
https://www.youtube.com/watch?v=kVKnhJN-BrQ&index=7&list=PLa1F2ddGya_9XER0wnFS6Mgnp3T-hgSZO

I have embedded the formula used in Blender
(https://en.wikipedia.org/wiki/ASC_CDL) into the profile correction
module of dt (using the same parameters for each RGB channel). The
result looks more natural than the current version, without gamut or
saturation issues in the highlights. It also speeds-up the worflow,
since all is needed is this module to adjust the dynamic range, then a
tone curve in auto RGB mode shaped as a stiff S to bring back the
contrast. The result is much better than with the tonemapping modules,
with less color fixes.

I'm a newbie at C and it's the first time I achieve something inside dt,
so I could use some reviews on my code and also some help on the OpenCL
part (the kernel does not load, I don't know why) :
https://github.com/aurelienpierre/darktable/tree/color-grading

Thanks a lot !

Aurélien.


___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] support New nikon Z7

[Germano Massullo]

For information completeness: https://redmine.darktable.org/issues/12322
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] An algorithm to downscale raw data BEFORE demosaic by whatever scale factor

[johannes hanika]

nice! an interesting lot of work indeed :)

the results look really promising. doesn't seem to do much on the sky,
but the wood is cleaned up nicely. some colour noise seems to stick
around at the edges, which is expected with nlm.

yeah multi-scale is hard and costly with nlm. one of the reasons why i
like wavelets more.

anscombe transform is great but has difficulties near black. we use
the anscombe + unbiased inverse in the profiled denoising module. if
you want to run before black point subtraction, you'd get noisy black
somewhere > 0 which may confuse the transform. not sure what to do
about it. i had a module at some point that was doing raw bayer
wavelets with a kind of Fisz transform to normalise variance to unity.
it was super slow and only really marginally better than what we had
for values near the black level, so i stopped developing it. i'm not
entirely sure we'll need such a thing for nlmeans though. curious to
hear if you have any findings!

cheers,
 jo
On Wed, Sep 12, 2018 at 9:22 AM rawfiner  wrote:
>
> hi,
>
> Le lun. 10 sept. 2018 à 14:47, johannes hanika  a écrit :
>>
>> hi!
>>
>> nice, your downscaled images look impeccable :) maybe they jump up or
>> down by one pixel or so?
>
>
> I think this is partly due to rounding errors as we multiply a width by a 
> float factor to find the new width.
> Also, as we get fewer and fewer lines and colums, it may be possible that 
> some details appear slightly moved, due to the algorithm itself.
>
>>
>>
>> do you have any example result images for the denoising already? are
>> you running before or after black point subtraction? if you can, i
>> think you should run before.
>
>
> For the moment, I develop the new algorithm as part of the raw denoise 
> module, so I guess it is after black point substraction.
> I will consider to make it run before.
>
> You can find to before/after examples here: 
> https://drive.google.com/open?id=18UU3AAHqyE0oE_zAvTdQbFL2i7zBAgCV
> Images that end with _01 are the "after".
> Note that only raw non local means was used to denoise these images, nothing 
> else. It reduces both luminance and chrominance noise.
>
> Yet, this is only a preliminary result, with a basic nlm algorithm, and I 
> have mainy ideas of improvements:
> - use anscombe transformation (and maybe improve it: see 
> https://ieeexplore.ieee.org/document/7812567/)
> - use downscaling and upscaling to get a multiscale denoising. This should 
> relax the constraints on the neighborhood parameter of the nlm algorithm, and 
> would allow better denoising for very noisy image
> - adapt total variation to raw data so that total variation could be used to 
> get a better patch comparison (MSE+TV), and to avoid denoising pixels which 
> are not noisy
> - use MSE+TV only to get a list of 8-16 most similar patches, and use "two 
> direction non local model" with these patches 
> (https://ieeexplore.ieee.org/abstract/document/6307863/). This should 
> completely solve the "rare patch" issue that we encounter with nlm.
> - and of course, make this run as fast as possible for preview!
>
> So basically, a lot of work remaining, but very exciting work ;-)
>
> cheers,
> rawfiner
>
>>
>>
>> exciting stuff :)
>>
>> cheers,
>>  jo
>>
>>
>> On Wed, Sep 5, 2018 at 9:34 PM rawfiner  wrote:
>> >
>> > Hi!
>> > Some of you may now that I am working on a raw denoising algorithm.
>> > One of the hard thing was that prior to demosaic, the algorithms are 
>> > computed on unscaled data, while after demosaic the algorithms can compute 
>> > a preview on a downscaled image, which is easier in terms of speed.
>> >
>> > So I tried to downscale the raw data, to be able to use that for preview.
>> > There was 2 existing algorithm in darktable to do that (thank you Johannes 
>> > for showing me these algorithms!): dt_iop_clip_and_zoom_mosaic_half_size_f 
>> > for bayer files and dt_iop_clip_and_zoom_mosaic_third_size_xtrans_f for 
>> > xtrans files
>> > My problem was that they only allow to downscale the images by a fixed 
>> > factor (2 in the case of bayer, and 3 in the case of xtrans).
>> >
>> > So I designed an algorithm, that work on both bayer and xtrans, and that 
>> > can be used whatever the scale factor.
>> > The source code is available here:
>> > https://github.com/rawfiner/darktable/tree/rawfiner-fix-downscale-algo
>> > (commit 9992cf66fc8510f637e5e5f8ae26c49c2cba2eaa)
>> > The graphic interface in raw denoise module is just here to be able to see 
>> > the effect of the algorithm in "fit to screen" zoom mode, and to activate 
>> > or desactivate the algorithm. It allows to compare what we get by 
>> > downscaling the picture before demosaic to what we would obtain without 
>> > this downscaling.
>> > The first slider controls the downscaling factor (0.25 means that width is 
>> > multiplied by 0.25, thus divided by 4)
>> > The second slider is useless for now.
>> >
>> > I made a quick video to compare the algorithm with the existing ones, and 
>> > to explain how the a