[darktable-dev] Re: freeze mode for histogram module

[Dan Torop]

Ah, just catching up. See RYB is merged!


On Tue, Sep 7, 2021, at 10:52 AM, Dan Torop wrote:
> Hi All,
> 
> That sounds good. I'll look forward to seeing Aurélien's work.
> 
> It's so exciting to see all these changes/developments to the "scopes". 
> (Though it looks like RYB may need a bit more rebasing, more work for 
> Philippe...)
> 
> At some point, if it doesn't make it into some other changes, I'd like 
> to make one more pass at the vectorscope visuals. This could include 
> changing drawing mode to "OVER" 
> (https://github.com/darktable-org/darktable/pull/9904#issuecomment-912847738) 
> as well as thinking again how the various overlays (center point, samples) 
> are drawn.
> 
> Currently the center point is a dark dot which can obscure graph 
> detail. Perhaps an alpha gradient there could make for a legible point 
> and still show some central detail.
> 
> The samples are currently drawn as light dots over a dimmed scope. As 
> samples will always appear over "bright" graph data, perhaps they could 
> be drawn as dark dots/circles (with alpha gradient?) over an undimmed 
> scope.
> 
> Also, long-standing ideas to draw input/work profile hue rings are 
> still interesting to me. And the vectorscope processing code needs some 
> tightening up.
> 
> Anyhow, all for the future...
> 
> Dan
> 
> 
> On Tue, Sep 7, 2021, at 7:33 AM, Pascal Obry wrote:
> > Hello developers,
> > 
> > Please refrain from changing the histogram module at this time. Aurélien 
> > has some large refactoring and enhancements started and any changes will 
> > require a rebasing and invalidate most of the work which is not nice.
> > 
> > I'll let you know when we exit the histogram module freeze mode. For now I 
> > won't merge any changes even small in histogram module.
> > 
> > Thanks,
> > 
> > -- 
> >   Pascal Obry /  Magny Les Hameaux (78)
> > 
> >   The best way to travel is by means of imagination
> > 
> >   http://photos.obry.net
> > 
> >   gpg --keyserver keys.gnupg.net --recv-key F949BD3B
> 
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

[darktable-dev] Re: freeze mode for histogram module

[Dan Torop]

Hi All,

That sounds good. I'll look forward to seeing Aurélien's work.

It's so exciting to see all these changes/developments to the "scopes". (Though 
it looks like RYB may need a bit more rebasing, more work for Philippe...)

At some point, if it doesn't make it into some other changes, I'd like to make 
one more pass at the vectorscope visuals. This could include changing drawing 
mode to "OVER" 
(https://github.com/darktable-org/darktable/pull/9904#issuecomment-912847738) 
as well as thinking again how the various overlays (center point, samples) are 
drawn.

Currently the center point is a dark dot which can obscure graph detail. 
Perhaps an alpha gradient there could make for a legible point and still show 
some central detail.

The samples are currently drawn as light dots over a dimmed scope. As samples 
will always appear over "bright" graph data, perhaps they could be drawn as 
dark dots/circles (with alpha gradient?) over an undimmed scope.

Also, long-standing ideas to draw input/work profile hue rings are still 
interesting to me. And the vectorscope processing code needs some tightening up.

Anyhow, all for the future...

Dan


On Tue, Sep 7, 2021, at 7:33 AM, Pascal Obry wrote:
> Hello developers,
> 
> Please refrain from changing the histogram module at this time. Aurélien has 
> some large refactoring and enhancements started and any changes will require 
> a rebasing and invalidate most of the work which is not nice.
> 
> I'll let you know when we exit the histogram module freeze mode. For now I 
> won't merge any changes even small in histogram module.
> 
> Thanks,
> 
> -- 
>   Pascal Obry /  Magny Les Hameaux (78)
> 
>   The best way to travel is by means of imagination
> 
>   http://photos.obry.net
> 
>   gpg --keyserver keys.gnupg.net --recv-key F949BD3B
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Translation of "scope"

[Dan Torop]

Interesting query! In the case of the manual, this would be a subset of the 
fourth definition:

  4: electronic equipment that provides visual images of varying
 electrical quantities [syn: {oscilloscope}, {scope},
 {cathode-ray oscilloscope}, {CRO}]

The use of this equipment became standard in video work to understand the video 
signal. Hence a video scope is a more specific version of this. See, for 
contemporary usage:

https://www.vegascreativesoftware.com/us/video-editing/video-scopes-waveform-vectorscope-for-accurate-color-grading/
https://www.bhphotovideo.com/explora/video/tips-and-solutions/introduction-waveforms-scopes-and-exposure
https://blog.frame.io/2017/09/27/introduction-to-video-scopes/

As the histogram readout in darktable has expanded to waveform and now RGB 
parade, it seems to make sense to use the more general term.

(Of course, the real change would be to rename src/libs/histogram.c to 
src/libs/scopes.c, but that might be a bridge too far...)

Dan



On Wed, Jul 29, 2020, at 9:09 AM, Patrick Shanahan wrote:
> * Jeronimo Pellegrini  [07-29-20 05:27]:
> > Hello,
> > 
> > I am updating the translation of the manual, and
> > I see that the work "scope" was introduced in the
> > last few days.
> > I'd like to confirm this: in this context, "scope"
> > has the same meaning as "screen of a measuring
> > device", such as "the screen of a oscilloscope"
> > (hence the similar name).  It does make a lot of sense
> > if I read the text like that. Is it correct?
> 
> scope is an area affected or an area of controll
> or the state of the environment in which a situation exists
> 
> ie:
>   n 1: an area in which something acts or operates or has power or
>  control: "the range of a supersonic jet"; "a piano has a
>  greater range than the human voice"; "the ambit of
>  municipal legislation"; "within the compass of this
>  article"; "within the scope of an investigation"; "outside
>  the reach of the law"; "in the political orbit of a world
>  power" [syn: {scope}, {range}, {reach}, {orbit}, {compass},
>  {ambit}]
>   2: the state of the environment in which a situation exists;
>  "you can't do that in a university setting" [syn: {setting},
>  {background}, {scope}]
>   3: a magnifier of images of distant objects [syn: {telescope},
>  {scope}]
>   4: electronic equipment that provides visual images of varying
>  electrical quantities [syn: {oscilloscope}, {scope},
>  {cathode-ray oscilloscope}, {CRO}]
>   
>   
> 
> 
> 
> 
> 
> 
> 
>  oscilloscope},
> 
> 
> {CRO}]
>   
>   
> -- 
> (paka)Patrick Shanahan   Plainfield, Indiana, USA  @ptilopteri
> http://en.opensuse.orgopenSUSE Community Memberfacebook/ptilopteri
> Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode
> ___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
> 
>
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] preparing 3.2

[Dan Torop]

I'm interested as well in ways for Filmic 4 to be as responsive as possible for 
interactive use. I believe that it's actually quite fast in its core work 
(parametrically creating and applying a tone curve to the image), even without 
OpenCL. But highlight reconstruction can bog things down, and even more so if 
high-quality reconstruction is > 0.

One way to handle this is to create an auto-applied preset which sets 
"threshold" in the "reconstruct" tab to 6. This should effectively turn off 
highlight reconstruction on a newly loaded image. Then make the needed 
adjustments via the "scene" and "look" tabs. All should be quite responsive. 
Following that, go to the "reconstruct" tab and double-click the "threshold" 
slider to get back to the default of -1, and do what is necessary to help the 
highlights.

I believe that the trade-off here is that you lose any black/white settings 
which may be pre-calculated from the EXIF exposure bias data. And, of course, 
there's a bit of extra mechanics.

-Dan



On Thu, Jul 16, 2020, at 1:48 AM, Peter Harde wrote:
> Hi,
> 
> my hardware is relatively new and has relatively high performance 
> (i7-8700, 6 core, 12 threads, Nvidia RTX 2060). But also with this 
> hardware, filmic 4 slows down the processing markedly. And combining 
> filmic 4 with tone equalizer increases the problem. It is obvious that 
> both modules are "heavy" due to the algorithms behind them. All the more 
> it seems very reasonable to me, to get OpenCL support for filmic 4.
> 
> Best regards
> 
> Peter Harde
> 
> ___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
> 
>
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Re: Custom Image Order by Drag and Drop

[Dan Torop]

Hi Mario,

Thank you! You should also, certainly, chat with the darktable
developers (IRC?) before going too far. They've been clear in the
past that they don't want someone putting in lots of work on a
feature if they don't think it'll make it in, or make it in in the
initial form. I'd be curious as to their thoughts on how custom image
sorting could work...
Dan



On Mon, Apr 16, 2018, at 3:47 PM, Mario Lüder wrote:
> Hi Dan, 
> 
> thanks for your suggestion and your interest. 
> 
> To your suggestion
>> Perhaps a simpler version is best: sort order associated not with a
>> particular "collect images" filter but as another column in the
>> "images" table. This would cause less worry about a lengthy join and
>> indexing on a TEXT column in "custom_image_order". And once
>> "places|beach" is given a custom sequence, it will maintain that even
>> if the collection is further filtered. And somewhat maintain that
>> sequence for other collections which overlap it. This would also not
>> leave "ghost" custom image orders laying order when a tag or film
>> roll is deleted. On the other hand, it wouldn't allow for creating
>> tags "sequence|v1" and then "sequence|v2" with utterly different
>> orders.>  
> I like your idea to simplify it. So I will implement it that way.
> First a version without the tags.>
>> Worth updating https://redmine.darktable.org/issues/9641 with a note
>> of your progress/interest, to add to the record. Easier to search
>> redmine than the mailing list.> 
> I'll check that.
> 
> Thanks again and best regards,
> Mario
> 
> 
> 
> 
> 
> 
> _-
> __ darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Re: Custom Image Order by Drag and Drop

[Dan Torop]

Hi Mario,

The patch looks sympathetic with the spirit of the darktable code, and you've 
clearly gotten deep into the code. A query, from reading it through (I haven't 
run it), is about sort order depending upon the setting of "collect images" as 
a string:

- What if the user filters the images by tags "places|beach" and 
"darktable|printed|%", and hand-sorts that collection. Then some other day the 
user reverses this filter (by "darktable|printed|%" and "places|beach"). The 
same images will display, but the sort order is lost.

- Similarly, what if the user filters the images by "places|beach" and makes a 
good sequence. Then they sub-filter by narrowing down to "darktable|printed|%". 
The sort order is now lost.

I wonder what the ideal behavior is here. Should each tag, film roll, etc. have 
its own potential custom sort order, and should successive filtering via 
"collect images" have no result on that order, hence only the first chosen 
filter (or the first chosen filter with a custom order) matter?

Perhaps a simpler version is best: sort order associated not with a particular 
"collect images" filter but as another column in the "images" table. This would 
cause less worry about a lengthy join and indexing on a TEXT column in 
"custom_image_order". And once "places|beach" is given a custom sequence, it 
will maintain that even if the collection is further filtered. And somewhat 
maintain that sequence for other collections which overlap it. This would also 
not leave "ghost" custom image orders laying order when a tag or film roll is 
deleted. On the other hand, it wouldn't allow for creating tags "sequence|v1" 
and then "sequence|v2" with utterly different orders.

To add to what Heiko wrote about contributing code:

- Worth updating https://redmine.darktable.org/issues/9641 with a note of your 
progress/interest, to add to the record. Easier to search redmine than the 
mailing list.

- It's highly recommended to visit irc.freenode.net, channel #darktable to 
check in with developers. A great way for them to get a sense of what you're up 
to, and for you to get a sense of their interests/concerns.

And to echo, making a PR is a great way to put your code on-the-record and 
allow a forum there for discussion/improvement.

I'm another who came to darktable hoping to find a way to reorder images. My 
work-around finally ended up being using the print module to make 4x6 work 
prints, and to arrange those images on a table. This works quite well, though 
of course requires more resources. I'd be thrilled if your work turns into a 
good way to resequence images in lighttable.

Regards,
Dan



On Mon, Apr 16, 2018, at 3:55 AM, Heiko Bauke wrote:
> Hi,
> 
> Am 15.04.2018 um 22:16 schrieb Mario Lüder:
> > Thanks for your interest. You may find my patch attached.
> 
> looks interesting and potentially useful.
> 
> I think the canonical way to contribute code to draktabe is to open a 
> pull request on github, see https://github.com/darktable-org/darktable/
> To get something into darktable, it is not sufficient to implement a 
> useful feature.  Darktable core developers also consider code quality.
> Be patient if your pull request is not merged immediately into 
> darktable.  After you have opened a pull request wait for feedback and 
> try to respond in a constructive manner to comments and criticism.
> 
> 
>   Heiko
> 
> 
> -- 
> -- Number Crunch Blog @ https://www.numbercrunch.de
> --  Cluster Computing @ https://www.clustercomputing.de
> --   Professional @ https://www.mpi-hd.mpg.de/personalhomes/bauke
> --  Social Networking @ https://www.researchgate.net/profile/Heiko_Bauke
> ___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
> 
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Planning of the upcoming 2.4.0 release

[Dan Torop]

Ulrich Pegelow  writes:

>>> Please consider that changes in that place also will require changes to
>>> the equivalent OpenCL code which might be anything but trivial.
>> 
>> Oh no! I was thinking of the dt_iop_clip_and_zoom_demosaic_{half,third}_*() 
>> functions in imageop_math.c. Those don't have OpenCL implementations, 
>> though? Certainly getting into revising OpenCL code would be a way more 
>> invasive effort.
>> 
>
> Well there are:
>
> demosaic_vng.cl:clip_and_zoom_demosaic_third_size_xtrans()
> demosaic_ppg.cl:clip_and_zoom_demosaic_half_size()
>

Oh no indeed! I haven't touched the floating point variants, and utterly failed 
to note this.

I've only been working on the non-float versions. These at least have no OpenCL 
equivalents, yes? They are only called from _init_f() in mipmap_cache.c for 
previews (which then never pass through the float downscale), as these are now 
downscaled while still mosaiced (which, alas, results in artifacts, but allows 
for processing mosaiced data through the pre-demosaic stages of the pipeline).

I believe that the float downscale code (CPU or OpenCL) is called in demosaic 
in the case of low quality thumbnails or in the full pipeline when the image is 
zoomed out and "demosaicing for zoomed out darkroom mode" is "always bilinear 
(fast)".

>> I also remember now that I wanted to look into whether taking advantage of 
>> box filters being separable would speed up the code. But that would involve 
>> allocating a bit of memory so might also be something about which to be 
>> conservative.
>
> I have not looked into your code, so I don't know how large the base of 
> the box filters is. But probably you implement them as gliding window 
> filters. That's exactly something you can't implement efficiently in OpenCL.
>

Right now the implementation is way too naive, not even a gliding window, just 
iterating over the source pixels. But a gliding window would be better.

> We have that situation in some modules like highpass coming from times 
> much before OpenCL. In the end I implemented a gaussian filter in OpenCL 
> that tries to mimic the box filter as close as possible. Still that is 
> not an ideal way ...

I actually tried a Gaussian filter and overall it produced very nice results, 
but was slow and gave artifacts around the edges of highlights (see 
https://redmine.darktable.org/issues/11167#note-28). If I recall, those 
highlight artifacts were actually worse, as the larger sigma of the Gaussian 
filter spread them around a bit more. But your OpenCL Gaussian implementation 
looks really nice, and makes me want try it for the float variants which are 
called later on in the pipeline, when highlights shouldn't be a problem.

The upshot of 11167 seemed to be to implement good-enough and fast-enough 
downscaling code for now (CPU and SSE path), and find a better algorithm 
eventually. If the uint16 variants have no OpenCL equivalents, I think there's 
latitude there to improve the CPU/SSE paths for 2.4.0.

For the float variants, in the case of thumbnails and zoomed out darkroom view 
I don't see any artifacts with the extant code on the CPU path. I'm thinking 
it's fine for now to leave the float code as is, hence requiring no changes to 
the corresponding OpenCL code. But eventually to consider converting the float 
variants to use a Gaussian filter.

Dan

> Ulrich
>

[...]

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Planning of the upcoming 2.4.0 release

[Dan Torop]

Ulrich Pegelow <ulrich.pege...@tongareva.de> writes:

> Am 21.06.2017 um 18:03 schrieb Dan Torop:
>>>
>>> I'd say that is well withing the time frame – provided it's not too 
>>> invasive.
>>>
>> 
>> That is great. Should be a tweak to the Bayer downscale code (making 
>> start/endpoints of sample right), bringing that work to X-Trans, and then 
>> SSE variants.
>> 
>
> Please consider that changes in that place also will require changes to 
> the equivalent OpenCL code which might be anything but trivial.

Oh no! I was thinking of the dt_iop_clip_and_zoom_demosaic_{half,third}_*() 
functions in imageop_math.c. Those don't have OpenCL implementations, though? 
Certainly getting into revising OpenCL code would be a way more invasive effort.

I also remember now that I wanted to look into whether taking advantage of box 
filters being separable would speed up the code. But that would involve 
allocating a bit of memory so might also be something about which to be 
conservative.

>
> Ulrich

[...]

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Planning of the upcoming 2.4.0 release

[Dan Torop]

Germano Massullo  writes:

> Il 21/06/2017 17:43, Tobias Ellinghaus ha scritto
>>> I've also been working on Wayland support
>>> (https://redmine.darktable.org/issues/11535). This is mostly done, but
>>> isn't worth enabling until GTK+ >= 3.22.16 makes its way into
>>> distributions. Wayland support doesn't seem to me to be critical for 2.4.0.
>> Ack, IMO Wayland is still to be considered experimental and not a top 
>> priority 
>> to fix.
> No intention of being pushy, but for information completeness, on Fedora
> Workstation Wayland is the default graphic server

Interesting... The code in git master should work with Wayland due to PR 1453, 
which requests GDK to use the XWayland compatibility layer as a backend in 
preference to Wayland. The continued work on bug 11535 should allow GTK/GDK to 
use the straight Wayland backend.

Wayland has exposed some quirks in darktable's UI code which X11 is too 
tolerant to notice. One might argue this is circular reasoning, and as these 
are problems only in the eyes of the Wayland display server.

[...]

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Planning of the upcoming 2.4.0 release

[Dan Torop]

Tobias Ellinghaus <m...@houz.org> writes:

> Am Mittwoch, 21. Juni 2017, 11:39:30 CEST schrieb Dan Torop:

[...]

>> I will be able to get to this again in early July. Is that soon enough to
>> make its way into 2.4.0? Additional improvement would be slightly better
>> quality zoomed-in rendering in darkroom view, and faster initial rendering
>> of the preview when loading a new image.
>
> I'd say that is well withing the time frame – provided it's not too invasive.
>

That is great. Should be a tweak to the Bayer downscale code (making 
start/endpoints of sample right), bringing that work to X-Trans, and then SSE 
variants.

[...]

> Ack, IMO Wayland is still to be considered experimental and not a top 
> priority 
> to fix.

Indeed. And it seems that Nvidia proprietary drivers and Wayland don't interact 
well (something about EGLStreams vs. GBM). In which case, Nvidia GPU users may 
not be deep into Wayland...

[...]

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Re: Unusual stability problem

[Dan Torop]

Thank you for the report & the very helpful sample file. I'm mortified, though, 
that the bug made its way into 2.2.2.

Dan


David Vincent-Jones  writes:

> My problem appears to be fixed now ... thanks for the quick work
>
> David
>
> On 01/30/2017 09:31 AM, Ulrich Pegelow wrote:
>
>  Great, this seems to have fixed the issue for me! 
>
>  Ulrich 
>
>  Am 30.01.2017 um 09:45 schrieb johannes hanika: 
>
>  thanks for providing the fix! seems correct to me, since the loop is y 
>  <= max and y+1 is accessed inside. i pushed this PR to master. let's 
>  see if it fixes it for everyone. 
>
>  cheers, 
>  jo 
>
>  ___ 
>  darktable developer mailing list 
>  to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org 
>
> ___ 
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org 

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Re: Unusual stability problem

[Dan Torop]

Many apologies. This was carelessness on my part. PR 1431 should fix this.

The bug was in changes I made to the Bayer downscaling code. For certain raw 
image dimensions, the downscaling to produce the preview would look at a pixel 
past the edge of the image.

Thank you very much for reporting & to Ulrich for the bisect work to narrow 
this down.

Dan



Ulrich Pegelow <ulrich.pege...@tongareva.de> writes:

> I checked further and it's obviously not (only) related to my system 
> upgrade, 'cause if I go back a few commits darktable runs stable.
>
> git bisect helped me to find the offending part:
>
> 07dc9664df548c7f775ade36cbdb7875a4aa4c9f is the first bad commit
> commit 07dc9664df548c7f775ade36cbdb7875a4aa4c9f
> Author: Dan Torop <d...@pnym.net>
> Date:   Mon Jan 23 21:27:48 2017 -0500
>
>  imageop_math: take advantage of CFA pattern on downscale
>
>  This increase speed to be faster than prior code w/out blurring.
>
> :04 04 88e97f48a9f6671c9e328008c1ad63b0ec0a4ab0 
> 153545841ee127841b78ee5b01f8334098ac4fcf M  src
>
> My backtraces indicate that the crash always is linked to
>
> dt_iop_clip_and_zoom_mosaic_half_size_plain._omp_fn.1 () at 
> /home/pegelow/darktable/src/develop/imageop_math.c:230
>
> which somehow makes it plausible that the a.m. commit might be involved.
>
> Ulrich
>
> Am 29.01.2017 um 08:03 schrieb Roman Lebedev:
>> On Sun, Jan 29, 2017 at 1:09 AM, Ulrich Pegelow
>> <ulrich.pege...@tongareva.de> wrote:
>>> Same here with current master on specific images with and without OpenCL. I
>>> have several images from one session, all have the same raw size. One image
>>> with an uncommon crop crashes whenever I open it in the darkroom.
>> Looks like the crash is in gtk / x?
>>
>>> Ulrich
>> Roman.
>>
>
> ___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] hotpixel seems to have a problem with an image

[Dan Torop]

Hi Marc,

That is great! Tobias merged the fix into master branch as of yesterday.
Thank you for pointing out the problem...

Dan


On Sat, Jul 23, 2016, at 11:23 AM, Marc Cousin wrote:
> I just applied your patch. It works perfectly now… I don't see a point
> in trying to prevent it on already processed images, as it works so much
> better now.
> 
> Thanks !
> 
> On 23/07/2016 02:20, Dan Torop wrote:
> > There should be a fix for this as PR 1225
> > (https://github.com/darktable-org/darktable/pull/1225). The column # was
> > off by one from the column of the pixel being examined, hence the code
> > was reading the wrong color values from the X-Trans sensor pattern. As
> > hotpixels works by comparing a given pixel with its neighbors of the
> > same color, due to this bug, often a pixel was compared with neighbors
> > of different color. In the example image, the code saw the intense red
> > as too-hot green pixels, "correcting" them with unfortunate results.
> > 
> > Note that this will change the behavior of the hotpixels iop for X-Trans
> > images (that is, threshold/strength values may need to be changed on
> > already processed images). I'm not sure if there is a useful way to
> > address this via iop version, or to (as with a recent change to the
> > highlight reconstruction iop) figure that if this is a long-standing bug
> > (and, in this case, to still-experimental X-Trans code), that the change
> > should just happen?
> > 
> > Dan
> > 
> > 
> > 
> > On Thu, Jul 21, 2016, at 11:45 AM, Marc Cousin wrote:
> >> I hope this one works.
> >>
> >>
> >> http://www.filedropper.com/xe021880
> >>
> >> Putting the ML back in CC in case someone else is interested :)
> >>
> >> On 21/07/2016 16:51, Dan Torop wrote:
> >>> Hi Marc,
> >>>
> >>> I just tried to download the example RAF you posted, but it seems to be 
> >>> expired already? If you're able to send me another link, I'd be curious 
> >>> to look -- though it might take a moment these days. I wrote that x-trans 
> >>> hotpixels code some time ago...
> >>>
> >>> Dan
> >>>
> >>>
> >>> Marc Cousin <cousinm...@gmail.com> writes:
> >>>
> >>>> Hi,
> >>>>
> >>>> I have a picture (Fuji XE2, XTrans, sorry :) ) with two hot (white) 
> >>>> pixels. One of them is perfectly corrected by the hotpixels module, the 
> >>>> other not. If I try to raise the strength, several other pixels (which 
> >>>> are correct) are corrected that shouldn't (in the red blurry flower).
> >>>>
> >>>> Both pixels are below the damselfly, and demosaiced bright white.
> >>>>
> >>>> As this happens even with an empty history, I just share the RAF for 
> >>>> now. Please tell me if an XMP is also needed.
> >>>>
> >>>> Here is the link to this RAF:
> >>>>
> >>>> http://dl.free.fr/rm.pl?h=hADGGHNK4=79426743=CcmFA50FqLHFmVY0exVdmcwI2CiOJZ27
> >>>>
> >>>>
> >>>> I don't know where it comes from, and I lack experience correcting hot 
> >>>> pixels: these two appeared on this picture and disappeared since then. I 
> >>>> suspect the camera was quite hot (very hot weather currently here, and 
> >>>> the camera stayed in the sun quite a bit). So 
> >>>>
> >>>> I don't know if this is a problem of me using it badly, a bug, or they 
> >>>> are not hot pixels and I'm wrong in assuming they should be corrected :)
> >>>>
> >>>> Regards
> >>>>
> >>>> Marc
> >>>> ___
> >>>> darktable developer mailing list
> >>>> to unsubscribe send a mail to 
> >>>> darktable-dev+unsubscr...@lists.darktable.org
> >>>
> >> ___
> >> darktable developer mailing list
> >> to unsubscribe send a mail to
> >> darktable-dev+unsubscr...@lists.darktable.org
> >>
> ___
> darktable developer mailing list
> to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
> 
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Usermanual update for 2.2

[Dan Torop]

PR #1230 is a pass at usermanual updates regarding X-Trans. There
actually isn't too much that is usermanual-visible. Most of the changes
have been bugfixes, optimizations, and general fixing up of existing
code.


On Sat, Jul 23, 2016, at 04:52 AM, Ulrich Pegelow wrote:
> Dear all,
> 
> I'd like to start an update of the usermanual to make it ready for 2.2.
> 
> Let's begin by compiling a list of required changes. From my side 
> following comes to mind and I added a proposal of whom might take care. 
> Please and your proposals.
> 
> * new module color check lut [Jo?]
> * new binary darktable-chart [Jo?]
> * new module liquify [Pascal?]
> * new module perspective correction [Ulrich]
> * various improvements for x-trans sensors [Dan?]
> * the "other" dropdown box [Tobias?]
> * new preferences tab for Lua options [Jeremy?]
> * updates to the collection module [Aldric?]
> * correction of outdated issues and factual errors [Roman?]
> * general style and proofreading [Ulrich]
> * .
> 
> My idea is to start working now. Maybe we can have text for the new 
> features complete by early September and have everything ready with all 
> proofreading etc. by mid October.
> 
> Ulrich
> ___
> darktable developer mailing list
> to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
> 
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] hotpixel seems to have a problem with an image

[Dan Torop]

There should be a fix for this as PR 1225
(https://github.com/darktable-org/darktable/pull/1225). The column # was
off by one from the column of the pixel being examined, hence the code
was reading the wrong color values from the X-Trans sensor pattern. As
hotpixels works by comparing a given pixel with its neighbors of the
same color, due to this bug, often a pixel was compared with neighbors
of different color. In the example image, the code saw the intense red
as too-hot green pixels, "correcting" them with unfortunate results.

Note that this will change the behavior of the hotpixels iop for X-Trans
images (that is, threshold/strength values may need to be changed on
already processed images). I'm not sure if there is a useful way to
address this via iop version, or to (as with a recent change to the
highlight reconstruction iop) figure that if this is a long-standing bug
(and, in this case, to still-experimental X-Trans code), that the change
should just happen?

Dan



On Thu, Jul 21, 2016, at 11:45 AM, Marc Cousin wrote:
> I hope this one works.
> 
> 
> http://www.filedropper.com/xe021880
> 
> Putting the ML back in CC in case someone else is interested :)
> 
> On 21/07/2016 16:51, Dan Torop wrote:
> > Hi Marc,
> > 
> > I just tried to download the example RAF you posted, but it seems to be 
> > expired already? If you're able to send me another link, I'd be curious to 
> > look -- though it might take a moment these days. I wrote that x-trans 
> > hotpixels code some time ago...
> > 
> > Dan
> > 
> > 
> > Marc Cousin <cousinm...@gmail.com> writes:
> > 
> >> Hi,
> >>
> >> I have a picture (Fuji XE2, XTrans, sorry :) ) with two hot (white) 
> >> pixels. One of them is perfectly corrected by the hotpixels module, the 
> >> other not. If I try to raise the strength, several other pixels (which are 
> >> correct) are corrected that shouldn't (in the red blurry flower).
> >>
> >> Both pixels are below the damselfly, and demosaiced bright white.
> >>
> >> As this happens even with an empty history, I just share the RAF for now. 
> >> Please tell me if an XMP is also needed.
> >>
> >> Here is the link to this RAF:
> >>
> >> http://dl.free.fr/rm.pl?h=hADGGHNK4=79426743=CcmFA50FqLHFmVY0exVdmcwI2CiOJZ27
> >>
> >>
> >> I don't know where it comes from, and I lack experience correcting hot 
> >> pixels: these two appeared on this picture and disappeared since then. I 
> >> suspect the camera was quite hot (very hot weather currently here, and the 
> >> camera stayed in the sun quite a bit). So 
> >>
> >> I don't know if this is a problem of me using it badly, a bug, or they are 
> >> not hot pixels and I'm wrong in assuming they should be corrected :)
> >>
> >> Regards
> >>
> >> Marc
> >> ___
> >> darktable developer mailing list
> >> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
> > 
> ___
> darktable developer mailing list
> to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
> 
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] weird colours when using several color modules

[Dan Torop]

I've been debugging various codepaths but nothing definitive -- it happens 
every few parameter changes, with no apparent pattern to occurrences. I don't 
have a deep understanding of the pixelpipe. Could it be the piece->pipe->xtrans 
for some mosaic-dependent iop after rawprepare gets initialized to the 
non-shifted CFA and never catches the shift done in rawprepare? Could there be 
a race condition with pipe/iop initializations which could cause this?

This definitely is in a083e4c and successive commits including 5590478.


Roman Lebedev <lebedev...@gmail.com> writes:

> On Wed, May 18, 2016 at 6:20 PM, Dan Torop <d...@pnym.net> wrote:
>> From here a bisect suggests 
>> https://github.com/darktable-org/darktable/commit/a083e4c0f9ffc0423613f3a75e5e3457aa11430c
>>  as the troublemaking commit.
> Not surprising. Thanks for bisecting it though.
> (but does it still happen with the next commit
> https://github.com/darktable-org/darktable/commit/5590478c7bede4061ab06b013d8d227135d08005
> i guess it does)
>
>> Fiddling with color balance seems a good way to cause this. I've also seen 
>> it when fiddling with white balance.
>
>>The preview in the navigation panel shows correct color, while the full image 
>>view shows the weird colors.
> Yep, because preview is demosaiced way before any pipe runs on it.
>
>> I can see this when the view is "fit to screen" or 100%.
>
>>In the later case, the image looks suspiciously like a CFA problem.See 
>>https://i.imgur.com/NG4G1y6.jpg.
> It definitely is.
> Either i forgot to change something to use  piece->pipe->xtrans
> instead of xtrans field in  dt_image_t,
> or missed some codepath that leads to it being set to wrong values.
>
>>
>> This is with OpenCL disabled and an X-Trans image. Will keep looking into 
>> this.
> No opencl here, and so far i have not seen this on bayer images.
>
>> Dan
> Roman.
>
>> Ulrich Pegelow <ulrich.pege...@tongareva.de> writes:
>>
>>> Am 17.05.2016 um 20:21 schrieb Roman Lebedev:
>>>> On Tue, May 17, 2016 at 9:13 PM, Ulrich Pegelow
>>>> <ulrich.pege...@tongareva.de> wrote:
>>>>> Hi,
>>>>>
>>>>> I can confirm your issue. You are running with OpenCL enabled? It looks 
>>>>> like
>>>>
>>>>> I have introduced an issue when porting xtrans demosaicing to OpenCL.
>>>>
>>>> Are you really sure?
>>>
>>> No, not sure. You are probably right, the issue has been introduced
>>> lately. If I go back in time a few days (commit
>>> 1b955380447ac717674ce05f3e9c6e6f0a3b8cc6) all seems good.
>>>
>>> Unfortunately I cannot bisect as my gcc 4.8.3 has issues compiling
>>> in-between commits.
>>>
>>> Maybe you can try to do that. At least here the problem can be quickly
>>> reproduced when quickly wheel-scrolling a few times over the exposure
>>> slider in the exposure module.
>>>
>>> Best wishes
>>>
>>> Ulrich
>>>
>>>
>>>> I'm somewhat expecting this to be due to
>>>> https://github.com/darktable-org/darktable/commit/a083e4c0f9ffc0423613f3a75e5e3457aa11430c
>>>> and
>>>> https://github.com/darktable-org/darktable/commit/5590478c7bede4061ab06b013d8d227135d08005
>>>>
>>>> but i do not understand what is missing...
>>>>
>>>> FWIW i was able to reproduce the issue without opencl, and only rarely,
>>>> when loading image in darkroom and just only changing colorzones once.
>>>>
>>>>> What I
>>>>> see are spurious color distortions (typically towards magenta) when 
>>>>> quickly
>>>>> changing some slider value like the exposure compensation. At the moment I
>>>>> cannot really say why we have these intermittent errors. This needs some 
>>>>> in
>>>>> depth investigations.
>>>>>
>>>>> Ulrich
>>>> Roman.
>>>
>>> ___
>>> darktable developer mailing list
>>> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
>>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] weird colours when using several color modules

[Dan Torop]

>From here a bisect suggests 
>https://github.com/darktable-org/darktable/commit/a083e4c0f9ffc0423613f3a75e5e3457aa11430c
> as the troublemaking commit.

Fiddling with color balance seems a good way to cause this. I've also seen it 
when fiddling with white balance. The preview in the navigation panel shows 
correct color, while the full image view shows the weird colors. I can see this 
when the view is "fit to screen" or 100%. In the later case, the image looks 
suspiciously like a CFA problem. See https://i.imgur.com/NG4G1y6.jpg.

This is with OpenCL disabled and an X-Trans image. Will keep looking into this.

Dan


Ulrich Pegelow  writes:

> Am 17.05.2016 um 20:21 schrieb Roman Lebedev:
>> On Tue, May 17, 2016 at 9:13 PM, Ulrich Pegelow
>>  wrote:
>>> Hi,
>>>
>>> I can confirm your issue. You are running with OpenCL enabled? It looks like
>>
>>> I have introduced an issue when porting xtrans demosaicing to OpenCL.
>>
>> Are you really sure?
>
> No, not sure. You are probably right, the issue has been introduced 
> lately. If I go back in time a few days (commit 
> 1b955380447ac717674ce05f3e9c6e6f0a3b8cc6) all seems good.
>
> Unfortunately I cannot bisect as my gcc 4.8.3 has issues compiling 
> in-between commits.
>
> Maybe you can try to do that. At least here the problem can be quickly 
> reproduced when quickly wheel-scrolling a few times over the exposure 
> slider in the exposure module.
>
> Best wishes
>
> Ulrich
>
>
>> I'm somewhat expecting this to be due to
>> https://github.com/darktable-org/darktable/commit/a083e4c0f9ffc0423613f3a75e5e3457aa11430c
>> and
>> https://github.com/darktable-org/darktable/commit/5590478c7bede4061ab06b013d8d227135d08005
>>
>> but i do not understand what is missing...
>>
>> FWIW i was able to reproduce the issue without opencl, and only rarely,
>> when loading image in darkroom and just only changing colorzones once.
>>
>>> What I
>>> see are spurious color distortions (typically towards magenta) when quickly
>>> changing some slider value like the exposure compensation. At the moment I
>>> cannot really say why we have these intermittent errors. This needs some in
>>> depth investigations.
>>>
>>> Ulrich
>> Roman.
>
> ___
> darktable developer mailing list
> to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Possible alternative to Markesteijn x-trans demosaic?

[Dan Torop]

Hi Ingo et al,

Exciting to see the continued work. I love the idea of working
diagonally on x-trans, rather than replicating Bayer methods. The
fringes on thin tree branches (as in example 1) have been a recurring
artifact for me in Markesteijn demosaic for me. I've used color
smoothing and median filtering on them, but also have had to
post-process by hand (e.g. desaturate bright magentas/greens). It
doesn't look like either algorithm has that great a time with such fine
detail.

The Markesteijn results do look sharper. Not always for the best, e.g.
the neutral background in example 4 is notably smoother in Liebhardt
version.

The results of Markesteijn through the default darktable pipeline look
so much nicer than straight dcraw output, I assume the Liebhardt
algorithm would be similarly improved.

Ingo: Have you read the ChromaSoft blog entries on X-Trans demosaicing?
(Starting at
http://chromasoft.blogspot.com/2012/05/demosaicing-fuji-x-pro1-and-its-x-trans.html
and ongoing.) I found them elucidating about the various
options/trade-offs and extant algorithms.

Dan


On Mon, Mar 21, 2016, at 04:35 PM, johannes hanika wrote:
> hi ingo,
> 
> very interesting. do you know why the shadow tone of the font on the
> toothpaste looks so different? looks like you're losing some contrast
> as compared to markesteijn? or is m. oversharpening?
> 
> sad to see how the thin branches in the trees have these colourful
> fringes. i guess that would go away with colour smoothing/median
> filtering?
> 
> cheers,
>  jo
> 
> On Tue, Mar 22, 2016 at 8:40 AM, Ingo Liebhardt <ingo.liebha...@ziggo.nl>
> wrote:
> > Hi Dan, and of course all potentially interested in x-trans demosaicing,
> >
> > In the meanwhile, I continued work on my algorithm and it’s now sufficiently
> > stable that I can start -
> > 1. documenting it, and
> > 2. trying to make a branch off of marketable master, following your below
> > suggestion.
> >
> > As you might have seen from yesterday’s mails, I’m trying to start getting
> > accustomed to dark table’s code and to where to find what.
> >
> > If you’re interested, I prepared some sample images using the original
> > Markesteijn and my approach (all tif, as they come out of dcraw, so beware
> > of the large files):
> > https://www.dropbox.com/sh/un1y11uimbqxjjk/AAD3L-Rs9-ztwyBIm4rnCzK-a?dl=0
> >
> > A couple of further notes:
> > The algorithm itself is clear and succinct, no worries. Please don’t be
> > deceived by the code, which doesn’t look succinct… It is plain OpenCL host
> > code without any wrapper, and that is awfully verbose. Concerning the
> > kernels, some also don’t look succinct, but that’s because I tried to use
> > constants rather than the far more succinct calculation rules, already
> > bearing performance in mind.
> >
> > Actually, I managed to reach a better quality in the meanwhile by not only
> > interpolating horizontally and vertically, but also diagonally. This is
> > something that you can only do with x-trans, so an advantage when compared
> > to Bayer.
> > Unfortunately it came with a performance penalty (no surprise), but I’m
> > confident that there’s room for further performance improvement later on.
> > Presently I’m back to around 13 seconds :-(
> > I also dropped the iterative approach, as I could not guarantee convergence.
> > Some images were even deteriorating with more than one iteration, others
> > were barely improving. All in all not satisfactory, so the approach is now
> > non-iterative: no loop.
> >
> > One other comment: I think that even now, darktable’s x-trans support is
> > more than just experimental. Even many commercial raw converters have quite
> > some difficulties with these x-trans files…
> > Some commercial products try to hide these problems by quite a degree of
> > softening, but they’re still there.
> > You also might want to consider that even Bayer CFAs cause quite some moiré
> > and false colour artifacts if they don’t have an AA filter.
> > Doing some expectation management here ;-)
> >
> >
> > Cheers,
> > Ingo
> >
> >
> > Am 15.02.2016 um 04:47 schrieb Dan Torop <d...@pnym.net>:
> >
> > Hi Ingo,
> >
> > Interesting to hear about the speed. That sounds promising. Here my
> > not-that-spectacular laptop is running darktable's Markesteijn
> > 1-pass/3-pass demosaic in approx. 1.4sec./3.8sec. respectively on the
> > CPU for 16 MP images. But there's no reason to worry about optimization
> > this early on. The code only got that fast after it was heavily worked
> > over by Ingo Weyrich from the RawTherapee project. Ulr