Thank you for all these explanations! Seems promising to me. Cheers,
rawfiner 2018-07-01 21:26 GMT+02:00 Aurélien Pierre <rese...@aurelienpierre.com>: > You're welcome ;-) > > That's true : the multiplication is equivalent to an "AND" operation, the > resulting mask has non-zero values where both TV AND Laplacian masks has > non-zero values, which - from my tests - is where the real noise is. > > That is because TV alone is too sensitive : when the image is noisy, it > works fine, but whenever the image is clean or barely noisy, it detect > edges as well, thus false-positive in the case of noise detection. > > The TV × Laplacian is a safety jacket that allows the TV to work as > expected on noisy images (see the example) but will protect sharp edges on > clean images (on the example, the masks barely grabs a few pixels in the > in-focus area). > > I have found that the only way we could overcome the oversensibility of > the TV alone is by setting a window (like a band-pass filter) instead of a > threshold (high-pass filter) because, in a noisy picture, depending on the > noise level, the TV values of noisy and edgy pixels are very close. From an > end-user perspective, this is tricky. > > Using TV × Laplacian, given that the noise stats should not vary much for > a given sensor at a given ISO, allows to confidently set a simple threshold > as a factor of the standard deviation. It gives more reproductibility and > allows to build preset/styles for given camera/ISO. Assuming gaussian > noise, if you set your threshold factor to X (which means "unmask > everything above the mean (TV × Laplacian) + X standard deviation), you > know beforehand how many high-frequency pixels will be affected, no matter > what : > > - X = -1 => 84 %, > - 0 => 50 %, > - 1 => 16 % , > - 2 => 2.5 %, > - 3 => 0.15 % > - … > > Le 01/07/2018 à 14:13, rawfiner a écrit : > > Thank you for this study Aurélien > > As far as I understand, TV and Laplacians are complementary as they detect > noise in different regions of the image (noise in sharp edge for Laplacian, > noise elsewhere for TV). > Though, I do not understand why you multiply the TV and Laplacian results > to get the mask. > Multiplying them would result in a mask containing non-zero values only > for pixels that are detected as noise both by TV and Laplacian. > Is there a particular reason for multiplying (or did I misunderstood > something?), or could we take the maximum value among TV and Laplacian for > each pixel instead? > > Thanks again > > Cheers, > rawfiner > > > 2018-07-01 3:45 GMT+02:00 Aurélien Pierre <rese...@aurelienpierre.com>: > >> Hi, >> >> I have done experiments on that matter and took the opportunity to >> correct/test further my code. >> >> So here are my attempts to code a noise mask and a sharpness mask with >> total variation and laplacian norms : https://github.com/aurelienpie >> rre/Image-Cases-Studies/blob/master/notebooks/Total% >> 20Variation%20masking.ipynb >> >> Performance benchmarks are at the end. >> >> Cheers, >> >> Aurélien. >> >> Le 17/06/2018 à 15:03, rawfiner a écrit : >> >> >> >> Le dimanche 17 juin 2018, Aurélien Pierre <rese...@aurelienpierre.com> a >> écrit : >> >>> >>> >>> Le 13/06/2018 à 17:31, rawfiner a écrit : >>> >>> >>> >>> Le mercredi 13 juin 2018, Aurélien Pierre <rese...@aurelienpierre.com> >>> a écrit : >>> >>>> >>>> >>>>> On Thu, Jun 14, 2018 at 12:23 AM, Aurélien Pierre >>>>> <rese...@aurelienpierre.com> wrote: >>>>> > Hi, >>>>> > >>>>> > The problem of a 2-passes denoising method involving 2 differents >>>>> > algorithms, the later applied where the former failed, could be the >>>>> grain >>>>> > structure (the shape of the noise) would be different along the >>>>> picture, >>>>> > thus very unpleasing. >>>> >>>> >>>> I agree that the grain structure could be different. Indeed, the grain >>>> could be different, but my feeling (that may be wrong) is that it would be >>>> still better than just no further processing, that leaves some pixels >>>> unprocessed (they could form grain structures far from uniform if we are >>>> not lucky). >>>> If you think it is only due to a change of algorithm, I guess we could >>>> apply non local means again on pixels where a first pass failed, but with >>>> different parameters to be quite confident that the second pass will work. >>>> >>>> That sounds better to me… but practice will have the last word. >>>> >>> >>> Ok :-) >>> >>>> >>>> >>>>> > >>>>> > I thought maybe we could instead create some sort of total variation >>>>> > threshold on other denoising modules : >>>>> > >>>>> > compute the total variation of each channel of each pixel as the >>>>> divergence >>>>> > divided by the L1 norm of the gradient - we then obtain a "heatmap" >>>>> of the >>>>> > gradients over the picture (contours and noise) >>>>> > let the user define a total variation threshold and form a mask >>>>> where the >>>>> > weights above the threshold are the total variation and the weights >>>>> below >>>>> > the threshold are zeros (sort of a highpass filter actually) >>>>> > apply the bilateral filter according to this mask. >>>>> > >>>>> > This way, if the user wants to stack several denoising modules, he >>>>> could >>>>> > protect the already-cleaned areas from further denoising. >>>>> > >>>>> > What do you think ? >>>> >>>> >>>> That sounds interesting. >>>> This would maybe allow to keep some small variations/details that are >>>> not due to noise or not disturbing, while denoising the other parts. >>>> Also, it may be computationally interesting (depends on the complexity >>>> of the total variation computation, I don't know it), as it could reduce >>>> the number of pixels to process. >>>> I guess the user could use something like that also the other way?: to >>>> protect high detailed zones and apply the denoising on quite smoothed zones >>>> only, in order to be able to use stronger denoising on zones that are >>>> supposed to be background blur. >>>> >>>> >>>> The noise is high frequency, so the TV (total variation) threshold will >>>> have to be high pass only. The hypothesis behind the TV thresholding is >>>> noisy pixels should have abnormally higher gradients than true details, so >>>> you isolate them this way. Selecting noise in low frequencies areas would >>>> require in addition something like a guided filter, which I believe is what >>>> is used in the dehaze module. The complexity of the TV computation depends >>>> on the order of accuracy you expect. >>>> >>>> A classic approximation of the gradient is using a convolution product >>>> with Sobel or Prewitt operators (3×3 arrays, very efficient, fairly >>>> accurate for edges, probably less accurate for punctual noise). I have >>>> developped myself optimized methods using 2, 4, and 8 neighbouring pixels >>>> that give higher order accuracy, given the sparsity of the data, at the >>>> expense of computing cost : https://github.com/aurelienpie >>>> rre/Image-Cases-Studies/blob/947fd8d5c2e4c3384c80c1045d86f8c >>>> f89ddcc7e/lib/deconvolution.pyx#L342 (ignore the variable ut in the >>>> code, only u is relevant for us here). >>>> >>> Great, thanks for the explanations. >>> Looking at the code of the 8 neighbouring pixels, I wonder if we would >>> make sense to compute something like that on raw data considering only >>> neighbouring pixels of the same color? >>> >>> >>> the RAW data are even more sparse, so the gradient can't be computed >>> this way. One would have to tweak the Taylor theorem to find an expression >>> of gradient for sparse data. And that would be different for Bayer and >>> X-Trans patterns. It's a bit of a conundrum. >>> >> >> Ok, thank you for these explainations >> >> >>> >>> Also, when talking about the mask formed from the heat map, do you mean >>> that the "heat" would give for each pixel a weight to use between input and >>> output? (i.e. a mask that is not only ones and zeros, but that controls how >>> much input and output are used for each pixel) >>> If so, I think it is a good idea to explore! >>> >>> yes, exactly, think of it as an opacity mask where you remap the >>> user-input TV threshold and the lower values to 0, the max magnitude of TV >>> to 1, and all the values in between accordingly. >>> >> >> Ok that is really cool! It seems a good idea to try to use that! >> >> rawfiner >> >> >>> >>> >>> rawfiner >>> >>>> >>>> >>>> >>>>> > >>>>> > Aurélien. >>>>> > >>>>> > >>>>> > Le 13/06/2018 à 03:16, rawfiner a écrit : >>>>> > >>>>> > Hi, >>>>> > >>>>> > I don't have the feeling that increasing K is the best way to >>>>> improve noise >>>>> > reduction anymore. >>>>> > I will upload the raw next week (if I don't forget to), as I am not >>>>> at home >>>>> > this week. >>>>> > My feeling is that doing non local means on raw data gives much >>>>> bigger >>>>> > improvement than that. >>>>> > I still have to work on it yet. >>>>> > I am currently testing some raw downsizing ideas to allow a fast >>>>> execution >>>>> > of the algorithm. >>>>> > >>>>> > Apart of that, I also think that to improve noise reduction such as >>>>> the >>>>> > denoise profile in nlm mode and the denoise non local means, we >>>>> could do a 2 >>>>> > passes algorithm, with non local means applied first, and then a >>>>> bilateral >>>>> > filter (or median filter or something else) applied only on pixels >>>>> where non >>>>> > local means failed to find suitable patches (i.e. pixels where the >>>>> sum of >>>>> > weights was close to 0). >>>>> > The user would have a slider to adjust this setting. >>>>> > I think that it would make easier to have a "uniform" output (i.e. >>>>> an output >>>>> > where noise has been reduced quite uniformly) >>>>> > I have not tested this idea yet. >>>>> > >>>>> > Cheers, >>>>> > rawfiner >>>>> > >>>>> > Le lundi 11 juin 2018, johannes hanika <hana...@gmail.com> a écrit : >>>>> >> >>>>> >> hi, >>>>> >> >>>>> >> i was playing with noise reduction presets again and tried the large >>>>> >> neighbourhood search window. on my shots i could very rarely spot a >>>>> >> difference at all increasing K above 7, and even less so going above >>>>> >> 10. the image you posted earlier did show quite a substantial >>>>> >> improvement however. i was wondering whether you'd be able to share >>>>> >> the image so i can evaluate on it? maybe i just haven't found the >>>>> >> right test image yet, or maybe it's camera dependent? >>>>> >> >>>>> >> (and yes, automatic and adaptive would be better but if we can ship >>>>> a >>>>> >> simple slider that can improve matters, maybe we should) >>>>> >> >>>>> >> cheers, >>>>> >> jo >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Mon, Jan 29, 2018 at 2:05 AM, rawfiner <rawfi...@gmail.com> >>>>> wrote: >>>>> >> > Hi >>>>> >> > >>>>> >> > Yes, the patch size is set to 1 from the GUI, so it is not a >>>>> bilateral >>>>> >> > filter, and I guess it corresponds to a patch window size of 3x3 >>>>> in the >>>>> >> > code. >>>>> >> > The runtime difference is near the expected quadratic slowdown: >>>>> >> > 1,460 secs (8,379 CPU) for 7 and 12,794 secs (85,972 CPU) for 25, >>>>> which >>>>> >> > means about 10.26x slowdown >>>>> >> > >>>>> >> > If you want to make your mind on it, I have pushed a branch here >>>>> that >>>>> >> > integrates the K parameter in the GUI: >>>>> >> > https://github.com/rawfiner/darktable.git >>>>> >> > The branch is denoise-profile-GUI-K >>>>> >> > >>>>> >> > I think that it may be worth to see if an automated approach for >>>>> the >>>>> >> > choice >>>>> >> > of K may work, in order not to integrate the parameter in the GUI. >>>>> >> > I may try to implement the approach of Kervann and Boulanger (the >>>>> >> > reference >>>>> >> > from the darktable blog post) to see how it performs. >>>>> >> > >>>>> >> > cheers, >>>>> >> > rawfiner >>>>> >> > >>>>> >> > >>>>> >> > 2018-01-27 13:50 GMT+01:00 johannes hanika <hana...@gmail.com>: >>>>> >> >> >>>>> >> >> heya, >>>>> >> >> >>>>> >> >> thanks for the reference! interesting interpretation how the >>>>> blotches >>>>> >> >> form. not sure i'm entirely convinced by that argument. >>>>> >> >> your image does look convincing though. let me get this right.. >>>>> you >>>>> >> >> ran with radius 1 which means patch window size 3x3? not 1x1 >>>>> which >>>>> >> >> would be a bilateral filter effectively? >>>>> >> >> >>>>> >> >> also what was the run time difference? is it near the expected >>>>> >> >> quadratic slowdown from 7 (i.e. 15x15) to 25 (51x51) so about >>>>> 11.56x >>>>> >> >> slower with the large window size? (test with darktable -d perf) >>>>> >> >> >>>>> >> >> since nlmeans isn't the fastest thing, even with this coalesced >>>>> way of >>>>> >> >> implementing it, we should certainly keep an eye on this. >>>>> >> >> >>>>> >> >> that being said if we can often times get much better results we >>>>> >> >> should totally expose this in the gui, maybe with a big warning >>>>> that >>>>> >> >> it really severely impacts speed. >>>>> >> >> >>>>> >> >> cheers, >>>>> >> >> jo >>>>> >> >> >>>>> >> >> On Sat, Jan 27, 2018 at 7:34 AM, rawfiner <rawfi...@gmail.com> >>>>> wrote: >>>>> >> >> > Thank you for your answer >>>>> >> >> > I perfectly agree with the fact that the GUI should not become >>>>> >> >> > overcomplicated. >>>>> >> >> > >>>>> >> >> > As far as I understand, the pixels within a small zone may >>>>> suffer >>>>> >> >> > from >>>>> >> >> > correlated noise, and there is a risk of noise to noise >>>>> matching. >>>>> >> >> > That's why this paper suggest not to take pixels that are too >>>>> close >>>>> >> >> > to >>>>> >> >> > the >>>>> >> >> > zone we are correcting, but to take them a little farther (see >>>>> the >>>>> >> >> > caption >>>>> >> >> > of Figure 2 for a quick explaination): >>>>> >> >> > >>>>> >> >> > >>>>> >> >> > >>>>> >> >> > https://pdfs.semanticscholar.org/c458/71830cf535ebe6c2b7656f >>>>> 6a205033761fc0.pdf >>>>> >> >> > (in case you ask, unfortunately there is a patent associated >>>>> with >>>>> >> >> > this >>>>> >> >> > approach, so we cannot implement it) >>>>> >> >> > >>>>> >> >> > Increasing the neighborhood parameter results in having >>>>> >> >> > proportionally >>>>> >> >> > less >>>>> >> >> > problem of correlation between surrounding pixels, and >>>>> decreases the >>>>> >> >> > size of >>>>> >> >> > the visible spots. >>>>> >> >> > See for example the two attached pictures: one with size 1, >>>>> force 1, >>>>> >> >> > and >>>>> >> >> > K 7 >>>>> >> >> > and the other with size 1, force 1, and K 25. >>>>> >> >> > >>>>> >> >> > I think that the best would probably be to adapt K >>>>> automatically, in >>>>> >> >> > order >>>>> >> >> > not to affect the GUI, and as we may have different levels of >>>>> noise >>>>> >> >> > in >>>>> >> >> > different parts of an image. >>>>> >> >> > In this post >>>>> >> >> > >>>>> >> >> > (https://www.darktable.org/2012/12/profiling-sensor-and-phot >>>>> on-noise/), >>>>> >> >> > this >>>>> >> >> > paper is cited: >>>>> >> >> > >>>>> >> >> > [4] charles kervrann and jerome boulanger: optimal spatial >>>>> adaptation >>>>> >> >> > for >>>>> >> >> > patch-based image denoising. ieee trans. image process. vol. >>>>> 15, no. >>>>> >> >> > 10, >>>>> >> >> > 2006 >>>>> >> >> > >>>>> >> >> > As far as I understand, it gives a way to choose an adaptated >>>>> window >>>>> >> >> > size >>>>> >> >> > for each pixel, but I don't see in the code anything related >>>>> to that >>>>> >> >> > >>>>> >> >> > Maybe is this paper related to the TODOs in the code ? >>>>> >> >> > >>>>> >> >> > Was it planned to implement such a variable window approach ? >>>>> >> >> > >>>>> >> >> > Or if it is already implemented, could you point me where ? >>>>> >> >> > >>>>> >> >> > Thank you >>>>> >> >> > >>>>> >> >> > rawfiner >>>>> >> >> > >>>>> >> >> > >>>>> >> >> > >>>>> >> >> > >>>>> >> >> > 2018-01-26 9:05 GMT+01:00 johannes hanika <hana...@gmail.com>: >>>>> >> >> >> >>>>> >> >> >> hi, >>>>> >> >> >> >>>>> >> >> >> if you want, absolutely do play around with K. in my tests it >>>>> did >>>>> >> >> >> not >>>>> >> >> >> lead to any better denoising. to my surprise a larger K often >>>>> led to >>>>> >> >> >> worse results (for some reason often the relevance of >>>>> discovered >>>>> >> >> >> patches decreases with distance from the current point). >>>>> that's why >>>>> >> >> >> K >>>>> >> >> >> is not exposed in the gui, no need for another irrelevant and >>>>> >> >> >> cryptic >>>>> >> >> >> parameter. if you find a compelling case where this indeed >>>>> leads to >>>>> >> >> >> better denoising we could rethink that. >>>>> >> >> >> >>>>> >> >> >> in general NLM is a 0-th order denoising scheme, meaning the >>>>> prior >>>>> >> >> >> is >>>>> >> >> >> piecewise constant (you claim the pixels you find are trying >>>>> to >>>>> >> >> >> express /the same/ mean, so you average them). if you let that >>>>> >> >> >> algorithm do what it would really like to, it'll create >>>>> unpleasant >>>>> >> >> >> blotches of constant areas. so for best results we need to >>>>> tone it >>>>> >> >> >> down one way or another. >>>>> >> >> >> >>>>> >> >> >> cheers, >>>>> >> >> >> jo >>>>> >> >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >> >> >> On Fri, Jan 26, 2018 at 7:36 AM, rawfiner <rawfi...@gmail.com >>>>> > >>>>> >> >> >> wrote: >>>>> >> >> >> > Hi >>>>> >> >> >> > >>>>> >> >> >> > I am surprised to see that we cannot control the >>>>> neighborhood >>>>> >> >> >> > parameter >>>>> >> >> >> > for >>>>> >> >> >> > the NLM algorithm (neither for the denoise non local mean, >>>>> nor for >>>>> >> >> >> > the >>>>> >> >> >> > denoise profiled) from the GUI. >>>>> >> >> >> > I see in the code (denoiseprofile.c) this TODO that I don't >>>>> >> >> >> > understand: >>>>> >> >> >> > "// >>>>> >> >> >> > TODO: fixed K to use adaptive size trading variance and >>>>> bias!" >>>>> >> >> >> > And just some lines after that: "// TODO: adaptive K tests >>>>> here!" >>>>> >> >> >> > (K is the neighborhood parameter of the NLM algorithm). >>>>> >> >> >> > >>>>> >> >> >> > In practice, I think that being able to change the >>>>> neighborhood >>>>> >> >> >> > parameter >>>>> >> >> >> > allows to have a better noise reduction for one image. >>>>> >> >> >> > For example, choosing a bigger K allows to reduce the >>>>> spotted >>>>> >> >> >> > aspect >>>>> >> >> >> > that >>>>> >> >> >> > one can get on high ISO images. >>>>> >> >> >> > >>>>> >> >> >> > Of course, increasing K increase computational time, but I >>>>> think >>>>> >> >> >> > we >>>>> >> >> >> > could >>>>> >> >> >> > find an acceptable range that would still be useful. >>>>> >> >> >> > >>>>> >> >> >> > >>>>> >> >> >> > Is there any reason for not letting the user control the >>>>> >> >> >> > neighborhood >>>>> >> >> >> > parameter in the GUI ? >>>>> >> >> >> > Also, do you understand the TODOs ? >>>>> >> >> >> > I feel that we would probably get better denoising by fixing >>>>> >> >> >> > these, >>>>> >> >> >> > but >>>>> >> >> >> > I >>>>> >> >> >> > don't understand them. >>>>> >> >> >> > >>>>> >> >> >> > I can spend some time on these TODOs, or to add the K >>>>> parameter to >>>>> >> >> >> > the >>>>> >> >> >> > interface if you think it is worth it (I think so but it is >>>>> only >>>>> >> >> >> > my >>>>> >> >> >> > personal >>>>> >> >> >> > opinion), but I have to understand what the TODOs mean >>>>> before >>>>> >> >> >> > >>>>> >> >> >> > Thank you for your help >>>>> >> >> >> > >>>>> >> >> >> > rawfiner >>>>> >> >> >> > >>>>> >> >> >> > >>>>> >> >> >> > >>>>> >> >> >> > >>>>> >> >> >> > ______________________________ >>>>> _____________________________________________ >>>>> >> >> >> > darktable developer mailing list to unsubscribe send a mail >>>>> to >>>>> >> >> >> > darktable-dev+unsubscr...@lists.darktable.org >>>>> >> >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >> >> >> ____________________________________________________________ >>>>> _______________ >>>>> >> >> >> darktable developer mailing list >>>>> >> >> >> to unsubscribe send a mail to >>>>> >> >> >> darktable-dev+unsubscr...@lists.darktable.org >>>>> >> >> >> >>>>> >> >> > >>>>> >> >> >>>>> >> >> >>>>> >> >> ____________________________________________________________ >>>>> _______________ >>>>> >> >> darktable developer mailing list >>>>> >> >> to unsubscribe send a mail to >>>>> >> >> darktable-dev+unsubscr...@lists.darktable.org >>>>> >> >> >>>>> >> > >>>>> > >>>>> > >>>>> > ____________________________________________________________ >>>>> _______________ >>>>> > darktable developer mailing list to unsubscribe send a mail to >>>>> > darktable-dev+unsubscr...@lists.darktable.org >>>>> > >>>>> > >>>>> > >>>>> > ____________________________________________________________ >>>>> _______________ >>>>> > darktable developer mailing list to unsubscribe send a mail to >>>>> > darktable-dev+unsubscr...@lists.darktable.org >>>>> ____________________________________________________________ >>>>> _______________ >>>>> darktable developer mailing list >>>>> to unsubscribe send a mail to darktable-dev+unsubscribe@list >>>>> s.darktable.org >>>>> >>>>> >>>> ___________________________________________________________________________ >>>> darktable developer mailing list to unsubscribe send a mail to >>>> darktable-dev+unsubscr...@lists.darktable.org >>>> >>>> >>>> >>>> ___________________________________________________________________________ >>>> darktable developer mailing list to unsubscribe send a mail to >>>> darktable-dev+unsubscr...@lists.darktable.org >>>> >>> >>> >>> ___________________________________________________________________________ >>> darktable developer mailing list to unsubscribe send a mail to >>> darktable-dev+unsubscr...@lists.darktable.org >>> >>> >>> >> >> >> >> >> ___________________________________________________________________________ >> darktable developer mailing list to unsubscribe send a mail to >> darktable-dev+unsubscr...@lists.darktable.org >> >> >> > > ___________________________________________________________________________ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
