Kay, I understand the French-Wilson way of currently doing things, as you outline below. My point is that it is not optimal --- we could do things better --- since even French-Wilson accepts the idea of negative intensity measurements. I am trying to disabuse the (very stubborn) view that when the background is more than the spot, the only possible estimate of the intensity is a negative value. This is untrue, and unjustified by the physics involved. In principle, there is no reason to use French-Wilson, as we should never have reported a negative integrated intensity to begin with.
I also understand that (Iobs-Icalc)^2 is not the actual refinement target, but the same point applies, and the actual target is based on a fundamental Gaussian assumption for the Is. On Jun 20, 2013, at 2:13 PM, Kay Diederichs <kay.diederi...@uni-konstanz.de> wrote: > Douglas, > > the intensity is negative if the integrated spot has a lower intensity than > the estimate of the background under the spot. So yes, we are not _measuring_ > negative intensities, rather we are estimating intensities, and that estimate > may turn out to be negative. In a later step we try to "correct" for this, > because it is non-physical, as you say. At that point, the "proper > statistical model" comes into play. Essentially we use this as a "prior". In > the order of increasing information, we can have more or less informative > priors for weak reflections: > 1) I > 0 > 2) I has a distribution looking like the right half of a Gaussian, and we > estimate its width from the variance of the intensities in a resolution shell > 3) I follows a Wilson distribution, and we estimate its parameters from the > data in a resolution shell > 4) I must be related to Fcalc^2 (i.e. once the structure is solved, we > re-integrate using the Fcalc as prior) > For a given experiment, the problem is chicken-and-egg in the sense that only > if you know the characteristics of the data can you choose the correct prior. > I guess that using prior 4) would be heavily frowned upon because there is a > danger of model bias. You could say: A Bayesian analysis done properly should > not suffer from model bias. This is probably true, but the theory to ensure > the word "properly" is not available at the moment. > Crystallographers usually use prior 3) which, as I tried to point out, also > has its weak spots, namely if the data do not behave like those of an ideal > crystal - and today's projects often result in data that would have been > discarded ten years ago, so they are far from ideal. > Prior 2) is available as an option in XDSCONV > Prior 1) seems to be used, or is available, in ctruncate in certain cases (I > don't know the details) > > Using intensities instead of amplitudes in refinement would avoid having to > choose a prior, and refinement would therefore not be compromised in case of > data violating the assumptions underlying the prior. > > By the way, it is not (Iobs-Icalc)^2 that would be optimized in refinement > against intensities, but rather the corresponding maximum likelihood formula > (which I seem to remember is more complicated than the amplitude ML formula, > or is not an analytical formula at all, but maybe somebody knows better). > > best, > > Kay > > > On Thu, 20 Jun 2013 13:14:28 -0400, Douglas Theobald <dtheob...@brandeis.edu> > wrote: > >> I still don't see how you get a negative intensity from that. It seems you >> are saying that in many cases of a low intensity reflection, the integrated >> spot will be lower than the background. That is not equivalent to having a >> negative measurement (as the measurement is actually positive, and sometimes >> things are randomly less positive than backgroiund). If you are using a >> proper statistical model, after background correction you will end up with a >> positive (or 0) value for the integrated intensity. >> >> >> On Jun 20, 2013, at 1:08 PM, Andrew Leslie <and...@mrc-lmb.cam.ac.uk> wrote: >> >>> >>> The integration programs report a negative intensity simply because that is >>> the observation. >>> >>> Because of noise in the Xray background, in a large sample of intensity >>> estimates for reflections whose true intensity is very very small one will >>> inevitably get some measurements that are negative. These must not be >>> rejected because this will lead to bias (because some of these intensities >>> for symmetry mates will be estimated too large rather than too small). It >>> is not unusual for the intensity to remain negative even after averaging >>> symmetry mates. >>> >>> Andrew >>> >>> >>> On 20 Jun 2013, at 11:49, Douglas Theobald <dtheob...@brandeis.edu> wrote: >>> >>>> Seems to me that the negative Is should be dealt with early on, in the >>>> integration step. Why exactly do integration programs report negative Is >>>> to begin with? >>>> >>>> >>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini <dom.bell...@diamond.ac.uk> >>>> wrote: >>>> >>>>> Wouldnt be possible to take advantage of negative Is to >>>>> extrapolate/estimate the decay of scattering background (kind of Wilson >>>>> plot of background scattering) to flat out the background and push all >>>>> the Is to positive values? >>>>> >>>>> More of a question rather than a suggestion ... >>>>> >>>>> D >>>>> >>>>> >>>>> >>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian >>>>> Tickle >>>>> Sent: 20 June 2013 17:34 >>>>> To: ccp4bb >>>>> Subject: Re: [ccp4bb] ctruncate bug? >>>>> >>>>> Yes higher R factors is the usual reason people don't like I-based >>>>> refinement! >>>>> >>>>> Anyway, refining against Is doesn't solve the problem, it only postpones >>>>> it: you still need the Fs for maps! (though errors in Fs may be less >>>>> critical then). >>>>> -- Ian >>>>> >>>>> On 20 June 2013 17:20, Dale Tronrud >>>>> <det...@uoxray.uoregon.edu<mailto:det...@uoxray.uoregon.edu>> wrote: >>>>> If you are refining against F's you have to find some way to avoid >>>>> calculating the square root of a negative number. That is why people >>>>> have historically rejected negative I's and why Truncate and cTruncate >>>>> were invented. >>>>> >>>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't >>>>> care less if Iobs happens to be negative. >>>>> >>>>> As for why people still refine against F... When I was distributing >>>>> a refinement package it could refine against I but no one wanted to do >>>>> that. The "R values" ended up higher, but they were looking at R >>>>> values calculated from F's. Of course the F based R values are lower >>>>> when you refine against F's, that means nothing. >>>>> >>>>> If we could get the PDB to report both the F and I based R values >>>>> for all models maybe we could get a start toward moving to intensity >>>>> refinement. >>>>> >>>>> Dale Tronrud >>>>> >>>>> >>>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote: >>>>> Just trying to understand the basic issues here. How could refining >>>>> directly against intensities solve the fundamental problem of negative >>>>> intensity values? >>>>> >>>>> >>>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp >>>>> <hofkristall...@gmail.com<mailto:hofkristall...@gmail.com>> wrote: >>>>> As a maybe better alternative, we should (once again) consider to refine >>>>> against intensities (and I guess George Sheldrick would agree here). >>>>> >>>>> I have a simple question - what exactly, short of some sort of historic >>>>> inertia (or memory lapse), is the reason NOT to refine against >>>>> intensities? >>>>> >>>>> Best, BR >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> This e-mail and any attachments may contain confidential, copyright and >>>>> or privileged material, and are for the use of the intended addressee >>>>> only. If you are not the intended addressee or an authorised recipient of >>>>> the addressee please notify us of receipt by returning the e-mail and do >>>>> not use, copy, retain, distribute or disclose the information in or >>>>> attached to the e-mail. >>>>> >>>>> Any opinions expressed within this e-mail are those of the individual and >>>>> not necessarily of Diamond Light Source Ltd. >>>>> >>>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any >>>>> attachments are free from viruses and we cannot accept liability for any >>>>> damage which you may sustain as a result of software viruses which may be >>>>> transmitted in or with the message. >>>>> >>>>> Diamond Light Source Limited (company no. 4375679). Registered in England >>>>> and Wales with its registered office at Diamond House, Harwell Science >>>>> and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>
