Douglas, the intensity is negative if the integrated spot has a lower intensity than the estimate of the background under the spot. So yes, we are not _measuring_ negative intensities, rather we are estimating intensities, and that estimate may turn out to be negative. In a later step we try to "correct" for this, because it is non-physical, as you say. At that point, the "proper statistical model" comes into play. Essentially we use this as a "prior". In the order of increasing information, we can have more or less informative priors for weak reflections: 1) I > 0 2) I has a distribution looking like the right half of a Gaussian, and we estimate its width from the variance of the intensities in a resolution shell 3) I follows a Wilson distribution, and we estimate its parameters from the data in a resolution shell 4) I must be related to Fcalc^2 (i.e. once the structure is solved, we re-integrate using the Fcalc as prior) For a given experiment, the problem is chicken-and-egg in the sense that only if you know the characteristics of the data can you choose the correct prior. I guess that using prior 4) would be heavily frowned upon because there is a danger of model bias. You could say: A Bayesian analysis done properly should not suffer from model bias. This is probably true, but the theory to ensure the word "properly" is not available at the moment. Crystallographers usually use prior 3) which, as I tried to point out, also has its weak spots, namely if the data do not behave like those of an ideal crystal - and today's projects often result in data that would have been discarded ten years ago, so they are far from ideal. Prior 2) is available as an option in XDSCONV Prior 1) seems to be used, or is available, in ctruncate in certain cases (I don't know the details)
Using intensities instead of amplitudes in refinement would avoid having to choose a prior, and refinement would therefore not be compromised in case of data violating the assumptions underlying the prior. By the way, it is not (Iobs-Icalc)^2 that would be optimized in refinement against intensities, but rather the corresponding maximum likelihood formula (which I seem to remember is more complicated than the amplitude ML formula, or is not an analytical formula at all, but maybe somebody knows better). best, Kay On Thu, 20 Jun 2013 13:14:28 -0400, Douglas Theobald <dtheob...@brandeis.edu> wrote: >I still don't see how you get a negative intensity from that. It seems you >are saying that in many cases of a low intensity reflection, the integrated >spot will be lower than the background. That is not equivalent to having a >negative measurement (as the measurement is actually positive, and sometimes >things are randomly less positive than backgroiund). If you are using a >proper statistical model, after background correction you will end up with a >positive (or 0) value for the integrated intensity. > > >On Jun 20, 2013, at 1:08 PM, Andrew Leslie <and...@mrc-lmb.cam.ac.uk> wrote: > >> >> The integration programs report a negative intensity simply because that is >> the observation. >> >> Because of noise in the Xray background, in a large sample of intensity >> estimates for reflections whose true intensity is very very small one will >> inevitably get some measurements that are negative. These must not be >> rejected because this will lead to bias (because some of these intensities >> for symmetry mates will be estimated too large rather than too small). It is >> not unusual for the intensity to remain negative even after averaging >> symmetry mates. >> >> Andrew >> >> >> On 20 Jun 2013, at 11:49, Douglas Theobald <dtheob...@brandeis.edu> wrote: >> >>> Seems to me that the negative Is should be dealt with early on, in the >>> integration step. Why exactly do integration programs report negative Is >>> to begin with? >>> >>> >>> On Jun 20, 2013, at 12:45 PM, Dom Bellini <dom.bell...@diamond.ac.uk> wrote: >>> >>>> Wouldnt be possible to take advantage of negative Is to >>>> extrapolate/estimate the decay of scattering background (kind of Wilson >>>> plot of background scattering) to flat out the background and push all the >>>> Is to positive values? >>>> >>>> More of a question rather than a suggestion ... >>>> >>>> D >>>> >>>> >>>> >>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian >>>> Tickle >>>> Sent: 20 June 2013 17:34 >>>> To: ccp4bb >>>> Subject: Re: [ccp4bb] ctruncate bug? >>>> >>>> Yes higher R factors is the usual reason people don't like I-based >>>> refinement! >>>> >>>> Anyway, refining against Is doesn't solve the problem, it only postpones >>>> it: you still need the Fs for maps! (though errors in Fs may be less >>>> critical then). >>>> -- Ian >>>> >>>> On 20 June 2013 17:20, Dale Tronrud >>>> <det...@uoxray.uoregon.edu<mailto:det...@uoxray.uoregon.edu>> wrote: >>>> If you are refining against F's you have to find some way to avoid >>>> calculating the square root of a negative number. That is why people >>>> have historically rejected negative I's and why Truncate and cTruncate >>>> were invented. >>>> >>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't >>>> care less if Iobs happens to be negative. >>>> >>>> As for why people still refine against F... When I was distributing >>>> a refinement package it could refine against I but no one wanted to do >>>> that. The "R values" ended up higher, but they were looking at R >>>> values calculated from F's. Of course the F based R values are lower >>>> when you refine against F's, that means nothing. >>>> >>>> If we could get the PDB to report both the F and I based R values >>>> for all models maybe we could get a start toward moving to intensity >>>> refinement. >>>> >>>> Dale Tronrud >>>> >>>> >>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote: >>>> Just trying to understand the basic issues here. How could refining >>>> directly against intensities solve the fundamental problem of negative >>>> intensity values? >>>> >>>> >>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp >>>> <hofkristall...@gmail.com<mailto:hofkristall...@gmail.com>> wrote: >>>> As a maybe better alternative, we should (once again) consider to refine >>>> against intensities (and I guess George Sheldrick would agree here). >>>> >>>> I have a simple question - what exactly, short of some sort of historic >>>> inertia (or memory lapse), is the reason NOT to refine against intensities? >>>> >>>> Best, BR >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> This e-mail and any attachments may contain confidential, copyright and or >>>> privileged material, and are for the use of the intended addressee only. >>>> If you are not the intended addressee or an authorised recipient of the >>>> addressee please notify us of receipt by returning the e-mail and do not >>>> use, copy, retain, distribute or disclose the information in or attached >>>> to the e-mail. >>>> >>>> Any opinions expressed within this e-mail are those of the individual and >>>> not necessarily of Diamond Light Source Ltd. >>>> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any >>>> attachments are free from viruses and we cannot accept liability for any >>>> damage which you may sustain as a result of software viruses which may be >>>> transmitted in or with the message. >>>> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England >>>> and Wales with its registered office at Diamond House, Harwell Science and >>>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>
