subject:"\"\\\\\\\[ccp4bb\\\\\\\] ctruncate bug\\\\\\\?\""

On Jun 22, 2013, at 6:18 PM, Frank von Delft  
wrote:

> A fascinating discussion (I've learnt a lot!);  a quick sanity check, though: 
> 
> In what scenarios would these improved estimates make a significant 
> difference?  

Who knows?  I always think that improved estimates are always a good thing, 
ignoring computational complexity (by "improved" I mean making more accurate 
physical assumptions).  This may all be academic --- estimating Itrue with 
unphysical negative values, and then later correcting w/French-Wilson, may give 
approximately the same answers and make no tangible difference in the models.  
But that all seems a bit convoluted, ad hoc, and unnecessary, esp. now with the 
available computational power.  It might make a difference.  

> Or rather:  are there any existing programs (as opposed to vapourware) that 
> would benefit significantly?
> 
> Cheers
> phx
> 
> 
> 
> On 22/06/2013 18:04, Douglas Theobald wrote:
>> Ian, I really do think we are almost saying the same thing.  Let me try to 
>> clarify.
>> 
>> You say that the Gaussian model is not the "correct" data model, and that 
>> the Poisson is correct.  I more-or-less agree.  If I were being pedantic 
>> (me?) I would say that the Poisson is *more* physically realistic than the 
>> Gaussian, and more realistic in a very important and relevant way --- but in 
>> truth the Poisson model does not account for other physical sources of error 
>> that arise from real crystals and real detectors, such as dark noise and 
>> read noise (that's why I would prefer a gamma distribution).  I also agree 
>> that for x>10 the Gaussian is a good approximation to the Poisson.  I 
>> basically agree with every point you make about the Poisson vs the Gaussian, 
>> except for the following.
>> 
>> The Iobs=Ispot-Iback equation cannot be derived from a Poisson assumption, 
>> except as an approximation when  Ispot > Iback.  It *can* be derived from 
>> the Gaussian assumption (and in fact I think that is probably the *only* 
>> justification it has).   It is true that the difference between two Poissons 
>> can be negative.  It is also true that for moderate # of counts, the 
>> Gaussian is a good approximation to the Poisson.  But we are trying to 
>> estimate Itrue, and both of those points are irrelevant to estimating Itrue 
>> when Ispot < Iback.  Contrary to your assertion, we are not concerned with 
>> differences of Poissonians, only sums.  Here is why:
>> 
>> In the Poisson model you outline, Ispot is the sum of two Poisson variables, 
>> Iback and Iobs.  That means Ispot is also Poisson and can never be negative. 
>>  Again --- the observed data (Ispot) is a *sum*, so that is what we must 
>> deal with.  The likelihood function for this model is:
>> 
>> L(a) = (a+b)^k exp(-a-b)
>> 
>> where 'k' is the # of counts in Ispot, 'a' is the mean of the Iobs Poisson 
>> (i.e., a = Itrue), and 'b' is the   mean of the Iback Poisson.  Of 
>> course k>=0, and both parameters a>0 and b>0.  Our job is to estimate 'a', 
>> Itrue.  Given the likelihood function above, there is no valid estimate of 
>> 'a' that will give a negative value.  For example, the ML estimate of 'a' is 
>> always non-negative.  Specifically, if we assume 'b' is known from 
>> background extrapolation, the ML estimate of 'a' is:
>> 
>> a = k-b   if k>b
>> 
>> a = 0   if k<=b
>> 
>> You can verify this visually by plotting the likelihood function (vs 'a' as 
>> variable) for any combination of k and b you want.  The SD is a bit more 
>> difficult, but it is approximately (a+b)/sqrt(k), where 'a' is now the ML 
>> estimate of 'a'.  
>> 
>> Note that the ML estimate of 'a', when k>b (Ispot>Iback), is equivalent to 
>> Ispot-Iback.  
>> 
>> Now, to restate:  as an estimate of Itrue, Ispot-Iback cannot be derived 
>> from the Poisson model.  In contrast, Ispot-Iback *can* be derived from a 
>> Gaussian model (as the ML and LS estimate of Itrue).  In fact, I'll wager 
>> the Gaussian is the only reasonable model that gives Ispot-Iback as an 
>> estimate of Itrue.  This is why I claim that using Ispot-Iback as an 
>> estimate of Itrue, even when Ispot> (non-physical) Gaussian model.  Feel free to prove me wrong --- can you 
>> derive Ispot-Iback, as an estimate of Itrue, from anything besides a 
>> Gaussian?
>> 
>> Cheers,
>> 
>> Douglas
>> 
>> 
>> 
>> 
>> On Sat, Jun 22, 2013 at 12:06 PM, Ian Tickle  wrote:
>> On 21 June 2013 19:45, Douglas Theobald  wrote:
>> 
>> The current way of doing things is summarized by Ed's equation: 
>> Ispot-Iback=Iobs.  Here Ispot is the # of counts in the spot (the area 
>> encompassing the predicted reflection), and Iback is # of counts in the 
>> background (usu. some area around the spot).  Our job is to estimate the 
>> true intensity Itrue.  Ed and others argue that Iobs is a reasonable 
>> estimate of Itrue, but I say it isn't because Itrue can never be negative, 
>> whereas Iobs can.
>> 
>> Now where does the Ispot-Iback=Iobs equation come

Re: [ccp4bb] ctruncate bug?

2013-06-22 Thread Frank von Delft

A fascinating discussion (I've learnt a lot!); a quick sanity check,
though:

In what scenarios would these improved estimates make a significant
difference?

Or rather: are there any existing programs (as opposed to vapourware)
that would benefit significantly?

Cheers
phx

On 22/06/2013 18:04, Douglas Theobald wrote:
Ian, I really do think we are almost saying the same thing. Let me
try to clarify.

You say that the Gaussian model is not the "correct" data model, and
that the Poisson is correct. I more-or-less agree. If I were being
pedantic (me?) I would say that the Poisson is *more* physically
realistic than the Gaussian, and more realistic in a very important
and relevant way --- but in truth the Poisson model does not account
for other physical sources of error that arise from real crystals and
real detectors, such as dark noise and read noise (that's why I would
prefer a gamma distribution). I also agree that for x>10 the Gaussian
is a good approximation to the Poisson. I basically agree with every
point you make about the Poisson vs the Gaussian, except for the
following.

The Iobs=Ispot-Iback equation cannot be derived from a Poisson
assumption, except as an approximation when Ispot > Iback. It *can*
be derived from the Gaussian assumption (and in fact I think that is
probably the *only* justification it has). It is true that the
difference between two Poissons can be negative. It is also true that
for moderate # of counts, the Gaussian is a good approximation to the
Poisson. But we are trying to estimate Itrue, and both of those
points are irrelevant to estimating Itrue when Ispot < Iback.
Contrary to your assertion, we are not concerned with differences of
Poissonians, only sums. Here is why:

In the Poisson model you outline, Ispot is the sum of two Poisson
variables, Iback and Iobs. That means Ispot is also Poisson and can
never be negative. Again --- the observed data (Ispot) is a *sum*, so
that is what we must deal with. The likelihood function for this
model is:

L(a) = (a+b)^k exp(-a-b)

where 'k' is the # of counts in Ispot, 'a' is the mean of the Iobs
Poisson (i.e., a = Itrue), and 'b' is the mean of the Iback Poisson.
Of course k>=0, and both parameters a>0 and b>0. Our job is to
estimate 'a', Itrue. Given the likelihood function above, there is no
valid estimate of 'a' that will give a negative value. For example,
the ML estimate of 'a' is always non-negative. Specifically, if we
assume 'b' is known from background extrapolation, the ML estimate of
'a' is:

a = k-b if k>b

a = 0 if k<=b

You can verify this visually by plotting the likelihood function (vs
'a' as variable) for any combination of k and b you want. The SD is a
bit more difficult, but it is approximately (a+b)/sqrt(k), where 'a'
is now the ML estimate of 'a'.

Note that the ML estimate of 'a', when k>b (Ispot>Iback), is
equivalent to Ispot-Iback.

Now, to restate: as an estimate of Itrue, Ispot-Iback cannot be
derived from the Poisson model. In contrast, Ispot-Iback *can* be
derived from a Gaussian model (as the ML and LS estimate of Itrue).
In fact, I'll wager the Gaussian is the only reasonable model that
gives Ispot-Iback as an estimate of Itrue. This is why I claim that
using Ispot-Iback as an estimate of Itrue, even when Ispotimplicitly means you are using a (non-physical) Gaussian model. Feel
free to prove me wrong --- can you derive Ispot-Iback, as an estimate
of Itrue, from anything besides a Gaussian?

Cheers,

Douglas

On Sat, Jun 22, 2013 at 12:06 PM, Ian Tickle > wrote:

On 21 June 2013 19:45, Douglas Theobald mailto:dtheob...@brandeis.edu>> wrote:

The current way of doing things is summarized by Ed's
equation: Ispot-Iback=Iobs. Here Ispot is the # of counts in
the spot (the area encompassing the predicted reflection), and
Iback is # of counts in the background (usu. some area around
the spot). Our job is to estimate the true intensity Itrue.
Ed and others argue that Iobs is a reasonable estimate of
Itrue, but I say it isn't because Itrue can never be negative,
whereas Iobs can.

Now where does the Ispot-Iback=Iobs equation come from? It
implicitly assumes that both Iobs and Iback come from a
Gaussian distribution, in which Iobs and Iback can have
negative values. Here's the implicit data model:

Ispot = Iobs + Iback

There is an Itrue, to which we add some Gaussian noise and
randomly generate an Iobs. To that is added some background
noise, Iback, which is also randomly generated from a Gaussian
with a "true" mean of Ibtrue. This gives us the Ispot, the
measured intensity in our spot. Given this data model, Ispot
will also have a Gaussian distribution, with mean equal to the
sum of Itrue + Ibtrue. From the properties of

Re: [ccp4bb] ctruncate bug?

On Sat, Jun 22, 2013 at 1:56 PM, Ian Tickle  wrote:

> On 22 June 2013 18:04, Douglas Theobald  wrote:
>
>>  --- but in truth the Poisson model does not account for other physical
>> sources of error that arise from real crystals and real detectors, such as
>> dark noise and read noise (that's why I would prefer a gamma distribution).
>>
>
> A photon counter is a digital device, not an analogue one.  It starts at
> zero and adds 1 every time it detects a photon (or what it thinks is a
> photon).  Once added, it is physically impossible for it to subtract 1 from
> its accumulated count: it contains no circuit to do that.  It can certainly
> miss photons, so you end up with less than you should, and it can certainly
> 'see' photons where there were none (e.g. from instrumental noise), so you
> end up with more than you should.  However once a count has been
> accumulated in the digital memory it stays there until the memory is
> cleared for the next measurement, and you can never end up with less than
> that accumulated count and in particular not less than zero; the bits of
> memory where the counts are accumulated are simply not programmed to return
> negative numbers.  It has nothing to do with whether the crystal is real or
> not, all that matters is that photons from "somewhere" are arriving at and
> being counted by the detector.  The accumulated counts at any moment in
> time have a Poisson distribution since the photons arrive completely
> randomly in time.
>

I might add that if you are correct --- that the naive Poisson model is
appropriate (perhaps true for the latest and greatest detectors, evidently
Pilatus has no read-out noise or dark current) --- then the ML solution I
outlined is a good one (much better than the crude Ispot-Iback background
subtraction), and it provides rigorous SD estimates too.

Re: [ccp4bb] ctruncate bug?

2013-06-22 Thread Ian Tickle

On 22 June 2013 18:04, Douglas Theobald  wrote:

> Ian, I really do think we are almost saying the same thing.  Let me try to
> clarify.
>

I agree, but still only "almost"!

>  --- but in truth the Poisson model does not account for other physical
> sources of error that arise from real crystals and real detectors, such as
> dark noise and read noise (that's why I would prefer a gamma distribution).
>

A photon counter is a digital device, not an analogue one.  It starts at
zero and adds 1 every time it detects a photon (or what it thinks is a
photon).  Once added, it is physically impossible for it to subtract 1 from
its accumulated count: it contains no circuit to do that.  It can certainly
miss photons, so you end up with less than you should, and it can certainly
'see' photons where there were none (e.g. from instrumental noise), so you
end up with more than you should.  However once a count has been
accumulated in the digital memory it stays there until the memory is
cleared for the next measurement, and you can never end up with less than
that accumulated count and in particular not less than zero; the bits of
memory where the counts are accumulated are simply not programmed to return
negative numbers.  It has nothing to do with whether the crystal is real or
not, all that matters is that photons from "somewhere" are arriving at and
being counted by the detector.  The accumulated counts at any moment in
time have a Poisson distribution since the photons arrive completely
randomly in time.

> In the Poisson model you outline, Ispot is the sum of two Poisson
> variables, Iback and Iobs.  That means Ispot is also Poisson and can never
> be negative.  Again --- the observed data (Ispot) is a *sum*, so that is
> what we must deal with.  The likelihood function for this model is:
>
> No, Iobs is _not_ a Poisson variable, indeed I never said it was: I
explained that it's the difference of 2 Poissonians Ispot and Iback and
therefore approximately Gaussian (please re-read my previous email).  So
the sum of Poissonians does not come into it.  The only Poissonian variates
here are Ispot and Iback.  Neither is the background under Ispot a
Poissonian (let's call it Iback', so strictly speaking Ispot = Iobs +
Iback' and Iback is an estimate of Iback', quite possibly with a non-random
error).  This is because Iobs and Iback' are not observable photon counts.
QM does not allow you to separate Ispot into separate photon counts,
because photons are indistinguishable.  If the photons were labelled
'spot', 'back' and 'obs' then you could count Iobs independently and it
would be a Poissonian (and that would indeed solve all our problems!).
But, sadly, photons are indistinguishable, they don't arrive with handy
labels!

Does any of that change your view?

Cheers

-- Ian

Re: [ccp4bb] ctruncate bug?

On Sat, Jun 22, 2013 at 1:04 PM, Douglas Theobald wrote:

> Feel free to prove me wrong --- can you derive Ispot-Iback, as an estimate
> of Itrue, from anything besides a Gaussian?
>

OK, I'll prove myself wrong.   Ispot-Iback can be derived as an estimate of
Itrue, even when Ispot

Re: [ccp4bb] ctruncate bug?

Ian, I really do think we are almost saying the same thing.  Let me try to
clarify.

You say that the Gaussian model is not the "correct" data model, and that
the Poisson is correct.  I more-or-less agree.  If I were being pedantic
(me?) I would say that the Poisson is *more* physically realistic than the
Gaussian, and more realistic in a very important and relevant way --- but
in truth the Poisson model does not account for other physical sources of
error that arise from real crystals and real detectors, such as dark noise
and read noise (that's why I would prefer a gamma distribution).  I also
agree that for x>10 the Gaussian is a good approximation to the Poisson.  I
basically agree with every point you make about the Poisson vs the
Gaussian, except for the following.

The Iobs=Ispot-Iback equation cannot be derived from a Poisson assumption,
except as an approximation when  Ispot > Iback.  It *can* be derived from
the Gaussian assumption (and in fact I think that is probably the *only*
justification it has).   It is true that the difference between two
Poissons can be negative.  It is also true that for moderate # of counts,
the Gaussian is a good approximation to the Poisson.  But we are trying to
estimate Itrue, and both of those points are irrelevant to estimating Itrue
when Ispot < Iback.  Contrary to your assertion, we are not concerned with
differences of Poissonians, only sums.  Here is why:

In the Poisson model you outline, Ispot is the sum of two Poisson
variables, Iback and Iobs.  That means Ispot is also Poisson and can never
be negative.  Again --- the observed data (Ispot) is a *sum*, so that is
what we must deal with.  The likelihood function for this model is:

L(a) = (a+b)^k exp(-a-b)

where 'k' is the # of counts in Ispot, 'a' is the mean of the Iobs Poisson
(i.e., a = Itrue), and 'b' is the mean of the Iback Poisson.  Of course
k>=0, and both parameters a>0 and b>0.  Our job is to estimate 'a', Itrue.
 Given the likelihood function above, there is no valid estimate of 'a'
that will give a negative value.  For example, the ML estimate of 'a' is
always non-negative.  Specifically, if we assume 'b' is known from
background extrapolation, the ML estimate of 'a' is:

a = k-b   if k>b

a = 0   if k<=b

You can verify this visually by plotting the likelihood function (vs 'a' as
variable) for any combination of k and b you want.  The SD is a bit more
difficult, but it is approximately (a+b)/sqrt(k), where 'a' is now the ML
estimate of 'a'.

Note that the ML estimate of 'a', when k>b (Ispot>Iback), is equivalent to
Ispot-Iback.

Now, to restate:  as an estimate of Itrue, Ispot-Iback cannot be derived
from the Poisson model.  In contrast, Ispot-Iback *can* be derived from a
Gaussian model (as the ML and LS estimate of Itrue).  In fact, I'll wager
the Gaussian is the only reasonable model that gives Ispot-Iback as an
estimate of Itrue.  This is why I claim that using Ispot-Iback as an
estimate of Itrue, even when Ispot wrote:

> On 21 June 2013 19:45, Douglas Theobald  wrote:
>
>>
>> The current way of doing things is summarized by Ed's equation:
>> Ispot-Iback=Iobs.  Here Ispot is the # of counts in the spot (the area
>> encompassing the predicted reflection), and Iback is # of counts in the
>> background (usu. some area around the spot).  Our job is to estimate the
>> true intensity Itrue.  Ed and others argue that Iobs is a reasonable
>> estimate of Itrue, but I say it isn't because Itrue can never be negative,
>> whereas Iobs can.
>>
>> Now where does the Ispot-Iback=Iobs equation come from?  It implicitly
>> assumes that both Iobs and Iback come from a Gaussian distribution, in
>> which Iobs and Iback can have negative values.  Here's the implicit data
>> model:
>>
>> Ispot = Iobs + Iback
>>
>> There is an Itrue, to which we add some Gaussian noise and randomly
>> generate an Iobs.  To that is added some background noise, Iback, which is
>> also randomly generated from a Gaussian with a "true" mean of Ibtrue.  This
>> gives us the Ispot, the measured intensity in our spot.  Given this data
>> model, Ispot will also have a Gaussian distribution, with mean equal to the
>> sum of Itrue + Ibtrue.  From the properties of Gaussians, then, the ML
>> estimate of Itrue will be Ispot-Iback, or Iobs.
>>
>
> Douglas, sorry I still disagree with your model.  Please note that I do
> actually support your position, that Ispot-Iback is not the best estimate
> of Itrue.  I stress that I am not arguing against this conclusion, merely
> (!) with your data model, i.e. you are arriving at the correct conclusion
> despite using the wrong model!  So I think it's worth clearing that up.
>
> First off, I can assure you that there is no assumption, either implicit
> or explicit, that Ispot and Iback come from a Gaussian distribution.  They
> are both essentially measured photon counts (perhaps indirectly), so it is
> logically impossible that they could ever be negative, even with any
> experimental error you can i

Re: [ccp4bb] ctruncate bug?

2013-06-22 Thread Ian Tickle

On 21 June 2013 19:45, Douglas Theobald  wrote:

>
> The current way of doing things is summarized by Ed's equation:
> Ispot-Iback=Iobs.  Here Ispot is the # of counts in the spot (the area
> encompassing the predicted reflection), and Iback is # of counts in the
> background (usu. some area around the spot).  Our job is to estimate the
> true intensity Itrue.  Ed and others argue that Iobs is a reasonable
> estimate of Itrue, but I say it isn't because Itrue can never be negative,
> whereas Iobs can.
>
> Now where does the Ispot-Iback=Iobs equation come from?  It implicitly
> assumes that both Iobs and Iback come from a Gaussian distribution, in
> which Iobs and Iback can have negative values.  Here's the implicit data
> model:
>
> Ispot = Iobs + Iback
>
> There is an Itrue, to which we add some Gaussian noise and randomly
> generate an Iobs.  To that is added some background noise, Iback, which is
> also randomly generated from a Gaussian with a "true" mean of Ibtrue.  This
> gives us the Ispot, the measured intensity in our spot.  Given this data
> model, Ispot will also have a Gaussian distribution, with mean equal to the
> sum of Itrue + Ibtrue.  From the properties of Gaussians, then, the ML
> estimate of Itrue will be Ispot-Iback, or Iobs.
>

Douglas, sorry I still disagree with your model.  Please note that I do
actually support your position, that Ispot-Iback is not the best estimate
of Itrue.  I stress that I am not arguing against this conclusion, merely
(!) with your data model, i.e. you are arriving at the correct conclusion
despite using the wrong model!  So I think it's worth clearing that up.

First off, I can assure you that there is no assumption, either implicit or
explicit, that Ispot and Iback come from a Gaussian distribution.  They are
both essentially measured photon counts (perhaps indirectly), so it is
logically impossible that they could ever be negative, even with any
experimental error you can imagine.  The concept of a photon counter
counting a negative number of photons is simply a logical impossibility (it
would be like counting the coins in your pocket and coming up with a
negative number, even allowing for mistakes in counting!).  This
immediately rules out the idea that they are Gaussian.  Photon counting
where the photons appear completely randomly in time (essentially as a
consequence of the Heisenberg Uncertainly Principle) obeys a Poisson
distribution.  In fact we routinely estimate the standard uncertainties of
Ispot & Iback on the basis that they are Poissonian, i.e. using var(count)
= count.  That is hardly a Gaussian assumption for the uncertainty!

Here is the correct data model: there is a true Ispot which is (or is
proportional to) the diffracted energy from the _sum_ of the Bragg
diffraction spot and the background under the spot (this is not the same as
Iback).  This energy ends up as individual photons being counted at the
detector (I know there's a complication that some detectors are not
actually photon counters, but the result is the same: you end up with a
photon count, or something proportional to it).  However photons are
indistinguishable (they do not carry labels telling us where they came
from), so quantum mechanics doesn't even allow us to talk about photons
coming from different places: all we see are indistinguishable photons
arriving at the detector and literally being counted.  Therefore the
estimated Ispot being the total number of photons counted from Bragg +
background has a Poisson distribution.  There will be some experimental
error associated with the random-in-time appearance of photons and also
instrumental errors (e.g we might simply fail to count some of the photons,
or we might count extra photons coming from somewhere else), but whatever
the source of the error there is no way that the measured count of photons
can ever be negative.

Now obviously we want to estimate the background under the spot but we
can't do that by looking at the spot itself (because the photons are
indistinguishable).  So completely independently of the Ispot measurement
we look at a nearby representative (hopefully!) area where there are no
Bragg spots and count that also: there is a true Iback associated with this
and our estimate of it from counting photons.  Again, being a photon count
it is also Poissonian and will have some experimental error associated with
it, but regardless of what the error is Iback, like Ispot, can never be
negative.

Now we have two Poissonian variables Ispot & Iback and traditionally we
perform the calculation Iobs = Ispot - Iback (whatever meaning you want to
attach to Iobs).  Provided Ispot and Iback are 'sufficiently' large numbers
a Poisson distribution can be approximated by a Gaussian with the same mean
and standard deviation, but with the proviso that the variate of this
approximate Gaussian can never be negative.  In fact you only need about 10
counts or more in _both_ Ispot and Iback for the approximation to be pretty
goo

Re: [ccp4bb] ctruncate bug?

2013-06-21 Thread Terwilliger, Thomas C

I hope I am not duplicating too much of this fascinating discussion with these 
comments:  perhaps the main reason there is confusion about what to do is that 
neither F nor I is really the most suitable thing to use in refinement.  As 
pointed out several times in different ways, we don't measure F or I, we only 
measure counts on a detector.  As a convenience, we "process" our diffraction 
images to estimate I or F and their uncertainties and model these uncertainties 
as simple functions (e.g., a Gaussian).  There is no need in principle to do 
that, and if we were to refine instead against the raw image data these issues 
about positivity would disappear and our structures might even be a little 
better.

Our standard procedure is to estimate F or I from counts on the detector, then 
to use these estimates of F or I in refinement.  This is not so easy to do 
right because F or I contain many terms coming from many pixels and it is hard 
to model their statistics in detail.  Further, attempts we make to estimate 
either F or I as physically plausible values (e.g., using the fact that they 
are not negative) will generally be biased (the values after correction will 
generally be systematically low or systematically high, as is true for the 
French and Wilson correction and as would be true for the truncation of I at 
zero or above).

Randy's method for intensity refinement is an improvement because the 
statistics are treated more fully than just using an estimate of F or I and 
assuming its uncertainty has a simple distribution.  So why not avoid all the 
problems with modeling the statistics of processed data and instead refine 
against the raw data.  From the structural model you calculate F, from F and a 
detailed model of the experiment (the same model that is currently used in data 
processing) you calculate the counts expected on each pixel. Then you calculate 
the likelihood of the data given your models of the structure and of the 
experiment.  This would have lots of benefits because it would allow improved 
descriptions of the experiment (decay, absorption, detector sensitivity, 
diffuse scattering and other "background" on the images,on and on) that 
could lead to more accurate structures in the end.  Of course there are some 
minor issues about putting all this in computer memory for refinement

-Tom T

From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] on behalf of Phil 
[p...@mrc-lmb.cam.ac.uk]
Sent: Friday, June 21, 2013 2:50 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] ctruncate bug?

However you decide to argue the point, you must consider _all_ the observations 
of a reflection (replicates and symmetry related) together when you infer Itrue 
or F etc, otherwise you will bias the result even more. Thus you cannot 
(easily) do it during integration

Phil

Sent from my iPad

On 21 Jun 2013, at 20:30, Douglas Theobald  wrote:

> On Jun 21, 2013, at 2:48 PM, Ed Pozharski  wrote:
>
>> Douglas,
>>>> Observed intensities are the best estimates that we can come up with in an 
>>>> experiment.
>>> I also agree with this, and this is the clincher.  You are arguing that 
>>> Ispot-Iback=Iobs is the best estimate we can come up with.  I claim that is 
>>> absurd.  How are you quantifying "best"?  Usually we have some sort of 
>>> discrepancy measure between true and estimate, like RMSD, mean absolute 
>>> distance, log distance, or somesuch.  Here is the important point --- by 
>>> any measure of discrepancy you care to use, the person who estimates Iobs 
>>> as 0 when Iback>Ispot will *always*, in *every case*, beat the person who 
>>> estimates Iobs with a negative value.   This is an indisputable fact.
>>
>> First off, you may find it useful to avoid such words as absurd and 
>> indisputable fact.  I know political correctness may be sometimes overrated, 
>> but if you actually plan to have meaningful discussion, let's assume that 
>> everyone responding to your posts is just trying to help figure this out.
>
> I apologize for offending and using the strong words --- my intention was not 
> to offend.  This is just how I talk when brainstorming with my colleagues 
> around a blackboard, but of course then you can see that I smile when I say 
> it.
>
>> To address your point, you are right that J=0 is closer to "true intensity" 
>> then a negative value.  The problem is that we are not after a single 
>> intensity, but rather all of them, as they all contribute to electron 
>> density reconstruction.  If you replace negative Iobs with E(J), you would 
>> systematically inflate the averages, which may turn problematic in some 
>> cases.
>
> So, I get the point.  But even then, using any reas

Re: [ccp4bb] ctruncate bug?

2013-06-21 Thread Phil

However you decide to argue the point, you must consider _all_ the observations 
of a reflection (replicates and symmetry related) together when you infer Itrue 
or F etc, otherwise you will bias the result even more. Thus you cannot 
(easily) do it during integration

Phil

Sent from my iPad

On 21 Jun 2013, at 20:30, Douglas Theobald  wrote:

> On Jun 21, 2013, at 2:48 PM, Ed Pozharski  wrote:
> 
>> Douglas,
 Observed intensities are the best estimates that we can come up with in an 
 experiment.
>>> I also agree with this, and this is the clincher.  You are arguing that 
>>> Ispot-Iback=Iobs is the best estimate we can come up with.  I claim that is 
>>> absurd.  How are you quantifying "best"?  Usually we have some sort of 
>>> discrepancy measure between true and estimate, like RMSD, mean absolute 
>>> distance, log distance, or somesuch.  Here is the important point --- by 
>>> any measure of discrepancy you care to use, the person who estimates Iobs 
>>> as 0 when Iback>Ispot will *always*, in *every case*, beat the person who 
>>> estimates Iobs with a negative value.   This is an indisputable fact.
>> 
>> First off, you may find it useful to avoid such words as absurd and 
>> indisputable fact.  I know political correctness may be sometimes overrated, 
>> but if you actually plan to have meaningful discussion, let's assume that 
>> everyone responding to your posts is just trying to help figure this out.
> 
> I apologize for offending and using the strong words --- my intention was not 
> to offend.  This is just how I talk when brainstorming with my colleagues 
> around a blackboard, but of course then you can see that I smile when I say 
> it.  
> 
>> To address your point, you are right that J=0 is closer to "true intensity" 
>> then a negative value.  The problem is that we are not after a single 
>> intensity, but rather all of them, as they all contribute to electron 
>> density reconstruction.  If you replace negative Iobs with E(J), you would 
>> systematically inflate the averages, which may turn problematic in some 
>> cases.  
> 
> So, I get the point.  But even then, using any reasonable criterion, the 
> whole estimated dataset will be closer to the true data if you set all 
> "negative" intensity estimates to 0.  
> 
>> It is probably better to stick with "raw intensities" and construct 
>> theoretical predictions properly to account for their properties.
>> 
>> What I was trying to tell you is that observed intensities is what we get 
>> from experiment.  
> 
> But they are not what you get from the detector.  The detector spits out a 
> positive value for what's inside the spot.  It is we, as human agents, who 
> later manipulate and massage that data value by subtracting the background 
> estimate.  A value that has been subjected to a crude background subtraction 
> is not the raw experimental value.  It has been modified, and there must be 
> some logic to why we massage the data in that particular manner.  I agree, of 
> course, that the background should be accounted for somehow.  But why just 
> subtract it away?  There are other ways to massage the data --- see my other 
> post to Ian.  My argument is that however we massage the experimentally 
> observed value should be physically informed, and allowing negative intensity 
> estimates violates the basic physics.  
> 
> [snip]
> 
 These observed intensities can be negative because while their true 
 underlying value is positive, random errorsmay result in Iback>Ispot.  
 There is absolutely nothing unphysical here.
>>> Yes there is.  The only way you can get a negative estimate is to make 
>>> unphysical assumptions.  Namely, the estimate Ispot-Iback=Iobs assumes that 
>>> both the true value of I and the background noise come from a Gaussian 
>>> distribution that is allowed to have negative values.  Both of those 
>>> assumptions are unphysical.
>> 
>> See, I have a problem with this.  Both common sense and laws of physics 
>> dictate that number of photons hitting spot on a detector is a positive 
>> number.  There is no law of physics that dictates that under no 
>> circumstances there could be Ispot 
> That's not what I'm saying.  Sure, Ispot can be less than Iback randomly.  
> That does not mean we have to estimate the detected intensity as negative, 
> after accounting for background.
> 
>> Yes, E(Ispot)>=E(Iback).  Yes, E(Ispot-Iback)>=0.  But P(Ispot-Iback=0)>0, 
>> and therefore experimental sampling of Ispot-Iback is bound to occasionally 
>> produce negative values.  What law of physics is broken when for a given 
>> reflection total number of photons in spot pixels is less that total number 
>> of photons in equal number of pixels in the surrounding background mask?
>> 
>> Cheers,
>> 
>> Ed.
>> 
>> -- 
>> Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
>>   Julian, King of Lemurs

Re: [ccp4bb] ctruncate bug?

On Jun 21, 2013, at 2:52 PM, James Holton  wrote:

> Yes, but the DIFFERENCE between two Poisson-distributed values can be 
> negative.  This is, unfortunately, what you get when you subtract the 
> background out from under a spot.  Perhaps this is the source of confusion 
> here?

Maybe, but if you assume Poisson background and intensities, the ML estimate 
when background > measured intensity is not negative, nor is it the difference 
Ispot-Iback.  The ML estimate is 0.  (With a finite non-zero SD, smaller SD the 
smaller the Ispot/Iback ratio).

> On Fri, Jun 21, 2013 at 11:34 AM, Douglas Theobald  
> wrote:
> I kinda think we're saying the same thing, sort of.
> 
> You don't like the Gaussian assumption, and neither do I.  If you make the 
> reasonable Poisson assumptions, then you don't get the Ispot-Iback=Iobs for 
> the best estimate of Itrue.  Except as an approximation for large values, but 
> we are talking about the case when Iback>Ispot, where the Gaussian 
> approximation to the Poisson no longer holds.  The sum of two Poisson 
> variates is also Poisson, which also can never be negative, unlike the 
> Gaussian.
> 
> So I reiterate: the Ispot-Iback=Iobs equation assumes Gaussians and hence 
> negativity.  The Ispot-Iback=Iobs does not follow from a Poisson assumption.
> 
> 
> On Jun 21, 2013, at 1:13 PM, Ian Tickle  wrote:
> 
> > On 21 June 2013 17:10, Douglas Theobald  wrote:
> >> Yes there is.  The only way you can get a negative estimate is to make 
> >> unphysical assumptions.  Namely, the estimate Ispot-Iback=Iobs assumes 
> >> that both the true value of I and the background noise come from a 
> >> Gaussian distribution that is allowed to have negative values.  Both of 
> >> those assumptions are unphysical.
> >
> > Actually that's not correct: Ispot and Iback are both assumed to come from 
> > a _Poisson_ distribution which by definition is zero for negative values of 
> > its argument (you can't have a negative number of photons), so are _not_ 
> > allowed to have negative values.  For large values of the argument (in fact 
> > the approximation is pretty good even for x ~ 10) a Poisson approximates to 
> > a Gaussian, and then of course the difference Ispot-Iback is also 
> > approximately Gaussian.
> >
> > But I think that doesn't affect your argument.
> >
> > Cheers
> >
> > -- Ian
>

Re: [ccp4bb] ctruncate bug?

On Jun 21, 2013, at 2:48 PM, Ed Pozharski  wrote:

> Douglas,
>>> Observed intensities are the best estimates that we can come up with in an 
>>> experiment.
>> I also agree with this, and this is the clincher.  You are arguing that 
>> Ispot-Iback=Iobs is the best estimate we can come up with.  I claim that is 
>> absurd.  How are you quantifying "best"?  Usually we have some sort of 
>> discrepancy measure between true and estimate, like RMSD, mean absolute 
>> distance, log distance, or somesuch.  Here is the important point --- by any 
>> measure of discrepancy you care to use, the person who estimates Iobs as 0 
>> when Iback>Ispot will *always*, in *every case*, beat the person who 
>> estimates Iobs with a negative value.   This is an indisputable fact.
> 
> First off, you may find it useful to avoid such words as absurd and 
> indisputable fact.  I know political correctness may be sometimes overrated, 
> but if you actually plan to have meaningful discussion, let's assume that 
> everyone responding to your posts is just trying to help figure this out.

I apologize for offending and using the strong words --- my intention was not 
to offend.  This is just how I talk when brainstorming with my colleagues 
around a blackboard, but of course then you can see that I smile when I say it. 
 

> To address your point, you are right that J=0 is closer to "true intensity" 
> then a negative value.  The problem is that we are not after a single 
> intensity, but rather all of them, as they all contribute to electron density 
> reconstruction.  If you replace negative Iobs with E(J), you would 
> systematically inflate the averages, which may turn problematic in some 
> cases.  

So, I get the point.  But even then, using any reasonable criterion, the whole 
estimated dataset will be closer to the true data if you set all "negative" 
intensity estimates to 0.  

> It is probably better to stick with "raw intensities" and construct 
> theoretical predictions properly to account for their properties.
> 
> What I was trying to tell you is that observed intensities is what we get 
> from experiment.  

But they are not what you get from the detector.  The detector spits out a 
positive value for what's inside the spot.  It is we, as human agents, who 
later manipulate and massage that data value by subtracting the background 
estimate.  A value that has been subjected to a crude background subtraction is 
not the raw experimental value.  It has been modified, and there must be some 
logic to why we massage the data in that particular manner.  I agree, of 
course, that the background should be accounted for somehow.  But why just 
subtract it away?  There are other ways to massage the data --- see my other 
post to Ian.  My argument is that however we massage the experimentally 
observed value should be physically informed, and allowing negative intensity 
estimates violates the basic physics.  

[snip]

>>> These observed intensities can be negative because while their true 
>>> underlying value is positive, random errorsmay result in Iback>Ispot.  
>>> There is absolutely nothing unphysical here.
>> Yes there is.  The only way you can get a negative estimate is to make 
>> unphysical assumptions.  Namely, the estimate Ispot-Iback=Iobs assumes that 
>> both the true value of I and the background noise come from a Gaussian 
>> distribution that is allowed to have negative values.  Both of those 
>> assumptions are unphysical.
> 
> See, I have a problem with this.  Both common sense and laws of physics 
> dictate that number of photons hitting spot on a detector is a positive 
> number.  There is no law of physics that dictates that under no circumstances 
> there could be Ispot Yes, E(Ispot)>=E(Iback).  Yes, E(Ispot-Iback)>=0.  But P(Ispot-Iback=0)>0, 
> and therefore experimental sampling of Ispot-Iback is bound to occasionally 
> produce negative values.  What law of physics is broken when for a given 
> reflection total number of photons in spot pixels is less that total number 
> of photons in equal number of pixels in the surrounding background mask?
> 
> Cheers,
> 
> Ed.
> 
> -- 
> Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
>Julian, King of Lemurs

Re: [ccp4bb] ctruncate bug?

2013-06-21 Thread Ed Pozharski

Douglas,

Observed intensities are the best estimates that we can come up with in an
experiment.

I also agree with this, and this is the clincher. You are arguing that Ispot-Iback=Iobs is
the best estimate we can come up with. I claim that is absurd. How are you quantifying
"best"? Usually we have some sort of discrepancy measure between true and
estimate, like RMSD, mean absolute distance, log distance, or somesuch. Here is the
important point --- by any measure of discrepancy you care to use, the person who estimates
Iobs as 0 when Iback>Ispot will *always*, in *every case*, beat the person who estimates
Iobs with a negative value. This is an indisputable fact.

First off, you may find it useful to avoid such words as absurd and
indisputable fact. I know political correctness may be sometimes
overrated, but if you actually plan to have meaningful discussion, let's
assume that everyone responding to your posts is just trying to help
figure this out.

To address your point, you are right that J=0 is closer to "true
intensity" then a negative value. The problem is that we are not after
a single intensity, but rather all of them, as they all contribute to
electron density reconstruction. If you replace negative Iobs with
E(J), you would systematically inflate the averages, which may turn
problematic in some cases. It is probably better to stick with "raw
intensities" and construct theoretical predictions properly to account
for their properties.

What I was trying to tell you is that observed intensities is what we
get from experiment. They may be negative, and there is nothing
unphysical about it. Then you build a theoretical estimate of observed
intensities, and if you do it right (i.e. by including experimental
errors), they will actually have some probability of being negative.

This background has to be subtracted and what is perhaps the most useful form
of observation is Ispot-Iback=Iobs.

How can that be the most useful form, when 0 is always a better estimate than a
negative value, by any criterion?

Given your propensity to refer to what others might say as absurd, I am
tempted to encourage *you* to come up with a better estimate.
Nevertheless, let me try to clarify my point.

What is measured in the experiment is Ispot. It contains Iback which
our theoretical models cannot possibly account for (because we have no
information at the refinement stage about crystal shape and other
parameters that define background). Strategy that has been in use for
decades is to obtain estimates of Iback from pixels surrounding the
integration spot. I hope you find that reasonable.

Once we have Iback estimated, Ispot-Iback becomes Iobs - observed
intensity. There is no need to convert that value simply to avoid bad
feeling brought by negative values. Correctly formulated theoretical
model predicts Iobs and accounts for error in it.

Let me state this again - Iobs are not true intensities and not
estimates of true intensities. They are experimental values sampling
Ispot-Iback. These can be negative. If a theoretical model that
approximates Iobs does not allow for negative Iobs, the model is flawed.

These observed intensities can be negative because while their true underlying
value is positive, random errorsmay result in Iback>Ispot. There is absolutely
nothing unphysical here.

Yes there is. The only way you can get a negative estimate is to make
unphysical assumptions. Namely, the estimate Ispot-Iback=Iobs assumes that
both the true value of I and the background noise come from a Gaussian
distribution that is allowed to have negative values. Both of those
assumptions are unphysical.

See, I have a problem with this. Both common sense and laws of physics
dictate that number of photons hitting spot on a detector is a positive
number. There is no law of physics that dictates that under no
circumstances there could be Ispot=E(Iback).
Yes, E(Ispot-Iback)>=0. But P(Ispot-Iback=0)>0, and therefore
experimental sampling of Ispot-Iback is bound to occasionally produce
negative values. What law of physics is broken when for a given
reflection total number of photons in spot pixels is less that total
number of photons in equal number of pixels in the surrounding
background mask?

Cheers,

Ed.

--
Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
Julian, King of Lemurs

Re: [ccp4bb] ctruncate bug?

surement (as the measurement is actually positive, and sometimes 
> things are randomly less positive than backgroiund).  If you are using a 
> proper statistical model, after background correction you will end up with a 
> positive (or 0) value for the integrated intensity.
> 
> 
> On Jun 20, 2013, at 1:08 PM, Andrew Leslie  wrote:
> 
> >
> > The integration programs report a negative intensity simply because that is 
> > the observation.
> >
> > Because of noise in the Xray background, in a large sample of intensity 
> > estimates for reflections whose true intensity is very very small one will 
> > inevitably get some measurements that are negative. These must not be 
> > rejected because this will lead to bias (because some of these intensities 
> > for symmetry mates will be estimated too large rather than too small). It 
> > is not unusual for the intensity to remain negative even after averaging 
> > symmetry mates.
> >
> > Andrew
> >
> >
> > On 20 Jun 2013, at 11:49, Douglas Theobald  wrote:
> >
> >> Seems to me that the negative Is should be dealt with early on, in the 
> >> integration step.  Why exactly do integration programs report negative Is 
> >> to begin with?
> >>
> >>
> >> On Jun 20, 2013, at 12:45 PM, Dom Bellini  
> >> wrote:
> >>
> >>> Wouldnt be possible to take advantage of negative Is to 
> >>> extrapolate/estimate the decay of scattering background (kind of Wilson 
> >>> plot of background scattering) to flat out the background and push all 
> >>> the Is to positive values?
> >>>
> >>> More of a question rather than a suggestion ...
> >>>
> >>> D
> >>>
> >>>
> >>>
> >>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
> >>> Tickle
> >>> Sent: 20 June 2013 17:34
> >>> To: ccp4bb
> >>> Subject: Re: [ccp4bb] ctruncate bug?
> >>>
> >>> Yes higher R factors is the usual reason people don't like I-based 
> >>> refinement!
> >>>
> >>> Anyway, refining against Is doesn't solve the problem, it only postpones 
> >>> it: you still need the Fs for maps! (though errors in Fs may be less 
> >>> critical then).
> >>> -- Ian
> >>>
> >>> On 20 June 2013 17:20, Dale Tronrud 
> >>> mailto:det...@uoxray.uoregon.edu>> wrote:
> >>> If you are refining against F's you have to find some way to avoid
> >>> calculating the square root of a negative number.  That is why people
> >>> have historically rejected negative I's and why Truncate and cTruncate
> >>> were invented.
> >>>
> >>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
> >>> care less if Iobs happens to be negative.
> >>>
> >>> As for why people still refine against F...  When I was distributing
> >>> a refinement package it could refine against I but no one wanted to do
> >>> that.  The "R values" ended up higher, but they were looking at R
> >>> values calculated from F's.  Of course the F based R values are lower
> >>> when you refine against F's, that means nothing.
> >>>
> >>> If we could get the PDB to report both the F and I based R values
> >>> for all models maybe we could get a start toward moving to intensity
> >>> refinement.
> >>>
> >>> Dale Tronrud
> >>>
> >>>
> >>> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
> >>> Just trying to understand the basic issues here.  How could refining 
> >>> directly against intensities solve the fundamental problem of negative 
> >>> intensity values?
> >>>
> >>>
> >>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
> >>> mailto:hofkristall...@gmail.com>> wrote:
> >>> As a maybe better alternative, we should (once again) consider to refine 
> >>> against intensities (and I guess George Sheldrick would agree here).
> >>>
> >>> I have a simple question - what exactly, short of some sort of historic 
> >>> inertia (or memory lapse), is the reason NOT to refine against 
> >>> intensities?
> >>>
> >>> Best, BR
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> This e-mail and any attachments may contain confidential, copyright and 
> >>> or privileged material, and are for the use of the intended addressee 
> >>> only. If you are not the intended addressee or an authorised recipient of 
> >>> the addressee please notify us of receipt by returning the e-mail and do 
> >>> not use, copy, retain, distribute or disclose the information in or 
> >>> attached to the e-mail.
> >>>
> >>> Any opinions expressed within this e-mail are those of the individual and 
> >>> not necessarily of Diamond Light Source Ltd.
> >>>
> >>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
> >>> attachments are free from viruses and we cannot accept liability for any 
> >>> damage which you may sustain as a result of software viruses which may be 
> >>> transmitted in or with the message.
> >>>
> >>> Diamond Light Source Limited (company no. 4375679). Registered in England 
> >>> and Wales with its registered office at Diamond House, Harwell Science 
> >>> and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >
>

Re: [ccp4bb] ctruncate bug?

I kinda think we're saying the same thing, sort of.

You don't like the Gaussian assumption, and neither do I.  If you make the 
reasonable Poisson assumptions, then you don't get the Ispot-Iback=Iobs for the 
best estimate of Itrue.  Except as an approximation for large values, but we 
are talking about the case when Iback>Ispot, where the Gaussian approximation 
to the Poisson no longer holds.  The sum of two Poisson variates is also 
Poisson, which also can never be negative, unlike the Gaussian.  

So I reiterate: the Ispot-Iback=Iobs equation assumes Gaussians and hence 
negativity.  The Ispot-Iback=Iobs does not follow from a Poisson assumption.  

On Jun 21, 2013, at 1:13 PM, Ian Tickle  wrote:

> On 21 June 2013 17:10, Douglas Theobald  wrote:
>> Yes there is.  The only way you can get a negative estimate is to make 
>> unphysical assumptions.  Namely, the estimate Ispot-Iback=Iobs assumes that 
>> both the true value of I and the background noise come from a Gaussian 
>> distribution that is allowed to have negative values.  Both of those 
>> assumptions are unphysical.
> 
> Actually that's not correct: Ispot and Iback are both assumed to come from a 
> _Poisson_ distribution which by definition is zero for negative values of its 
> argument (you can't have a negative number of photons), so are _not_ allowed 
> to have negative values.  For large values of the argument (in fact the 
> approximation is pretty good even for x ~ 10) a Poisson approximates to a 
> Gaussian, and then of course the difference Ispot-Iback is also approximately 
> Gaussian.
> 
> But I think that doesn't affect your argument.
> 
> Cheers
> 
> -- Ian

Re: [ccp4bb] ctruncate bug?

2013-06-21 Thread Ed Pozharski


On 06/21/2013 10:19 AM, Ian Tickle wrote:
If you observe the symptoms of translational NCS in the diffraction 
pattern (i.e. systematically weak zones of reflections) you must take 
it into account when calculating the averages, i.e. if you do it 
properly parity groups should be normalised separately (though I 
concede there may be a practical issue in that I'm not aware of any 
software that currently has this feature). 


Ian,

I think this is exactly what I was trying to emphasize, that applying 
some conversion to raw intensities may have negative impact when 
conversion is based on incorrect or incomplete assumptions.


Cheers,

Ed.

--
Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
Julian, King of Lemurs

Re: [ccp4bb] ctruncate bug?

2013-06-21 Thread Ian Tickle

On 21 June 2013 17:10, Douglas Theobald  wrote:

> Yes there is.  The only way you can get a negative estimate is to make
> unphysical assumptions.  Namely, the estimate Ispot-Iback=Iobs assumes that
> both the true value of I and the background noise come from a Gaussian
> distribution that is allowed to have negative values.  Both of those
> assumptions are unphysical.
>

Actually that's not correct: Ispot and Iback are both assumed to come from
a _Poisson_ distribution which by definition is zero for negative values of
its argument (you can't have a negative number of photons), so are _not_
allowed to have negative values.  For large values of the argument (in fact
the approximation is pretty good even for x ~ 10) a Poisson approximates to
a Gaussian, and then of course the difference Ispot-Iback is also
approximately Gaussian.

But I think that doesn't affect your argument.

Cheers

-- Ian

Re: [ccp4bb] ctruncate bug?

On Jun 21, 2013, at 8:36 AM, Ed Pozharski  wrote:

> On 06/20/2013 01:07 PM, Douglas Theobald wrote:
>> How can there be nothing "wrong" with something that is unphysical?  
>> Intensities cannot be negative.
> 
> I think you are confusing two things - the true intensities and observed 
> intensities.

But I'm not.  Let me try to convince you ...

> True intensities represent the number of photons that diffract off a crystal 
> in a specific direction or, for QED-minded, relative probabilities of a 
> single photon being found in a particular area of the detector when it's 
> probability wave function finally collapses.

I agree. 

> True intensities certainly cannot be negative and in crystallographic method 
> they never are. They are represented by the best theoretical estimates 
> possible, Icalc.  These are always positive.

I also very much agree.  

> Observed intensities are the best estimates that we can come up with in an 
> experiment.  

I also agree with this, and this is the clincher.  You are arguing that 
Ispot-Iback=Iobs is the best estimate we can come up with.  I claim that is 
absurd.  How are you quantifying "best"?  Usually we have some sort of 
discrepancy measure between true and estimate, like RMSD, mean absolute 
distance, log distance, or somesuch.  Here is the important point --- by any 
measure of discrepancy you care to use, the person who estimates Iobs as 0 when 
Iback>Ispot will *always*, in *every case*, beat the person who estimates Iobs 
with a negative value.   This is an indisputable fact.  

> These are determined by integrating pixels around the spot where particular 
> reflection is expected to hit the detector.  Unfortunately, science did not 
> yet invent a method that would allow to suspend a crystal in vacuum while 
> also removing all of the outside solvent.  Neither we have included diffuse 
> scatter in our theoretical model.  Because of that, full reflection intensity 
> contains background signal in addition to the Icalc.  This background has to 
> be subtracted and what is perhaps the most useful form of observation is 
> Ispot-Iback=Iobs.

How can that be the most useful form, when 0 is always a better estimate than a 
negative value, by any criterion?

> These observed intensities can be negative because while their true 
> underlying value is positive, random errorsmay result in Iback>Ispot.  There 
> is absolutely nothing unphysical here.

Yes there is.  The only way you can get a negative estimate is to make 
unphysical assumptions.  Namely, the estimate Ispot-Iback=Iobs assumes that 
both the true value of I and the background noise come from a Gaussian 
distribution that is allowed to have negative values.  Both of those 
assumptions are unphysical.  

> Replacing Iobs with E(J) is not only unnecessary, it's ill-advised as it will 
> distort intensity statistics.  For example, let's say you have translational 
> NCS aligned with crystallographic axes, and hence some set of reflections is 
> systematically absent.  If all is well, ~0 for the subset while  
> is systematically positive.  This obviously happens because the standard 
> Wilson prior is wrong for these reflections, but I digress, as usual.
> 
> In summary, there is indeed nothing wrong, imho, with negative Iobs.  The 
> fact that some of these may become negative is correctly accounted for once 
> sigI is factored into the ML target.
> 
> Cheers,
> 
> Ed.
> 
> -- 
> Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
>Julian, King of Lemurs
>

Re: [ccp4bb] ctruncate bug?

2013-06-21 Thread Ian Tickle

On 21 June 2013 13:36, Ed Pozharski  wrote:

> Replacing Iobs with E(J) is not only unnecessary, it's ill-advised as it
> will distort intensity statistics.  For example, let's say you have
> translational NCS aligned with crystallographic axes, and hence some set of
> reflections is systematically absent.  If all is well, ~0 for the
> subset while  is systematically positive.  This obviously happens
> because the standard Wilson prior is wrong for these reflections, but I
> digress, as usual.
>

Ed,

If you observe the symptoms of translational NCS in the diffraction pattern
(i.e. systematically weak zones of reflections) you must take it into
account when calculating the averages, i.e. if you do it properly parity
groups should be normalised separately (though I concede there may be a
practical issue in that I'm not aware of any software that currently has
this feature).  In that case  will be ~ 0, as expected.  If you don't
do that then clearly you can't expect to get the right answer!  The
theoretical intensities are based on the assumption that the intensity
distributions are all positive, so it makes no sense to compare them with
an experimental distribution where a significant fraction are negative.
How exactly do you propose to deal properly with the P-Y L test that I
described? - because of course that also inherently assumes that the
intensities are all positive and it's certainly not valid to assume that
E(J) = E(F)^2 !

Another point is that (to paraphrase G. Orwell) "not all reflections are
created equal, just some are more equal than others".  What I mean is that
in counting reflections for the cumulative distributions (i.e. you count
the number of reflections in ranges of intensity or in ranges of L), a weak
reflection should be counted as fractional with a contribution to the total
which is less than 1, on a continuous scale from 0 to 1 related to
I/sigma(I).  In fact referring to your original posting reflections with h
< -4 will get such a small weight that it will be effectively zero and they
won't be counted at all (or it won't make the slightest difference whether
you count them or not).  Of course when it comes to outputting reflections
you can't have a fractional reflection, it's either included or it isn't.
So then you may have to have an arbitrary cutoff, though such reflections
would likely end up with zero intensity and large SD (but the programs may
not currently be good at estimating the latter in such a situation, which
is probably why they are currently rejected).

Another point worth mentioning is that the observed distributions of E^n (E
= normalised structure amplitude) tend to be very noisy, particularly for
large n, and I have a suspicion (as yet untested) that this may come from
weak reflections which have made a full contribution to the count when it
should have been fractional (or even zero).

I'm currently working on a revised version of TRUNCATE where some or all of
the above issues will be addressed.

Cheers

-- Ian

Re: [ccp4bb] ctruncate bug?

2013-06-21 Thread Ed Pozharski


On 06/20/2013 01:07 PM, Douglas Theobald wrote:

How can there be nothing "wrong" with something that is unphysical?  
Intensities cannot be negative.


I think you are confusing two things - the true intensities and observed 
intensities.


True intensities represent the number of photons that diffract off a 
crystal in a specific direction or, for QED-minded, relative 
probabilities of a single photon being found in a particular area of the 
detector when it's probability wave function finally collapses.


True intensities certainly cannot be negative and in crystallographic 
method they never are. They are represented by the best theoretical 
estimates possible, Icalc.  These are always positive.


Observed intensities are the best estimates that we can come up with in 
an experiment.  These are determined by integrating pixels around the 
spot where particular reflection is expected to hit the detector.  
Unfortunately, science did not yet invent a method that would allow to 
suspend a crystal in vacuum while also removing all of the outside 
solvent.  Neither we have included diffuse scatter in our theoretical 
model.  Because of that, full reflection intensity contains background 
signal in addition to the Icalc.  This background has to be subtracted 
and what is perhaps the most useful form of observation is Ispot-Iback=Iobs.


These observed intensities can be negative because while their true 
underlying value is positive, random errors may result in Iback>Ispot.  
There is absolutely nothing unphysical here. Replacing Iobs with E(J) is 
not only unnecessary, it's ill-advised as it will distort intensity 
statistics.  For example, let's say you have translational NCS aligned 
with crystallographic axes, and hence some set of reflections is 
systematically absent.  If all is well, ~0 for the subset while 
 is systematically positive.  This obviously happens because the 
standard Wilson prior is wrong for these reflections, but I digress, as 
usual.


In summary, there is indeed nothing wrong, imho, with negative Iobs.  
The fact that some of these may become negative is correctly accounted 
for once sigI is factored into the ML target.


Cheers,

Ed.

--
Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
Julian, King of Lemurs

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Randy Read

r 2) is available as an option in XDSCONV
>>> Prior 1) seems to be used, or is available, in ctruncate in certain cases 
>>> (I don't know the details)
>>> 
>>> Using intensities instead of amplitudes in refinement would avoid having to 
>>> choose a prior, and refinement would therefore not be compromised in case 
>>> of data violating the assumptions underlying the prior. 
>>> 
>>> By the way, it is not (Iobs-Icalc)^2 that would be optimized in refinement 
>>> against intensities, but rather the corresponding maximum likelihood 
>>> formula (which I seem to remember is more complicated than the amplitude ML 
>>> formula, or is not an analytical formula at all, but maybe somebody knows 
>>> better).
>>> 
>>> best,
>>> 
>>> Kay
>>> 
>>> 
>>> On Thu, 20 Jun 2013 13:14:28 -0400, Douglas Theobald 
>>>  wrote:
>>> 
>>>> I still don't see how you get a negative intensity from that.  It seems 
>>>> you are saying that in many cases of a low intensity reflection, the 
>>>> integrated spot will be lower than the background.  That is not equivalent 
>>>> to having a negative measurement (as the measurement is actually positive, 
>>>> and sometimes things are randomly less positive than backgroiund).  If you 
>>>> are using a proper statistical model, after background correction you will 
>>>> end up with a positive (or 0) value for the integrated intensity.  
>>>> 
>>>> 
>>>> On Jun 20, 2013, at 1:08 PM, Andrew Leslie  
>>>> wrote:
>>>> 
>>>>> 
>>>>> The integration programs report a negative intensity simply because that 
>>>>> is the observation. 
>>>>> 
>>>>> Because of noise in the Xray background, in a large sample of intensity 
>>>>> estimates for reflections whose true intensity is very very small one 
>>>>> will inevitably get some measurements that are negative. These must not 
>>>>> be rejected because this will lead to bias (because some of these 
>>>>> intensities for symmetry mates will be estimated too large rather than 
>>>>> too small). It is not unusual for the intensity to remain negative even 
>>>>> after averaging symmetry mates.
>>>>> 
>>>>> Andrew
>>>>> 
>>>>> 
>>>>> On 20 Jun 2013, at 11:49, Douglas Theobald  wrote:
>>>>> 
>>>>>> Seems to me that the negative Is should be dealt with early on, in the 
>>>>>> integration step.  Why exactly do integration programs report negative 
>>>>>> Is to begin with?
>>>>>> 
>>>>>> 
>>>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  
>>>>>> wrote:
>>>>>> 
>>>>>>> Wouldnt be possible to take advantage of negative Is to 
>>>>>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>>>>>> plot of background scattering) to flat out the background and push all 
>>>>>>> the Is to positive values?
>>>>>>> 
>>>>>>> More of a question rather than a suggestion ...
>>>>>>> 
>>>>>>> D
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of 
>>>>>>> Ian Tickle
>>>>>>> Sent: 20 June 2013 17:34
>>>>>>> To: ccp4bb
>>>>>>> Subject: Re: [ccp4bb] ctruncate bug?
>>>>>>> 
>>>>>>> Yes higher R factors is the usual reason people don't like I-based 
>>>>>>> refinement!
>>>>>>> 
>>>>>>> Anyway, refining against Is doesn't solve the problem, it only 
>>>>>>> postpones it: you still need the Fs for maps! (though errors in Fs may 
>>>>>>> be less critical then).
>>>>>>> -- Ian
>>>>>>> 
>>>>>>> On 20 June 2013 17:20, Dale Tronrud 
>>>>>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>>>>>> If you are refining against F's you have to find some way to avoid
>>>>>>> calculating the square root of a negative number.  That is why people
>>>>>>> have historical

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Randy Read

t;> On Jun 20, 2013, at 1:08 PM, Andrew Leslie  wrote:
>> 
>>> 
>>> The integration programs report a negative intensity simply because that is 
>>> the observation. 
>>> 
>>> Because of noise in the Xray background, in a large sample of intensity 
>>> estimates for reflections whose true intensity is very very small one will 
>>> inevitably get some measurements that are negative. These must not be 
>>> rejected because this will lead to bias (because some of these intensities 
>>> for symmetry mates will be estimated too large rather than too small). It 
>>> is not unusual for the intensity to remain negative even after averaging 
>>> symmetry mates.
>>> 
>>> Andrew
>>> 
>>> 
>>> On 20 Jun 2013, at 11:49, Douglas Theobald  wrote:
>>> 
>>>> Seems to me that the negative Is should be dealt with early on, in the 
>>>> integration step.  Why exactly do integration programs report negative Is 
>>>> to begin with?
>>>> 
>>>> 
>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  
>>>> wrote:
>>>> 
>>>>> Wouldnt be possible to take advantage of negative Is to 
>>>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>>>> plot of background scattering) to flat out the background and push all 
>>>>> the Is to positive values?
>>>>> 
>>>>> More of a question rather than a suggestion ...
>>>>> 
>>>>> D
>>>>> 
>>>>> 
>>>>> 
>>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
>>>>> Tickle
>>>>> Sent: 20 June 2013 17:34
>>>>> To: ccp4bb
>>>>> Subject: Re: [ccp4bb] ctruncate bug?
>>>>> 
>>>>> Yes higher R factors is the usual reason people don't like I-based 
>>>>> refinement!
>>>>> 
>>>>> Anyway, refining against Is doesn't solve the problem, it only postpones 
>>>>> it: you still need the Fs for maps! (though errors in Fs may be less 
>>>>> critical then).
>>>>> -- Ian
>>>>> 
>>>>> On 20 June 2013 17:20, Dale Tronrud 
>>>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>>>> If you are refining against F's you have to find some way to avoid
>>>>> calculating the square root of a negative number.  That is why people
>>>>> have historically rejected negative I's and why Truncate and cTruncate
>>>>> were invented.
>>>>> 
>>>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
>>>>> care less if Iobs happens to be negative.
>>>>> 
>>>>> As for why people still refine against F...  When I was distributing
>>>>> a refinement package it could refine against I but no one wanted to do
>>>>> that.  The "R values" ended up higher, but they were looking at R
>>>>> values calculated from F's.  Of course the F based R values are lower
>>>>> when you refine against F's, that means nothing.
>>>>> 
>>>>> If we could get the PDB to report both the F and I based R values
>>>>> for all models maybe we could get a start toward moving to intensity
>>>>> refinement.
>>>>> 
>>>>> Dale Tronrud
>>>>> 
>>>>> 
>>>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>>>>> Just trying to understand the basic issues here.  How could refining 
>>>>> directly against intensities solve the fundamental problem of negative 
>>>>> intensity values?
>>>>> 
>>>>> 
>>>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
>>>>> mailto:hofkristall...@gmail.com>> wrote:
>>>>> As a maybe better alternative, we should (once again) consider to refine 
>>>>> against intensities (and I guess George Sheldrick would agree here).
>>>>> 
>>>>> I have a simple question - what exactly, short of some sort of historic 
>>>>> inertia (or memory lapse), is the reason NOT to refine against 
>>>>> intensities?
>>>>> 
>>>>> Best, BR
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> 
>>>>> This e-mail and any attachments may contain confidential, copyright and 
>>>>> or privileged material, and are for the use of the intended addressee 
>>>>> only. If you are not the intended addressee or an authorised recipient of 
>>>>> the addressee please notify us of receipt by returning the e-mail and do 
>>>>> not use, copy, retain, distribute or disclose the information in or 
>>>>> attached to the e-mail.
>>>>> 
>>>>> Any opinions expressed within this e-mail are those of the individual and 
>>>>> not necessarily of Diamond Light Source Ltd. 
>>>>> 
>>>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>>>>> attachments are free from viruses and we cannot accept liability for any 
>>>>> damage which you may sustain as a result of software viruses which may be 
>>>>> transmitted in or with the message.
>>>>> 
>>>>> Diamond Light Source Limited (company no. 4375679). Registered in England 
>>>>> and Wales with its registered office at Diamond House, Harwell Science 
>>>>> and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>

Re: [ccp4bb] ctruncate bug?

On 20 June 2013 20:46, Douglas Theobald  wrote:

> Well, I tend to think Ian is probably right, that doing things the
> "proper" way (vs French-Wilson) will not make much of a difference in the
> end.
>
> Nevertheless, I don't think refining against the (possibly negative)
> intensities is a good solution to dealing with negative intensities ---
> that just ignores the problem, and will end up overweighting large negative
> intensities.  Wouldn't it be better to correct the negative intensities
> with FW and then refine against that?
>
>
Hmmm, I seem to recall suggesting that a while back (but there were no
takers!).

I also think that using corrected Is, as opposed to corrected Fs, (however
you choose to do it) is the right way to do twinning & other statistical
tests.  For example the Padilla/Yeates L test uses the cumulative
distribution of |I1 - I2| / (I1 + I2) where I1 & I2 are intensities of
unrelated reflections (but close in reciprocal space).  The denominator of
this expression is clearly going to have problems if you feed it negative
intensities!  Also I believe (my apologies if I'm wrong!) that the UCLA
twinning server obtains the Is by squaring the Fs (presumably obtained by
F-W).  This is a formally invalid procedure (the expectation of I is not
the square of the expectation of F).  See here for an explanation of the
difference: http://xtal.sourceforge.net/man/bayest-desc.html .

Cheers

-- Ian

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Felix Frolow

 you used a derived esd since they can't 
>>>> be formally generated from the sigma's on I, and are very much 
>>>> undetermined for small intensities and small F's. 
>>>> 
>>>> Small molecule crystallographers routinely refine on F^2 and use all of 
>>>> the data, even if the F^2's are negative.
>>>> 
>>>> Bernie
>>>> 
>>>> On Jun 20, 2013, at 11:49 AM, Douglas Theobald wrote:
>>>> 
>>>>> Seems to me that the negative Is should be dealt with early on, in the 
>>>>> integration step.  Why exactly do integration programs report negative Is 
>>>>> to begin with?
>>>>> 
>>>>> 
>>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  
>>>>> wrote:
>>>>> 
>>>>>> Wouldnt be possible to take advantage of negative Is to 
>>>>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>>>>> plot of background scattering) to flat out the background and push all 
>>>>>> the Is to positive values?
>>>>>> 
>>>>>> More of a question rather than a suggestion ...
>>>>>> 
>>>>>> D
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of 
>>>>>> Ian Tickle
>>>>>> Sent: 20 June 2013 17:34
>>>>>> To: ccp4bb
>>>>>> Subject: Re: [ccp4bb] ctruncate bug?
>>>>>> 
>>>>>> Yes higher R factors is the usual reason people don't like I-based 
>>>>>> refinement!
>>>>>> 
>>>>>> Anyway, refining against Is doesn't solve the problem, it only postpones 
>>>>>> it: you still need the Fs for maps! (though errors in Fs may be less 
>>>>>> critical then).
>>>>>> -- Ian
>>>>>> 
>>>>>> On 20 June 2013 17:20, Dale Tronrud 
>>>>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>>>>> If you are refining against F's you have to find some way to avoid
>>>>>> calculating the square root of a negative number.  That is why people
>>>>>> have historically rejected negative I's and why Truncate and cTruncate
>>>>>> were invented.
>>>>>> 
>>>>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
>>>>>> care less if Iobs happens to be negative.
>>>>>> 
>>>>>> As for why people still refine against F...  When I was distributing
>>>>>> a refinement package it could refine against I but no one wanted to do
>>>>>> that.  The "R values" ended up higher, but they were looking at R
>>>>>> values calculated from F's.  Of course the F based R values are lower
>>>>>> when you refine against F's, that means nothing.
>>>>>> 
>>>>>> If we could get the PDB to report both the F and I based R values
>>>>>> for all models maybe we could get a start toward moving to intensity
>>>>>> refinement.
>>>>>> 
>>>>>> Dale Tronrud
>>>>>> 
>>>>>> 
>>>>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>>>>>> Just trying to understand the basic issues here.  How could refining 
>>>>>> directly against intensities solve the fundamental problem of negative 
>>>>>> intensity values?
>>>>>> 
>>>>>> 
>>>>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
>>>>>> mailto:hofkristall...@gmail.com>> wrote:
>>>>>> As a maybe better alternative, we should (once again) consider to refine 
>>>>>> against intensities (and I guess George Sheldrick would agree here).
>>>>>> 
>>>>>> I have a simple question - what exactly, short of some sort of historic 
>>>>>> inertia (or memory lapse), is the reason NOT to refine against 
>>>>>> intensities?
>>>>>> 
>>>>>> Best, BR
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> 
>>>>>> This e-mail and any attachments may contain confidential, copyright and 
>>>>>> or privileged material, and are for the use of the intended addressee 
>>>>>> only. If you are not the intended addressee or an authorised recipient 
>>>>>> of the addressee please notify us of receipt by returning the e-mail and 
>>>>>> do not use, copy, retain, distribute or disclose the information in or 
>>>>>> attached to the e-mail.
>>>>>> 
>>>>>> Any opinions expressed within this e-mail are those of the individual 
>>>>>> and not necessarily of Diamond Light Source Ltd. 
>>>>>> 
>>>>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>>>>>> attachments are free from viruses and we cannot accept liability for any 
>>>>>> damage which you may sustain as a result of software viruses which may 
>>>>>> be transmitted in or with the message.
>>>>>> 
>>>>>> Diamond Light Source Limited (company no. 4375679). Registered in 
>>>>>> England and Wales with its registered office at Diamond House, Harwell 
>>>>>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United 
>>>>>> Kingdom
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Tim Gruene

ils)
>>>> 
>>>> Using intensities instead of amplitudes in refinement would
>>>> avoid having to choose a prior, and refinement would
>>>> therefore not be compromised in case of data violating the
>>>> assumptions underlying the prior.
>>>> 
>>>> By the way, it is not (Iobs-Icalc)^2 that would be optimized
>>>> in refinement against intensities, but rather the
>>>> corresponding maximum likelihood formula (which I seem to
>>>> remember is more complicated than the amplitude ML formula,
>>>> or is not an analytical formula at all, but maybe somebody
>>>> knows better).
>>>> 
>>>> best,
>>>> 
>>>> Kay
>>>> 
>>>> 
>>>> On Thu, 20 Jun 2013 13:14:28 -0400, Douglas Theobald
>>>>  wrote:
>>>> 
>>>>> I still don't see how you get a negative intensity from
>>>>> that.  It seems you are saying that in many cases of a low
>>>>> intensity reflection, the integrated spot will be lower
>>>>> than the background.  That is not equivalent to having a
>>>>> negative measurement (as the measurement is actually
>>>>> positive, and sometimes things are randomly less positive
>>>>> than backgroiund).  If you are using a proper statistical
>>>>> model, after background correction you will end up with a
>>>>> positive (or 0) value for the integrated intensity.
>>>>> 
>>>>> 
>>>>> On Jun 20, 2013, at 1:08 PM, Andrew Leslie
>>>>>  wrote:
>>>>> 
>>>>>> 
>>>>>> The integration programs report a negative intensity
>>>>>> simply because that is the observation.
>>>>>> 
>>>>>> Because of noise in the Xray background, in a large
>>>>>> sample of intensity estimates for reflections whose true
>>>>>> intensity is very very small one will inevitably get some
>>>>>> measurements that are negative. These must not be
>>>>>> rejected because this will lead to bias (because some of
>>>>>> these intensities for symmetry mates will be estimated
>>>>>> too large rather than too small). It is not unusual for
>>>>>> the intensity to remain negative even after averaging
>>>>>> symmetry mates.
>>>>>> 
>>>>>> Andrew
>>>>>> 
>>>>>> 
>>>>>> On 20 Jun 2013, at 11:49, Douglas Theobald
>>>>>>  wrote:
>>>>>> 
>>>>>>> Seems to me that the negative Is should be dealt with
>>>>>>> early on, in the integration step.  Why exactly do
>>>>>>> integration programs report negative Is to begin with?
>>>>>>> 
>>>>>>> 
>>>>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini
>>>>>>>  wrote:
>>>>>>> 
>>>>>>>> Wouldnt be possible to take advantage of negative Is
>>>>>>>> to extrapolate/estimate the decay of scattering
>>>>>>>> background (kind of Wilson plot of background
>>>>>>>> scattering) to flat out the background and push all
>>>>>>>> the Is to positive values?
>>>>>>>> 
>>>>>>>> More of a question rather than a suggestion ...
>>>>>>>> 
>>>>>>>> D
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> From: CCP4 bulletin board
>>>>>>>> [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian
>>>>>>>> Tickle Sent: 20 June 2013 17:34 To: ccp4bb Subject:
>>>>>>>> Re: [ccp4bb] ctruncate bug?
>>>>>>>> 
>>>>>>>> Yes higher R factors is the usual reason people don't
>>>>>>>> like I-based refinement!
>>>>>>>> 
>>>>>>>> Anyway, refining against Is doesn't solve the
>>>>>>>> problem, it only postpones it: you still need the Fs
>>>>>>>> for maps! (though errors in Fs may be less critical
>>>>>>>> then). -- Ian
>>>>>>>> 
>>>>>>>> On 20 June 2013 17:20, Dale Tronrud
>>>>>>>> mailt

Re: [ccp4bb] ctruncate bug?

>>> On Thu, 20 Jun 2013 13:14:28 -0400, Douglas Theobald 
>>>  wrote:
>>> 
>>>> I still don't see how you get a negative intensity from that.  It seems 
>>>> you are saying that in many cases of a low intensity reflection, the 
>>>> integrated spot will be lower than the background.  That is not equivalent 
>>>> to having a negative measurement (as the measurement is actually positive, 
>>>> and sometimes things are randomly less positive than backgroiund).  If you 
>>>> are using a proper statistical model, after background correction you will 
>>>> end up with a positive (or 0) value for the integrated intensity.
>>>> 
>>>> 
>>>> On Jun 20, 2013, at 1:08 PM, Andrew Leslie  
>>>> wrote:
>>>> 
>>>>> 
>>>>> The integration programs report a negative intensity simply because that 
>>>>> is the observation.
>>>>> 
>>>>> Because of noise in the Xray background, in a large sample of intensity 
>>>>> estimates for reflections whose true intensity is very very small one 
>>>>> will inevitably get some measurements that are negative. These must not 
>>>>> be rejected because this will lead to bias (because some of these 
>>>>> intensities for symmetry mates will be estimated too large rather than 
>>>>> too small). It is not unusual for the intensity to remain negative even 
>>>>> after averaging symmetry mates.
>>>>> 
>>>>> Andrew
>>>>> 
>>>>> 
>>>>> On 20 Jun 2013, at 11:49, Douglas Theobald  wrote:
>>>>> 
>>>>>> Seems to me that the negative Is should be dealt with early on, in the 
>>>>>> integration step.  Why exactly do integration programs report negative 
>>>>>> Is to begin with?
>>>>>> 
>>>>>> 
>>>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  
>>>>>> wrote:
>>>>>> 
>>>>>>> Wouldnt be possible to take advantage of negative Is to 
>>>>>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>>>>>> plot of background scattering) to flat out the background and push all 
>>>>>>> the Is to positive values?
>>>>>>> 
>>>>>>> More of a question rather than a suggestion ...
>>>>>>> 
>>>>>>> D
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of 
>>>>>>> Ian Tickle
>>>>>>> Sent: 20 June 2013 17:34
>>>>>>> To: ccp4bb
>>>>>>> Subject: Re: [ccp4bb] ctruncate bug?
>>>>>>> 
>>>>>>> Yes higher R factors is the usual reason people don't like I-based 
>>>>>>> refinement!
>>>>>>> 
>>>>>>> Anyway, refining against Is doesn't solve the problem, it only 
>>>>>>> postpones it: you still need the Fs for maps! (though errors in Fs may 
>>>>>>> be less critical then).
>>>>>>> -- Ian
>>>>>>> 
>>>>>>> On 20 June 2013 17:20, Dale Tronrud 
>>>>>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>>>>>> If you are refining against F's you have to find some way to avoid
>>>>>>> calculating the square root of a negative number.  That is why people
>>>>>>> have historically rejected negative I's and why Truncate and cTruncate
>>>>>>> were invented.
>>>>>>> 
>>>>>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
>>>>>>> care less if Iobs happens to be negative.
>>>>>>> 
>>>>>>> As for why people still refine against F...  When I was distributing
>>>>>>> a refinement package it could refine against I but no one wanted to do
>>>>>>> that.  The "R values" ended up higher, but they were looking at R
>>>>>>> values calculated from F's.  Of course the F based R values are lower
>>>>>>> when you refine against F's, that means nothing.
>>>>>>> 
>>>>>>> If we could get the PDB to report both the F and I based R values
>>>>>>> for all models maybe we could get a start toward moving to intensity
>>>>>>> refinement.
>>>>>>> 
>>>>>>> Dale Tronrud
>>>>>>> 
>>>>>>> 
>>>>>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>>>>>>> Just trying to understand the basic issues here.  How could refining 
>>>>>>> directly against intensities solve the fundamental problem of negative 
>>>>>>> intensity values?
>>>>>>> 
>>>>>>> 
>>>>>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
>>>>>>> mailto:hofkristall...@gmail.com>> wrote:
>>>>>>> As a maybe better alternative, we should (once again) consider to 
>>>>>>> refine against intensities (and I guess George Sheldrick would agree 
>>>>>>> here).
>>>>>>> 
>>>>>>> I have a simple question - what exactly, short of some sort of historic 
>>>>>>> inertia (or memory lapse), is the reason NOT to refine against 
>>>>>>> intensities?
>>>>>>> 
>>>>>>> Best, BR
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> 
>>>>>>> This e-mail and any attachments may contain confidential, copyright and 
>>>>>>> or privileged material, and are for the use of the intended addressee 
>>>>>>> only. If you are not the intended addressee or an authorised recipient 
>>>>>>> of the addressee please notify us of receipt by returning the e-mail 
>>>>>>> and do not use, copy, retain, distribute or disclose the information in 
>>>>>>> or attached to the e-mail.
>>>>>>> 
>>>>>>> Any opinions expressed within this e-mail are those of the individual 
>>>>>>> and not necessarily of Diamond Light Source Ltd.
>>>>>>> 
>>>>>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>>>>>>> attachments are free from viruses and we cannot accept liability for 
>>>>>>> any damage which you may sustain as a result of software viruses which 
>>>>>>> may be transmitted in or with the message.
>>>>>>> 
>>>>>>> Diamond Light Source Limited (company no. 4375679). Registered in 
>>>>>>> England and Wales with its registered office at Diamond House, Harwell 
>>>>>>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United 
>>>>>>> Kingdom
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Kay Diederichs

ms report negative Is to 
begin with?


On Jun 20, 2013, at 12:45 PM, Dom Bellini  wrote:


Wouldnt be possible to take advantage of negative Is to extrapolate/estimate 
the decay of scattering background (kind of Wilson plot of background 
scattering) to flat out the background and push all the Is to positive values?

More of a question rather than a suggestion ...

D



From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian Tickle
Sent: 20 June 2013 17:34
To: ccp4bb
Subject: Re: [ccp4bb] ctruncate bug?

Yes higher R factors is the usual reason people don't like I-based refinement!

Anyway, refining against Is doesn't solve the problem, it only postpones it: 
you still need the Fs for maps! (though errors in Fs may be less critical then).
-- Ian

On 20 June 2013 17:20, Dale Tronrud 
mailto:det...@uoxray.uoregon.edu>> wrote:
If you are refining against F's you have to find some way to avoid
calculating the square root of a negative number.  That is why people
have historically rejected negative I's and why Truncate and cTruncate
were invented.

When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
care less if Iobs happens to be negative.

As for why people still refine against F...  When I was distributing
a refinement package it could refine against I but no one wanted to do
that.  The "R values" ended up higher, but they were looking at R
values calculated from F's.  Of course the F based R values are lower
when you refine against F's, that means nothing.

If we could get the PDB to report both the F and I based R values
for all models maybe we could get a start toward moving to intensity
refinement.

Dale Tronrud


On 06/20/2013 09:06 AM, Douglas Theobald wrote:
Just trying to understand the basic issues here.  How could refining directly 
against intensities solve the fundamental problem of negative intensity values?


On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
mailto:hofkristall...@gmail.com>> wrote:
As a maybe better alternative, we should (once again) consider to refine 
against intensities (and I guess George Sheldrick would agree here).

I have a simple question - what exactly, short of some sort of historic inertia 
(or memory lapse), is the reason NOT to refine against intensities?

Best, BR




--

This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.

Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd.

Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.

Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Re: [ccp4bb] ctruncate bug?

t;> On 20 Jun 2013, at 11:49, Douglas Theobald  wrote:
>>> 
>>>> Seems to me that the negative Is should be dealt with early on, in the 
>>>> integration step.  Why exactly do integration programs report negative Is 
>>>> to begin with?
>>>> 
>>>> 
>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  
>>>> wrote:
>>>> 
>>>>> Wouldnt be possible to take advantage of negative Is to 
>>>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>>>> plot of background scattering) to flat out the background and push all 
>>>>> the Is to positive values?
>>>>> 
>>>>> More of a question rather than a suggestion ...
>>>>> 
>>>>> D
>>>>> 
>>>>> 
>>>>> 
>>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
>>>>> Tickle
>>>>> Sent: 20 June 2013 17:34
>>>>> To: ccp4bb
>>>>> Subject: Re: [ccp4bb] ctruncate bug?
>>>>> 
>>>>> Yes higher R factors is the usual reason people don't like I-based 
>>>>> refinement!
>>>>> 
>>>>> Anyway, refining against Is doesn't solve the problem, it only postpones 
>>>>> it: you still need the Fs for maps! (though errors in Fs may be less 
>>>>> critical then).
>>>>> -- Ian
>>>>> 
>>>>> On 20 June 2013 17:20, Dale Tronrud 
>>>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>>>> If you are refining against F's you have to find some way to avoid
>>>>> calculating the square root of a negative number.  That is why people
>>>>> have historically rejected negative I's and why Truncate and cTruncate
>>>>> were invented.
>>>>> 
>>>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
>>>>> care less if Iobs happens to be negative.
>>>>> 
>>>>> As for why people still refine against F...  When I was distributing
>>>>> a refinement package it could refine against I but no one wanted to do
>>>>> that.  The "R values" ended up higher, but they were looking at R
>>>>> values calculated from F's.  Of course the F based R values are lower
>>>>> when you refine against F's, that means nothing.
>>>>> 
>>>>> If we could get the PDB to report both the F and I based R values
>>>>> for all models maybe we could get a start toward moving to intensity
>>>>> refinement.
>>>>> 
>>>>> Dale Tronrud
>>>>> 
>>>>> 
>>>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>>>>> Just trying to understand the basic issues here.  How could refining 
>>>>> directly against intensities solve the fundamental problem of negative 
>>>>> intensity values?
>>>>> 
>>>>> 
>>>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
>>>>> mailto:hofkristall...@gmail.com>> wrote:
>>>>> As a maybe better alternative, we should (once again) consider to refine 
>>>>> against intensities (and I guess George Sheldrick would agree here).
>>>>> 
>>>>> I have a simple question - what exactly, short of some sort of historic 
>>>>> inertia (or memory lapse), is the reason NOT to refine against 
>>>>> intensities?
>>>>> 
>>>>> Best, BR
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> 
>>>>> This e-mail and any attachments may contain confidential, copyright and 
>>>>> or privileged material, and are for the use of the intended addressee 
>>>>> only. If you are not the intended addressee or an authorised recipient of 
>>>>> the addressee please notify us of receipt by returning the e-mail and do 
>>>>> not use, copy, retain, distribute or disclose the information in or 
>>>>> attached to the e-mail.
>>>>> 
>>>>> Any opinions expressed within this e-mail are those of the individual and 
>>>>> not necessarily of Diamond Light Source Ltd. 
>>>>> 
>>>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>>>>> attachments are free from viruses and we cannot accept liability for any 
>>>>> damage which you may sustain as a result of software viruses which may be 
>>>>> transmitted in or with the message.
>>>>> 
>>>>> Diamond Light Source Limited (company no. 4375679). Registered in England 
>>>>> and Wales with its registered office at Diamond House, Harwell Science 
>>>>> and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Kay Diederichs

Douglas,

the intensity is negative if the integrated spot has a lower intensity than the 
estimate of the background under the spot. So yes, we are not _measuring_ 
negative intensities, rather we are estimating intensities, and that estimate 
may turn out to be negative. In a later step we try to "correct" for this, 
because it is non-physical, as you say. At that point, the "proper statistical 
model" comes into play. Essentially we use this as a "prior". In the order of 
increasing information, we can have more or less informative priors for weak 
reflections:
1) I > 0
2) I has a distribution looking like the right half of a Gaussian, and we 
estimate its width from the variance of the intensities in a resolution shell
3) I follows a Wilson distribution, and we estimate its parameters from the 
data in a resolution shell
4) I must be related to Fcalc^2 (i.e. once the structure is solved, we 
re-integrate using the Fcalc as prior)
For a given experiment, the problem is chicken-and-egg in the sense that only 
if you know the characteristics of the data can you choose the correct prior.
I guess that using prior 4) would be heavily frowned upon because there is a 
danger of model bias. You could say: A Bayesian analysis done properly should 
not suffer from model bias. This is probably true, but the theory to ensure the 
word "properly" is not available at the moment.
Crystallographers usually use prior 3) which, as I tried to point out, also has 
its weak spots, namely if the data do not behave like those of an ideal crystal 
- and today's projects often result in data that would have been discarded ten 
years ago, so they are far from ideal.
Prior 2) is available as an option in XDSCONV
Prior 1) seems to be used, or is available, in ctruncate in certain cases (I 
don't know the details)

Using intensities instead of amplitudes in refinement would avoid having to 
choose a prior, and refinement would therefore not be compromised in case of 
data violating the assumptions underlying the prior. 

By the way, it is not (Iobs-Icalc)^2 that would be optimized in refinement 
against intensities, but rather the corresponding maximum likelihood formula 
(which I seem to remember is more complicated than the amplitude ML formula, or 
is not an analytical formula at all, but maybe somebody knows better).

best,

Kay


On Thu, 20 Jun 2013 13:14:28 -0400, Douglas Theobald  
wrote:

>I still don't see how you get a negative intensity from that.  It seems you 
>are saying that in many cases of a low intensity reflection, the integrated 
>spot will be lower than the background.  That is not equivalent to having a 
>negative measurement (as the measurement is actually positive, and sometimes 
>things are randomly less positive than backgroiund).  If you are using a 
>proper statistical model, after background correction you will end up with a 
>positive (or 0) value for the integrated intensity.  
>
>
>On Jun 20, 2013, at 1:08 PM, Andrew Leslie  wrote:
>
>> 
>> The integration programs report a negative intensity simply because that is 
>> the observation. 
>> 
>> Because of noise in the Xray background, in a large sample of intensity 
>> estimates for reflections whose true intensity is very very small one will 
>> inevitably get some measurements that are negative. These must not be 
>> rejected because this will lead to bias (because some of these intensities 
>> for symmetry mates will be estimated too large rather than too small). It is 
>> not unusual for the intensity to remain negative even after averaging 
>> symmetry mates.
>> 
>> Andrew
>> 
>> 
>> On 20 Jun 2013, at 11:49, Douglas Theobald  wrote:
>> 
>>> Seems to me that the negative Is should be dealt with early on, in the 
>>> integration step.  Why exactly do integration programs report negative Is 
>>> to begin with?
>>> 
>>> 
>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  wrote:
>>> 
>>>> Wouldnt be possible to take advantage of negative Is to 
>>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>>> plot of background scattering) to flat out the background and push all the 
>>>> Is to positive values?
>>>> 
>>>> More of a question rather than a suggestion ...
>>>> 
>>>> D
>>>> 
>>>> 
>>>> 
>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
>>>> Tickle
>>>> Sent: 20 June 2013 17:34
>>>> To: ccp4bb
>>>> Subject: Re: [ccp4bb] ctruncate bug?
>>>> 
>>>> Yes higher R factors is the usual reason people don't like I-based 
>>>>

Re: [ccp4bb] ctruncate bug?

Douglas, I think you are missing the point that estimation of the
parameters of the proper Bayesian statistical model (i.e. the Wilson prior)
in order to perform the integration in the manner you are suggesting
requires knowledge of the already integrated intensities!  I suppose we
could iterate, i.e. assume an approximate prior, integrate, calculate a
better prior, re-do the integration with the new prior and so on (hoping of
course that the whole process converges), but I think most people would
regard that as overkill.  Also dealing with the issue of averaging
estimates of intensities that no longer have a Gaussian error distribution,
and also crucially outlier rejection, would require some rethinking of the
algorithms. The question is would it make any difference in the end
compared with the 'post-correction' we're doing now?

Cheers

-- Ian


On 20 June 2013 18:14, Douglas Theobald  wrote:

> I still don't see how you get a negative intensity from that.  It seems
> you are saying that in many cases of a low intensity reflection, the
> integrated spot will be lower than the background.  That is not equivalent
> to having a negative measurement (as the measurement is actually positive,
> and sometimes things are randomly less positive than backgroiund).  If you
> are using a proper statistical model, after background correction you will
> end up with a positive (or 0) value for the integrated intensity.
>
>
> On Jun 20, 2013, at 1:08 PM, Andrew Leslie 
> wrote:
>
> >
> > The integration programs report a negative intensity simply because that
> is the observation.
> >
> > Because of noise in the Xray background, in a large sample of intensity
> estimates for reflections whose true intensity is very very small one will
> inevitably get some measurements that are negative. These must not be
> rejected because this will lead to bias (because some of these intensities
> for symmetry mates will be estimated too large rather than too small). It
> is not unusual for the intensity to remain negative even after averaging
> symmetry mates.
> >
> > Andrew
> >
> >
> > On 20 Jun 2013, at 11:49, Douglas Theobald 
> wrote:
> >
> >> Seems to me that the negative Is should be dealt with early on, in the
> integration step.  Why exactly do integration programs report negative Is
> to begin with?
> >>
> >>
> >> On Jun 20, 2013, at 12:45 PM, Dom Bellini 
> wrote:
> >>
> >>> Wouldnt be possible to take advantage of negative Is to
> extrapolate/estimate the decay of scattering background (kind of Wilson
> plot of background scattering) to flat out the background and push all the
> Is to positive values?
> >>>
> >>> More of a question rather than a suggestion ...
> >>>
> >>> D
> >>>
> >>>
> >>>
> >>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
> Ian Tickle
> >>> Sent: 20 June 2013 17:34
> >>> To: ccp4bb
> >>> Subject: Re: [ccp4bb] ctruncate bug?
> >>>
> >>> Yes higher R factors is the usual reason people don't like I-based
> refinement!
> >>>
> >>> Anyway, refining against Is doesn't solve the problem, it only
> postpones it: you still need the Fs for maps! (though errors in Fs may be
> less critical then).
> >>> -- Ian
> >>>
> >>> On 20 June 2013 17:20, Dale Tronrud  det...@uoxray.uoregon.edu>> wrote:
> >>> If you are refining against F's you have to find some way to avoid
> >>> calculating the square root of a negative number.  That is why people
> >>> have historically rejected negative I's and why Truncate and cTruncate
> >>> were invented.
> >>>
> >>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
> >>> care less if Iobs happens to be negative.
> >>>
> >>> As for why people still refine against F...  When I was distributing
> >>> a refinement package it could refine against I but no one wanted to do
> >>> that.  The "R values" ended up higher, but they were looking at R
> >>> values calculated from F's.  Of course the F based R values are lower
> >>> when you refine against F's, that means nothing.
> >>>
> >>> If we could get the PDB to report both the F and I based R values
> >>> for all models maybe we could get a start toward moving to intensity
> >>> refinement.
> >>>
> >>> Dale Tronrud
> >>>
> >>>
> >>> On 06/20/2013 09:06

Re: [ccp4bb] ctruncate bug?

On Jun 20, 2013, at 1:47 PM, Felix Frolow  wrote:

> Intensity is subtraction:  Inet=Iobs - Ibackground.  Iobs and Ibackground can 
> not be negative.  Inet CAN be negative if background is higher than Iobs. 

Just to reiterate, we know that the true value of Inet cannot be negative.  
Hence, the equation you quote is invalid and illogical --- it has no physical 
or statistical justification (except as an approximation for large Iobs and low 
Iback, when ironically background correction is unnecessary).  That equation 
does not account for random statistical fluctuations (e.g., simple Poisson 
counting statistics of shot noise).  


> We do not know how to model background scattering modulated my molecular 
> transform and mechanical motion of the molecule, 
> I recall we have called it TDS - thermal diffuse scattering. Many years ago 
> Boaz Shaanan and JH were fascinated by it.
> If we would know how deal with TDS, we would go to much nicer structures some 
> of us like and for sure to much lower 
> R factors all of us love excluding maybe referees who will claim over 
> refinement :-\
> Dr Felix Frolow   
> Professor of Structural Biology and Biotechnology, 
> Department of Molecular Microbiology and Biotechnology
> Tel Aviv University 69978, Israel
> 
> Acta Crystallographica F, co-editor
> 
> e-mail: mbfro...@post.tau.ac.il
> Tel:  ++972-3640-8723
> Fax: ++972-3640-9407
> Cellular: 0547 459 608
> 
> On Jun 20, 2013, at 20:07 , Douglas Theobald  wrote:
> 
>> How can there be nothing "wrong" with something that is unphysical?  
>> Intensities cannot be negative.  How could you measure a negative number of 
>> photons?  You can only have a Gaussian distribution around I=0 if you are 
>> using an incorrect, unphysical statistical model.  As I understand it, the 
>> physics predicts that intensities from diffraction should be gamma 
>> distributed (i.e., the square of a Gaussian variate), which makes sense as 
>> the gamma distribution assigns probability 0 to negative values.  
>> 
>> 
>> On Jun 20, 2013, at 1:00 PM, Bernard D Santarsiero  wrote:
>> 
>>> There's absolutely nothing "wrong" with negative intensities. They are 
>>> measurements of intensities that are near zero, and some will be negative, 
>>> and others positive.  The distribution around I=0 can still be Gaussian, 
>>> and you have true esd's.  With F's you used a derived esd since they can't 
>>> be formally generated from the sigma's on I, and are very much undetermined 
>>> for small intensities and small F's. 
>>> 
>>> Small molecule crystallographers routinely refine on F^2 and use all of the 
>>> data, even if the F^2's are negative.
>>> 
>>> Bernie
>>> 
>>> On Jun 20, 2013, at 11:49 AM, Douglas Theobald wrote:
>>> 
>>>> Seems to me that the negative Is should be dealt with early on, in the 
>>>> integration step.  Why exactly do integration programs report negative Is 
>>>> to begin with?
>>>> 
>>>> 
>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  
>>>> wrote:
>>>> 
>>>>> Wouldnt be possible to take advantage of negative Is to 
>>>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>>>> plot of background scattering) to flat out the background and push all 
>>>>> the Is to positive values?
>>>>> 
>>>>> More of a question rather than a suggestion ...
>>>>> 
>>>>> D
>>>>> 
>>>>> 
>>>>> 
>>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
>>>>> Tickle
>>>>> Sent: 20 June 2013 17:34
>>>>> To: ccp4bb
>>>>> Subject: Re: [ccp4bb] ctruncate bug?
>>>>> 
>>>>> Yes higher R factors is the usual reason people don't like I-based 
>>>>> refinement!
>>>>> 
>>>>> Anyway, refining against Is doesn't solve the problem, it only postpones 
>>>>> it: you still need the Fs for maps! (though errors in Fs may be less 
>>>>> critical then).
>>>>> -- Ian
>>>>> 
>>>>> On 20 June 2013 17:20, Dale Tronrud 
>>>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>>>> If you are refining against F's you have to find some way to avoid
>>>>> calculating the square root of a negative number.  That is why people
>>>>> have historically rejected neg

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Felix Frolow

Intensity is subtraction:  Inet=Iobs - Ibackground.  Iobs and Ibackground can 
not be negative.  Inet CAN be negative if background is higher than Iobs. 
We do not know how to model background scattering modulated my molecular 
transform and mechanical motion of the molecule, 
I recall we have called it TDS - thermal diffuse scattering. Many years ago 
Boaz Shaanan and JH were fascinated by it.
If we would know how deal with TDS, we would go to much nicer structures some 
of us like and for sure to much lower 
R factors all of us love excluding maybe referees who will claim over 
refinement :-\
Dr Felix Frolow   
Professor of Structural Biology and Biotechnology, 
Department of Molecular Microbiology and Biotechnology
Tel Aviv University 69978, Israel

Acta Crystallographica F, co-editor

e-mail: mbfro...@post.tau.ac.il
Tel:  ++972-3640-8723
Fax: ++972-3640-9407
Cellular: 0547 459 608

On Jun 20, 2013, at 20:07 , Douglas Theobald  wrote:

> How can there be nothing "wrong" with something that is unphysical?  
> Intensities cannot be negative.  How could you measure a negative number of 
> photons?  You can only have a Gaussian distribution around I=0 if you are 
> using an incorrect, unphysical statistical model.  As I understand it, the 
> physics predicts that intensities from diffraction should be gamma 
> distributed (i.e., the square of a Gaussian variate), which makes sense as 
> the gamma distribution assigns probability 0 to negative values.  
> 
> 
> On Jun 20, 2013, at 1:00 PM, Bernard D Santarsiero  wrote:
> 
>> There's absolutely nothing "wrong" with negative intensities. They are 
>> measurements of intensities that are near zero, and some will be negative, 
>> and others positive.  The distribution around I=0 can still be Gaussian, and 
>> you have true esd's.  With F's you used a derived esd since they can't be 
>> formally generated from the sigma's on I, and are very much undetermined for 
>> small intensities and small F's. 
>> 
>> Small molecule crystallographers routinely refine on F^2 and use all of the 
>> data, even if the F^2's are negative.
>> 
>> Bernie
>> 
>> On Jun 20, 2013, at 11:49 AM, Douglas Theobald wrote:
>> 
>>> Seems to me that the negative Is should be dealt with early on, in the 
>>> integration step.  Why exactly do integration programs report negative Is 
>>> to begin with?
>>> 
>>> 
>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  wrote:
>>> 
>>>> Wouldnt be possible to take advantage of negative Is to 
>>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>>> plot of background scattering) to flat out the background and push all the 
>>>> Is to positive values?
>>>> 
>>>> More of a question rather than a suggestion ...
>>>> 
>>>> D
>>>> 
>>>> 
>>>> 
>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
>>>> Tickle
>>>> Sent: 20 June 2013 17:34
>>>> To: ccp4bb
>>>> Subject: Re: [ccp4bb] ctruncate bug?
>>>> 
>>>> Yes higher R factors is the usual reason people don't like I-based 
>>>> refinement!
>>>> 
>>>> Anyway, refining against Is doesn't solve the problem, it only postpones 
>>>> it: you still need the Fs for maps! (though errors in Fs may be less 
>>>> critical then).
>>>> -- Ian
>>>> 
>>>> On 20 June 2013 17:20, Dale Tronrud 
>>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>>> If you are refining against F's you have to find some way to avoid
>>>> calculating the square root of a negative number.  That is why people
>>>> have historically rejected negative I's and why Truncate and cTruncate
>>>> were invented.
>>>> 
>>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
>>>> care less if Iobs happens to be negative.
>>>> 
>>>> As for why people still refine against F...  When I was distributing
>>>> a refinement package it could refine against I but no one wanted to do
>>>> that.  The "R values" ended up higher, but they were looking at R
>>>> values calculated from F's.  Of course the F based R values are lower
>>>> when you refine against F's, that means nothing.
>>>> 
>>>> If we could get the PDB to report both the F and I based R values
>>>> for all models maybe we could get a start toward moving t

Re: [ccp4bb] ctruncate bug?

I still don't see how you get a negative intensity from that.  It seems you are 
saying that in many cases of a low intensity reflection, the integrated spot 
will be lower than the background.  That is not equivalent to having a negative 
measurement (as the measurement is actually positive, and sometimes things are 
randomly less positive than backgroiund).  If you are using a proper 
statistical model, after background correction you will end up with a positive 
(or 0) value for the integrated intensity.  


On Jun 20, 2013, at 1:08 PM, Andrew Leslie  wrote:

> 
> The integration programs report a negative intensity simply because that is 
> the observation. 
> 
> Because of noise in the Xray background, in a large sample of intensity 
> estimates for reflections whose true intensity is very very small one will 
> inevitably get some measurements that are negative. These must not be 
> rejected because this will lead to bias (because some of these intensities 
> for symmetry mates will be estimated too large rather than too small). It is 
> not unusual for the intensity to remain negative even after averaging 
> symmetry mates.
> 
> Andrew
> 
> 
> On 20 Jun 2013, at 11:49, Douglas Theobald  wrote:
> 
>> Seems to me that the negative Is should be dealt with early on, in the 
>> integration step.  Why exactly do integration programs report negative Is to 
>> begin with?
>> 
>> 
>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  wrote:
>> 
>>> Wouldnt be possible to take advantage of negative Is to 
>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>> plot of background scattering) to flat out the background and push all the 
>>> Is to positive values?
>>> 
>>> More of a question rather than a suggestion ...
>>> 
>>> D
>>> 
>>> 
>>> 
>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
>>> Tickle
>>> Sent: 20 June 2013 17:34
>>> To: ccp4bb
>>> Subject: Re: [ccp4bb] ctruncate bug?
>>> 
>>> Yes higher R factors is the usual reason people don't like I-based 
>>> refinement!
>>> 
>>> Anyway, refining against Is doesn't solve the problem, it only postpones 
>>> it: you still need the Fs for maps! (though errors in Fs may be less 
>>> critical then).
>>> -- Ian
>>> 
>>> On 20 June 2013 17:20, Dale Tronrud 
>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>> If you are refining against F's you have to find some way to avoid
>>> calculating the square root of a negative number.  That is why people
>>> have historically rejected negative I's and why Truncate and cTruncate
>>> were invented.
>>> 
>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
>>> care less if Iobs happens to be negative.
>>> 
>>> As for why people still refine against F...  When I was distributing
>>> a refinement package it could refine against I but no one wanted to do
>>> that.  The "R values" ended up higher, but they were looking at R
>>> values calculated from F's.  Of course the F based R values are lower
>>> when you refine against F's, that means nothing.
>>> 
>>> If we could get the PDB to report both the F and I based R values
>>> for all models maybe we could get a start toward moving to intensity
>>> refinement.
>>> 
>>> Dale Tronrud
>>> 
>>> 
>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>>> Just trying to understand the basic issues here.  How could refining 
>>> directly against intensities solve the fundamental problem of negative 
>>> intensity values?
>>> 
>>> 
>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
>>> mailto:hofkristall...@gmail.com>> wrote:
>>> As a maybe better alternative, we should (once again) consider to refine 
>>> against intensities (and I guess George Sheldrick would agree here).
>>> 
>>> I have a simple question - what exactly, short of some sort of historic 
>>> inertia (or memory lapse), is the reason NOT to refine against intensities?
>>> 
>>> Best, BR
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> 
>>> This e-mail and any attachments may contain confidential, copyright and or 
>>> privileged material, and are for the use of the intended addressee only. If 
>>> you are not the intended addressee or an authorised recipient of the 
>>> addressee please notify us of receipt by returning the e-mail and do not 
>>> use, copy, retain, distribute or disclose the information in or attached to 
>>> the e-mail.
>>> 
>>> Any opinions expressed within this e-mail are those of the individual and 
>>> not necessarily of Diamond Light Source Ltd. 
>>> 
>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>>> attachments are free from viruses and we cannot accept liability for any 
>>> damage which you may sustain as a result of software viruses which may be 
>>> transmitted in or with the message.
>>> 
>>> Diamond Light Source Limited (company no. 4375679). Registered in England 
>>> and Wales with its registered office at Diamond House, Harwell Science and 
>>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>

Re: [ccp4bb] ctruncate bug?

How can there be nothing "wrong" with something that is unphysical?  
Intensities cannot be negative.  How could you measure a negative number of 
photons?  You can only have a Gaussian distribution around I=0 if you are using 
an incorrect, unphysical statistical model.  As I understand it, the physics 
predicts that intensities from diffraction should be gamma distributed (i.e., 
the square of a Gaussian variate), which makes sense as the gamma distribution 
assigns probability 0 to negative values.  


On Jun 20, 2013, at 1:00 PM, Bernard D Santarsiero  wrote:

> There's absolutely nothing "wrong" with negative intensities. They are 
> measurements of intensities that are near zero, and some will be negative, 
> and others positive.  The distribution around I=0 can still be Gaussian, and 
> you have true esd's.  With F's you used a derived esd since they can't be 
> formally generated from the sigma's on I, and are very much undetermined for 
> small intensities and small F's. 
> 
> Small molecule crystallographers routinely refine on F^2 and use all of the 
> data, even if the F^2's are negative.
> 
> Bernie
> 
> On Jun 20, 2013, at 11:49 AM, Douglas Theobald wrote:
> 
>> Seems to me that the negative Is should be dealt with early on, in the 
>> integration step.  Why exactly do integration programs report negative Is to 
>> begin with?
>> 
>> 
>> On Jun 20, 2013, at 12:45 PM, Dom Bellini  wrote:
>> 
>>> Wouldnt be possible to take advantage of negative Is to 
>>> extrapolate/estimate the decay of scattering background (kind of Wilson 
>>> plot of background scattering) to flat out the background and push all the 
>>> Is to positive values?
>>> 
>>> More of a question rather than a suggestion ...
>>> 
>>> D
>>> 
>>> 
>>> 
>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
>>> Tickle
>>> Sent: 20 June 2013 17:34
>>> To: ccp4bb
>>> Subject: Re: [ccp4bb] ctruncate bug?
>>> 
>>> Yes higher R factors is the usual reason people don't like I-based 
>>> refinement!
>>> 
>>> Anyway, refining against Is doesn't solve the problem, it only postpones 
>>> it: you still need the Fs for maps! (though errors in Fs may be less 
>>> critical then).
>>> -- Ian
>>> 
>>> On 20 June 2013 17:20, Dale Tronrud 
>>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>> If you are refining against F's you have to find some way to avoid
>>> calculating the square root of a negative number.  That is why people
>>> have historically rejected negative I's and why Truncate and cTruncate
>>> were invented.
>>> 
>>> When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
>>> care less if Iobs happens to be negative.
>>> 
>>> As for why people still refine against F...  When I was distributing
>>> a refinement package it could refine against I but no one wanted to do
>>> that.  The "R values" ended up higher, but they were looking at R
>>> values calculated from F's.  Of course the F based R values are lower
>>> when you refine against F's, that means nothing.
>>> 
>>> If we could get the PDB to report both the F and I based R values
>>> for all models maybe we could get a start toward moving to intensity
>>> refinement.
>>> 
>>> Dale Tronrud
>>> 
>>> 
>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>>> Just trying to understand the basic issues here.  How could refining 
>>> directly against intensities solve the fundamental problem of negative 
>>> intensity values?
>>> 
>>> 
>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
>>> mailto:hofkristall...@gmail.com>> wrote:
>>> As a maybe better alternative, we should (once again) consider to refine 
>>> against intensities (and I guess George Sheldrick would agree here).
>>> 
>>> I have a simple question - what exactly, short of some sort of historic 
>>> inertia (or memory lapse), is the reason NOT to refine against intensities?
>>> 
>>> Best, BR
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> 
>>> This e-mail and any attachments may contain confidential, copyright and or 
>>> privileged material, and are for the use of the intended addressee only. If 
>>> you are not the intended addressee or an authorised recipient of the 
>>> addressee please notify us of receipt by returning the e-mail and do not 
>>> use, copy, retain, distribute or disclose the information in or attached to 
>>> the e-mail.
>>> 
>>> Any opinions expressed within this e-mail are those of the individual and 
>>> not necessarily of Diamond Light Source Ltd. 
>>> 
>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>>> attachments are free from viruses and we cannot accept liability for any 
>>> damage which you may sustain as a result of software viruses which may be 
>>> transmitted in or with the message.
>>> 
>>> Diamond Light Source Limited (company no. 4375679). Registered in England 
>>> and Wales with its registered office at Diamond House, Harwell Science and 
>>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Andrew Leslie

The integration programs report a negative intensity simply because that is the 
observation. 

Because of noise in the Xray background, in a large sample of intensity 
estimates for reflections whose true intensity is very very small one will 
inevitably get some measurements that are negative. These must not be rejected 
because this will lead to bias (because some of these intensities for symmetry 
mates will be estimated too large rather than too small). It is not unusual for 
the intensity to remain negative even after averaging symmetry mates.

Andrew


On 20 Jun 2013, at 11:49, Douglas Theobald  wrote:

> Seems to me that the negative Is should be dealt with early on, in the 
> integration step.  Why exactly do integration programs report negative Is to 
> begin with?
> 
> 
> On Jun 20, 2013, at 12:45 PM, Dom Bellini  wrote:
> 
>> Wouldnt be possible to take advantage of negative Is to extrapolate/estimate 
>> the decay of scattering background (kind of Wilson plot of background 
>> scattering) to flat out the background and push all the Is to positive 
>> values?
>> 
>> More of a question rather than a suggestion ...
>> 
>> D
>> 
>> 
>> 
>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
>> Tickle
>> Sent: 20 June 2013 17:34
>> To: ccp4bb
>> Subject: Re: [ccp4bb] ctruncate bug?
>> 
>> Yes higher R factors is the usual reason people don't like I-based 
>> refinement!
>> 
>> Anyway, refining against Is doesn't solve the problem, it only postpones it: 
>> you still need the Fs for maps! (though errors in Fs may be less critical 
>> then).
>> -- Ian
>> 
>> On 20 June 2013 17:20, Dale Tronrud 
>> mailto:det...@uoxray.uoregon.edu>> wrote:
>>  If you are refining against F's you have to find some way to avoid
>> calculating the square root of a negative number.  That is why people
>> have historically rejected negative I's and why Truncate and cTruncate
>> were invented.
>> 
>>  When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
>> care less if Iobs happens to be negative.
>> 
>>  As for why people still refine against F...  When I was distributing
>> a refinement package it could refine against I but no one wanted to do
>> that.  The "R values" ended up higher, but they were looking at R
>> values calculated from F's.  Of course the F based R values are lower
>> when you refine against F's, that means nothing.
>> 
>>  If we could get the PDB to report both the F and I based R values
>> for all models maybe we could get a start toward moving to intensity
>> refinement.
>> 
>> Dale Tronrud
>> 
>> 
>> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>> Just trying to understand the basic issues here.  How could refining 
>> directly against intensities solve the fundamental problem of negative 
>> intensity values?
>> 
>> 
>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
>> mailto:hofkristall...@gmail.com>> wrote:
>> As a maybe better alternative, we should (once again) consider to refine 
>> against intensities (and I guess George Sheldrick would agree here).
>> 
>> I have a simple question - what exactly, short of some sort of historic 
>> inertia (or memory lapse), is the reason NOT to refine against intensities?
>> 
>> Best, BR
>> 
>> 
>> 
>> 
>> -- 
>> 
>> This e-mail and any attachments may contain confidential, copyright and or 
>> privileged material, and are for the use of the intended addressee only. If 
>> you are not the intended addressee or an authorised recipient of the 
>> addressee please notify us of receipt by returning the e-mail and do not 
>> use, copy, retain, distribute or disclose the information in or attached to 
>> the e-mail.
>> 
>> Any opinions expressed within this e-mail are those of the individual and 
>> not necessarily of Diamond Light Source Ltd. 
>> 
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>> attachments are free from viruses and we cannot accept liability for any 
>> damage which you may sustain as a result of software viruses which may be 
>> transmitted in or with the message.
>> 
>> Diamond Light Source Limited (company no. 4375679). Registered in England 
>> and Wales with its registered office at Diamond House, Harwell Science and 
>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>

Re: [ccp4bb] ctruncate bug?

The prior knowledge about Is is not merely that they are >= 0, it's more
than that: we know they have an (approximate) Wilson distribution.  AFAICS
incorporating that information at the integration stage would be almost
equivalent to the F&W procedure.  In fact it would probably not be as good
since the experimental estimates of I do have an (approximate) Gaussian
distribution, being the difference of 2 Poisson distributions with large
means (or at least >~ 10).  The corrected Is, being the best estimates of
the true Is would as you point out not have a Gaussian distribution, and
some of the assumptions made in averaging equivalent reflections would not
be valid.  You could still use the corrected Is instead of the experimental
Is in refinement but I suspect it would not make any difference to the
results (except you would get lower R factors!).

-- Ian


On 20 June 2013 17:49, Douglas Theobald  wrote:

> Seems to me that the negative Is should be dealt with early on, in the
> integration step.  Why exactly do integration programs report negative Is
> to begin with?
>
>
> On Jun 20, 2013, at 12:45 PM, Dom Bellini 
> wrote:
>
> > Wouldnt be possible to take advantage of negative Is to
> extrapolate/estimate the decay of scattering background (kind of Wilson
> plot of background scattering) to flat out the background and push all the
> Is to positive values?
> >
> > More of a question rather than a suggestion ...
> >
> > D
> >
> >
> >
> > From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
> Ian Tickle
> > Sent: 20 June 2013 17:34
> > To: ccp4bb
> > Subject: Re: [ccp4bb] ctruncate bug?
> >
> > Yes higher R factors is the usual reason people don't like I-based
> refinement!
> >
> > Anyway, refining against Is doesn't solve the problem, it only postpones
> it: you still need the Fs for maps! (though errors in Fs may be less
> critical then).
> > -- Ian
> >
> > On 20 June 2013 17:20, Dale Tronrud  det...@uoxray.uoregon.edu>> wrote:
> >   If you are refining against F's you have to find some way to avoid
> > calculating the square root of a negative number.  That is why people
> > have historically rejected negative I's and why Truncate and cTruncate
> > were invented.
> >
> >   When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
> > care less if Iobs happens to be negative.
> >
> >   As for why people still refine against F...  When I was distributing
> > a refinement package it could refine against I but no one wanted to do
> > that.  The "R values" ended up higher, but they were looking at R
> > values calculated from F's.  Of course the F based R values are lower
> > when you refine against F's, that means nothing.
> >
> >   If we could get the PDB to report both the F and I based R values
> > for all models maybe we could get a start toward moving to intensity
> > refinement.
> >
> > Dale Tronrud
> >
> >
> > On 06/20/2013 09:06 AM, Douglas Theobald wrote:
> > Just trying to understand the basic issues here.  How could refining
> directly against intensities solve the fundamental problem of negative
> intensity values?
> >
> >
> > On Jun 20, 2013, at 11:34 AM, Bernhard Rupp  <mailto:hofkristall...@gmail.com>> wrote:
> > As a maybe better alternative, we should (once again) consider to refine
> against intensities (and I guess George Sheldrick would agree here).
> >
> > I have a simple question - what exactly, short of some sort of historic
> inertia (or memory lapse), is the reason NOT to refine against intensities?
> >
> > Best, BR
> >
> >
> >
> >
> > --
> >
> > This e-mail and any attachments may contain confidential, copyright and
> or privileged material, and are for the use of the intended addressee only.
> If you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> >
> > Any opinions expressed within this e-mail are those of the individual
> and not necessarily of Diamond Light Source Ltd.
> >
> > Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> >
> > Diamond Light Source Limited (company no. 4375679). Registered in
> England and Wales with its registered office at Diamond House, Harwell
> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
> >
> >
> >
> >
> >
> >
> >
> >
> >
>

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Dom Bellini

Sorry, perhaps what I was thinking was to use the Icalc to proportionally push 
up the Iobs to push the negative Is to positive numbers.

But I guess that would bias the Iobs  ?

Again just questions rather than suggestions.

D 

-Original Message-
From: Douglas Theobald [mailto:dtheob...@brandeis.edu] 
Sent: 20 June 2013 17:49
To: Bellini, Domenico (DLSLtd,RAL,DIA); ccp4bb
Subject: Re: [ccp4bb] ctruncate bug?

Seems to me that the negative Is should be dealt with early on, in the 
integration step.  Why exactly do integration programs report negative Is to 
begin with?


On Jun 20, 2013, at 12:45 PM, Dom Bellini  wrote:

> Wouldnt be possible to take advantage of negative Is to extrapolate/estimate 
> the decay of scattering background (kind of Wilson plot of background 
> scattering) to flat out the background and push all the Is to positive values?
> 
> More of a question rather than a suggestion ...
> 
> D
> 
> 
> 
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of 
> Ian Tickle
> Sent: 20 June 2013 17:34
> To: ccp4bb
> Subject: Re: [ccp4bb] ctruncate bug?
> 
> Yes higher R factors is the usual reason people don't like I-based refinement!
> 
> Anyway, refining against Is doesn't solve the problem, it only postpones it: 
> you still need the Fs for maps! (though errors in Fs may be less critical 
> then).
> -- Ian
> 
> On 20 June 2013 17:20, Dale Tronrud 
> mailto:det...@uoxray.uoregon.edu>> wrote:
>   If you are refining against F's you have to find some way to avoid 
> calculating the square root of a negative number.  That is why people 
> have historically rejected negative I's and why Truncate and cTruncate 
> were invented.
> 
>   When refining against I, the calculation of (Iobs - Icalc)^2 
> couldn't care less if Iobs happens to be negative.
> 
>   As for why people still refine against F...  When I was distributing 
> a refinement package it could refine against I but no one wanted to do 
> that.  The "R values" ended up higher, but they were looking at R 
> values calculated from F's.  Of course the F based R values are lower 
> when you refine against F's, that means nothing.
> 
>   If we could get the PDB to report both the F and I based R values 
> for all models maybe we could get a start toward moving to intensity 
> refinement.
> 
> Dale Tronrud
> 
> 
> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
> Just trying to understand the basic issues here.  How could refining directly 
> against intensities solve the fundamental problem of negative intensity 
> values?
> 
> 
> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
> mailto:hofkristall...@gmail.com>> wrote:
> As a maybe better alternative, we should (once again) consider to refine 
> against intensities (and I guess George Sheldrick would agree here).
> 
> I have a simple question - what exactly, short of some sort of historic 
> inertia (or memory lapse), is the reason NOT to refine against intensities?
> 
> Best, BR
> 
> 
> 
> 
> --
> 
> This e-mail and any attachments may contain confidential, copyright and or 
> privileged material, and are for the use of the intended addressee only. If 
> you are not the intended addressee or an authorised recipient of the 
> addressee please notify us of receipt by returning the e-mail and do not use, 
> copy, retain, distribute or disclose the information in or attached to the 
> e-mail.
> 
> Any opinions expressed within this e-mail are those of the individual and not 
> necessarily of Diamond Light Source Ltd. 
> 
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
> attachments are free from viruses and we cannot accept liability for any 
> damage which you may sustain as a result of software viruses which may be 
> transmitted in or with the message.
> 
> Diamond Light Source Limited (company no. 4375679). Registered in 
> England and Wales with its registered office at Diamond House, Harwell 
> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United 
> Kingdom
> 
> 
> 
> 
> 
> 
> 
> 
> 




-- 

This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.

Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 

Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses

Re: [ccp4bb] ctruncate bug?

Seems to me that the negative Is should be dealt with early on, in the 
integration step.  Why exactly do integration programs report negative Is to 
begin with?


On Jun 20, 2013, at 12:45 PM, Dom Bellini  wrote:

> Wouldnt be possible to take advantage of negative Is to extrapolate/estimate 
> the decay of scattering background (kind of Wilson plot of background 
> scattering) to flat out the background and push all the Is to positive values?
> 
> More of a question rather than a suggestion ...
> 
> D
> 
> 
> 
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian 
> Tickle
> Sent: 20 June 2013 17:34
> To: ccp4bb
> Subject: Re: [ccp4bb] ctruncate bug?
> 
> Yes higher R factors is the usual reason people don't like I-based refinement!
> 
> Anyway, refining against Is doesn't solve the problem, it only postpones it: 
> you still need the Fs for maps! (though errors in Fs may be less critical 
> then).
> -- Ian
> 
> On 20 June 2013 17:20, Dale Tronrud 
> mailto:det...@uoxray.uoregon.edu>> wrote:
>   If you are refining against F's you have to find some way to avoid
> calculating the square root of a negative number.  That is why people
> have historically rejected negative I's and why Truncate and cTruncate
> were invented.
> 
>   When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
> care less if Iobs happens to be negative.
> 
>   As for why people still refine against F...  When I was distributing
> a refinement package it could refine against I but no one wanted to do
> that.  The "R values" ended up higher, but they were looking at R
> values calculated from F's.  Of course the F based R values are lower
> when you refine against F's, that means nothing.
> 
>   If we could get the PDB to report both the F and I based R values
> for all models maybe we could get a start toward moving to intensity
> refinement.
> 
> Dale Tronrud
> 
> 
> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
> Just trying to understand the basic issues here.  How could refining directly 
> against intensities solve the fundamental problem of negative intensity 
> values?
> 
> 
> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
> mailto:hofkristall...@gmail.com>> wrote:
> As a maybe better alternative, we should (once again) consider to refine 
> against intensities (and I guess George Sheldrick would agree here).
> 
> I have a simple question - what exactly, short of some sort of historic 
> inertia (or memory lapse), is the reason NOT to refine against intensities?
> 
> Best, BR
> 
> 
> 
> 
> -- 
> 
> This e-mail and any attachments may contain confidential, copyright and or 
> privileged material, and are for the use of the intended addressee only. If 
> you are not the intended addressee or an authorised recipient of the 
> addressee please notify us of receipt by returning the e-mail and do not use, 
> copy, retain, distribute or disclose the information in or attached to the 
> e-mail.
> 
> Any opinions expressed within this e-mail are those of the individual and not 
> necessarily of Diamond Light Source Ltd. 
> 
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
> attachments are free from viruses and we cannot accept liability for any 
> damage which you may sustain as a result of software viruses which may be 
> transmitted in or with the message.
> 
> Diamond Light Source Limited (company no. 4375679). Registered in England and 
> Wales with its registered office at Diamond House, Harwell Science and 
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
> 
> 
> 
> 
> 
> 
> 
> 
>

Re: [ccp4bb] ctruncate bug?

On Jun 20, 2013, at 12:20 PM, Dale Tronrud  wrote:

>   If you are refining against F's you have to find some way to avoid
> calculating the square root of a negative number.  That is why people
> have historically rejected negative I's and why Truncate and cTruncate
> were invented.
> 
>   When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
> care less if Iobs happens to be negative.

But we know that Is can't be negative.  Using (Iobs - Icalc)^2 does not 
incorporate that bit of physics, and it implicitly assumes a Gaussian 
distribution for the Is, which is impossible for a variable that is positive 
semi-definite.  Refining against (Iobs - Icalc)^2 is mathematically equivalent 
to shifting every I by the most negative I and refining against that, a crude 
baseline correction that I doubt most people would consider valid.  
Transforming the data to Fs at least makes the Gaussian assumption plausible, 
and I always assumed that was one main reason for working with Fs (since all 
the refinement programs assume Gaussians).  

>   As for why people still refine against F...  When I was distributing
> a refinement package it could refine against I but no one wanted to do
> that.  The "R values" ended up higher, but they were looking at R
> values calculated from F's.  Of course the F based R values are lower
> when you refine against F's, that means nothing.

R-values also implicitly assume a Gaussian, right?

> 
>   If we could get the PDB to report both the F and I based R values
> for all models maybe we could get a start toward moving to intensity
> refinement.
> 
> Dale Tronrud
> 
> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>> Just trying to understand the basic issues here.  How could refining 
>> directly against intensities solve the fundamental problem of negative 
>> intensity values?
>> 
>> 
>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp  wrote:
>> 
 As a maybe better alternative, we should (once again) consider to refine 
 against intensities (and I guess George Sheldrick would agree here).
>>> 
>>> I have a simple question - what exactly, short of some sort of historic 
>>> inertia (or memory lapse), is the reason NOT to refine against intensities?
>>> 
>>> Best, BR

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Dom Bellini

Wouldnt be possible to take advantage of negative Is to extrapolate/estimate 
the decay of scattering background (kind of Wilson plot of background 
scattering) to flat out the background and push all the Is to positive values?

More of a question rather than a suggestion ...

D



From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian Tickle
Sent: 20 June 2013 17:34
To: ccp4bb
Subject: Re: [ccp4bb] ctruncate bug?

Yes higher R factors is the usual reason people don't like I-based refinement!

Anyway, refining against Is doesn't solve the problem, it only postpones it: 
you still need the Fs for maps! (though errors in Fs may be less critical then).
-- Ian

On 20 June 2013 17:20, Dale Tronrud 
mailto:det...@uoxray.uoregon.edu>> wrote:
   If you are refining against F's you have to find some way to avoid
calculating the square root of a negative number.  That is why people
have historically rejected negative I's and why Truncate and cTruncate
were invented.

   When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
care less if Iobs happens to be negative.

   As for why people still refine against F...  When I was distributing
a refinement package it could refine against I but no one wanted to do
that.  The "R values" ended up higher, but they were looking at R
values calculated from F's.  Of course the F based R values are lower
when you refine against F's, that means nothing.

   If we could get the PDB to report both the F and I based R values
for all models maybe we could get a start toward moving to intensity
refinement.

Dale Tronrud


On 06/20/2013 09:06 AM, Douglas Theobald wrote:
Just trying to understand the basic issues here.  How could refining directly 
against intensities solve the fundamental problem of negative intensity values?


On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
mailto:hofkristall...@gmail.com>> wrote:
As a maybe better alternative, we should (once again) consider to refine 
against intensities (and I guess George Sheldrick would agree here).

I have a simple question - what exactly, short of some sort of historic inertia 
(or memory lapse), is the reason NOT to refine against intensities?

Best, BR




-- 

This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.

Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 

Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.

Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Re: [ccp4bb] ctruncate bug?

Yes higher R factors is the usual reason people don't like I-based
refinement!

Anyway, refining against Is doesn't solve the problem, it only postpones
it: you still need the Fs for maps! (though errors in Fs may be less
critical then).

-- Ian


On 20 June 2013 17:20, Dale Tronrud  wrote:

>If you are refining against F's you have to find some way to avoid
> calculating the square root of a negative number.  That is why people
> have historically rejected negative I's and why Truncate and cTruncate
> were invented.
>
>When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
> care less if Iobs happens to be negative.
>
>As for why people still refine against F...  When I was distributing
> a refinement package it could refine against I but no one wanted to do
> that.  The "R values" ended up higher, but they were looking at R
> values calculated from F's.  Of course the F based R values are lower
> when you refine against F's, that means nothing.
>
>If we could get the PDB to report both the F and I based R values
> for all models maybe we could get a start toward moving to intensity
> refinement.
>
> Dale Tronrud
>
>
> On 06/20/2013 09:06 AM, Douglas Theobald wrote:
>
>> Just trying to understand the basic issues here.  How could refining
>> directly against intensities solve the fundamental problem of negative
>> intensity values?
>>
>>
>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp 
>> wrote:
>>
>>  As a maybe better alternative, we should (once again) consider to refine
 against intensities (and I guess George Sheldrick would agree here).

>>>
>>> I have a simple question - what exactly, short of some sort of historic
>>> inertia (or memory lapse), is the reason NOT to refine against intensities?
>>>
>>> Best, BR
>>>
>>

Re: [ccp4bb] ctruncate bug?

2013-06-20 Thread Dale Tronrud


   If you are refining against F's you have to find some way to avoid
calculating the square root of a negative number.  That is why people
have historically rejected negative I's and why Truncate and cTruncate
were invented.

   When refining against I, the calculation of (Iobs - Icalc)^2 couldn't
care less if Iobs happens to be negative.

   As for why people still refine against F...  When I was distributing
a refinement package it could refine against I but no one wanted to do
that.  The "R values" ended up higher, but they were looking at R
values calculated from F's.  Of course the F based R values are lower
when you refine against F's, that means nothing.

   If we could get the PDB to report both the F and I based R values
for all models maybe we could get a start toward moving to intensity
refinement.

Dale Tronrud

On 06/20/2013 09:06 AM, Douglas Theobald wrote:

Just trying to understand the basic issues here.  How could refining directly 
against intensities solve the fundamental problem of negative intensity values?


On Jun 20, 2013, at 11:34 AM, Bernhard Rupp  wrote:


As a maybe better alternative, we should (once again) consider to refine 
against intensities (and I guess George Sheldrick would agree here).


I have a simple question - what exactly, short of some sort of historic inertia 
(or memory lapse), is the reason NOT to refine against intensities?

Best, BR

Re: [ccp4bb] ctruncate bug?