Dear Martin,
The concept of systematic absences is usually used for absences caused by the
crystal packing, e.g. glide planes etc. I guess that what you meant are
absences due to anisotropic cutoff. In the past, to get an Rmerge is low as
possible, people would cut there data at 2sigma, or even at 3 sigma, resulting
in data sets with significant high-resolution data missing. The same happened
if the detector was placed to far away from the crystal.
If the anisotropic cutoff is done correctly, e.g. at 1sigma or lower, the
measured data set is complete. The fourier terms that are missing do not have
any significant amplitude and are basically noise. So if all refinement and
other software would be perfect, there would be no artefacts due to missing
data, since there is no missing data, only missing noise. In your words, the
missing data is extremely weak, not strong.
However, and I guess this is what Frank was hinting at, this anisotropic
diffraction should be correctly handled by the refinement programs and should
be modeled correctly in the resulting models. A first step would be matching
overall anisotropic B-factors but I am sure there are more sophisticated
solutions.
My 2 cents,
Herman
Von: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> Im Auftrag von Martin Malý
Gesendet: Mittwoch, 5. Oktober 2022 14:13
An: CCP4BB@JISCMAIL.AC.UK
Betreff: Re: [ccp4bb] PAIREF, Anisotropy and STARANISO
Dear colleagues,
Firstly, I would like to thank Clemens, Claus, Ian and Gerard for the very
interesting analysis and remarks!
Similarly to Frank, I am wondering "whether any refinement program properly
extracts the high-resolution signal when there's anisotropy". Poor completeness
in high resolution can play a role. Somewhere in the "gear wheels" of
refinement programs (refmac5, phenix.refine, buster, ...), Fourier transform
has to take place. It is a sum through hkl indices which somehow assumes
isotropic data to "work reasonably", I think. However, anisotropic cutoffs
produce strong systematic absences in data. The refinement programs must fit
the structure model on significantly incomplete data, which is quite
questionable. On the other hand, structure refinement is usually stable and
converges also when using anisotropically-cut data so that is a good sign.
To conclude, I am still wondering if there should or should not be a criterion
for the minimal spherical completeness in high resolution. This would probably
vary for particular refinement programs/strategies.
Best regards,
Martin
On 05. 10. 22 11:36, Frank von Delft wrote:
Gerard, that's fascinating, thanks for the explanation.
I conclude in summary: nobody (including you) yet knows whether any refinement
program properly extracts the high-resolution signal when there's anisotropy.
My (similar) question a few days ago was: "Is that because of the
practicalities of implementing the numerical methods, or because of something
fundamental about the refinement formalism?"
I think you say below that the problem is (might be?) that the error models and
corrections assume isotropic behaviour.
So if that doesn't work, shouldn't the error models instead be derived from the
statistics of local reciprocal space? As you do in STARANISO (from memory).
Frank
On 04/10/2022 17:01, Gerard Bricogne wrote:
Dear all,
First of all, apologies for breaking the threads entitled "PAIREF -
Warning - not enough free reflections in resolution bin" and "Anisotropy" by
merging them into a new one, but it somehow felt rather against nature to
keep them separate.
Since the early days of the availability of STARANISO [1] (the actual
starting year for the Web server [2] was 2016), we had a hunch that much of
what was happening in the PAIREF procedure might simply be the detection of
the existence of significant data beyond an initially chosen resolution
cut-off not only as a result of an excessively conservative criterion having
been applied in that initial choice, but as a consequence of anisotropy in
the data. The latter would give rise to different diffraction limits in
different directions, and the choice of a single value for "the resolution"
at which the data were cut off would necessarily yield a compromise value
between the best and the worse diffraction limits. This would imply that
significant data would be excluded in the best diffracting directions, that
would subsequently drive PAIREF towards increasing the estimated resolution
compared to its compromise value.
This "hunch" was validated by a detailed comparison carried out on the
exact same examples that are considered in the 2020 paper by Maly et al.,
that is summarised in the attached PDF. In other words, whenever anisotropy
is present in the data, PAIREF will tend to indicate a higher value for an
isotropic cut-off than would have been estimated for the initial dataset.
The problem with taking the PAIREF result as the final answer is that the
higher cut-off it indicates is applied *isotropically*. The inclusion of the
significant data thus reclaimed is therefore unavoidably accompanied by that
of noisy data in the worst diffracting direction(s), resulting in alarmingly
poor statistics in the outermost shell (as pointed out in Eleanor's message)
that may cast doubts on the usefulness of the procedure. This consideration
was the basis of the rationale for implementing an *anisotropic* cut-off
surface in STARANISO, so that one could thus reclaim the significant data in
the best-diffracting direction(s) while avoiding the simultaneous inclusion
of the pure-noise measurements in the worse one(s). While this is clearly
and extensively explained in the documentation provided on the STARANISO
server [2], it seems to be far from having been assimilated. Of course this
would be perfect material for a publication, but life is somehow too short,
and our to-do list has remained too long, to leave us room for spending the
necessary time to go through the process of putting a paper together. The
truly important matter is to get our picture in front of the user community.
Now that the combined topics of PAIREF and anisotropy are being brought
to the foreground of the community's attention, this seems like the perfect
opportunity to present our analysis and position: what PAIREF achieves in
terms of an upward revision of an initial isotropic resolution cut-off is
likely to be achieved more straightforwardly by submitting the same data to
the STARANISO server (or using it within autoPROC [3]); and the STARANISO
output will have the advantage of being devoid of the large extra amount of
purely noisy, uninformative data that are retained in the output from PAIREF
according to its revised isotropic cut-off.
We would very much welcome feedback on this position: indeed we would
like to *crowd-source* the validation (or refutation) of this conclusion. In
our view, continuing to use the PAIREF procedure to revise an isotropic
resolution cut off misses the point about the consequences of anisotropy.
The only sensible use of a PAIREF-like procedure would be to adjust the
cut-off threshold for the local average of I/sig(I) in STARANISO, whose
default value is currently 1.2 but can be reset by the user through the Web
server's GUI. We occasionally see datasets of very high quality for which
the CC_1/2 value in the outermost shell stays above 0.6 or even 0.7, and it
is quite plausible that further useful data could be rescued if the local
I/sig(I) cut-off threshold were lowered below 1.2.
Concerning Eleanor's view that noisy data can't hurt refinement because
they are properly down-weighted by the consideration of e.g. Rfree values in
resolution shells, we would point out that any criterion based on statistics
in resolution shells will be polluted if the data are anisotropic and if the
noisy data that STARANISO would reject are retained. That will result in
excessive down-weighting of the significant data that STARANISO retains,
hence in losing the information they contain. Perhaps this is a matter for
later discussion, but the main idea is that retaining pure-noise data is not
neutral in refinement, and that every "isotropic thinking habit" on which
many views are based needs to be revisited.
With best wishes,
Clemens, Claus, Ian and Gerard.
[1] Tickle, I.J., Flensburg, C., Keller, P., Paciorek, W., Sharff, A.,
Vonrhein, C., Bricogne, G. (2018). STARANISO. Cambridge, United
Kingdom: Global Phasing Ltd.
https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A2=ind1806&L=CCP4BB&O=D&P=3971<https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A2=ind1806&L=CCP4BB&O=D&P=3971>
[2] https://staraniso.globalphasing.org/<https://staraniso.globalphasing.org/>
[3]
https://doi.org/10.1107/s0907444911007773<https://doi.org/10.1107/s0907444911007773>
https://www.globalphasing.com/autoproc/<https://www.globalphasing.com/autoproc/>
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
This message was issued to members of
www.jiscmail.ac.uk/CCP4BB<http://www.jiscmail.ac.uk/CCP4BB>, a mailing list
hosted by www.jiscmail.ac.uk<http://www.jiscmail.ac.uk>, terms & conditions are
available at
https://www.jiscmail.ac.uk/policyandsecurity/<https://www.jiscmail.ac.uk/policyandsecurity/>
________________________________
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
________________________________
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
