Dear all,
taking AlphaFold models for " true" experimental structures seems to
become a serious problem.
I am just returning from a meeting (not a structural biology meeting)
and saw one model after other. And the non-structural biologist used
terms like "we calculated a structure" or "a AlphaFold crystal
structure" or "the structure was accurate, is was all blue".
Alphafold models were used to predict ion channels, electron transfer
pathways, enzyme mechanisms, and yes, not tested by experiments. Wide
ranging conclusions were drawn on these pure models, which I would not
dare to draw on limited experimental data.
There is something going severely wrong.
And don't get me wrong, I think AlphaFold and other prediction software
is great to create testable models (like MR models) or try to figure out
how proteins might assemble and so on.
But I get the impression that too many of our colleagues got the
impression that pressing a button replaces experiments.
Seeing that grants are rejected by such arguments is alarming and we
should do something.
Best wishes,
Guenter
Very sorry to hear about your grant. I've been there. It is crushing
to be rejected, and frustrating when the reason given is ... wrong.
My journalist friends wonder why scientists don't like talking to
journalists. This is why. I remember when the first results from
XFELs were published, and it was immediately declared that there was
no longer a need for NASA, whose sole purpose (apparently) was to grow
bigger and better crystals in space. (?!) I find the idea that
AlphaFold has eliminated the need to solve any more structures equally
ludicrous.
I think the best analogy for what has happened in structural biology
is the same impact a Star Trek style "transporter device" would have
on your daily commute. Except this "transporter" is only accurate
enough to get you within a mile or two of your house. Most of the
time. Don't worry, its not going to beam you inside a rock or into the
sky, as it was trained on data with good Clashscores (we think). But,
you are on your own getting the rest of the way home. This "Last Mile"
of transportation networks is actually the most challenging, and
expensive, but also the most critical. In structural biology, the
"Last Angstrom" between prediction and actuality is equally important,
but also fraught with difficulty. It may seem like a short distance,
until you have to walk it. So, despite amazing progress, it is still
premature to dismantle infrastructure, and definitely a bad idea to
nail your front door shut.
Personally, I see this structure prediction revolution as nothing more
nor less than the fruition of Structural Genomics. It started in the
final days of the 20th century. I was there! The stated goal of that
worldwide initiative was to create the data set that would be needed
by some future (at the time) homology modelling technology to do
exactly what AlphaFold does: get us "close enough". And then Greg
Petsko asked: what is "close enough"? He called it "The Grail
problem". By what metric do you declare victory? He made an excellent
suggestion:
"But there is an obvious method of evaluation that will allow any
structure prediction method to be assessed. It is simply to demand
that the method produce a model that can be used to solve the
corresponding protein crystal structure by the method of molecular
replacement."
-Greg Petsko - June 9, 2000
https://doi.org/10.1186/gb-2000-1-1-comment002
This is the thing that just changed. Structure prediction has finally
crossed the "G-P threshold". Not 100% of the time, but impressively
often now, the predictions can be used for MR. This is a massively
useful tool! Not the end of the field, but rather the beginning of an
exciting new era where success rates skyrocket.
Scores like the GDT used in CASP were developed with this Grail
Problem criterion in mind, and I think that is what John Moult and
others meant when they said things that got quoted like this:
"Scores above 90 on the 100-point scale are considered on par with
experimental methods, Moult says."
https://www.science.org/doi/full/10.1126/science.370.6521.1144
Meaning that the predicted models work as search models for MR about
as often as search models derived from homologous (and yes,
"experimentally determined") structures. A GDT of 100 does NOT mean
the model is better than the data. That is not even how it works.
But, unfortunately, this seems to have gotten paraphrased and
sensationalized:
"generally considered to be competitive with the same results obtained
via experimental methods"
https://www.sciencealert.com/ai-solves-50-year-old-biology-grand-challenge-decades-before-experts-predicted
"software predictions finally match structures calculated from
experimental data"
https://www.science.org/doi/full/10.1126/science.370.6521.1144
"comparable in quality to experimental structures"
https://www.nature.com/articles/d41586-020-03348-4
"accuracy comparable to laboratory experiments"
https://www.bbc.com/news/science-environment-55133972
<sigh>
The only kind of diffraction where prediction is better than
experiment is that of monoatomic gasses. These curves can be derived
very accurately and completely from fundamental constants of physics.
This is where those tables of atomic scattering factors used by
refinement programs come from. For a while, the experimentally
measured curves were used, but once Hartree, Fock, Slater, Cromer,
Mann and others worked out how to do the self-consistent field
calculations accurately, by the late 1960s the calculated form factors
supplanted the measured ones.
You might also say that for "small molecule" crystals the models are
better than the data. Indeed, the CSD did not require experimental
data to be deposited until fairly recently. The coordinates were
considered more accurate than the intensities because publication
requirements for chemical crystallography R factors are low enough to
be dominated by experimental noise only. Nevertheless, despite the
phase problem being cracked by direct methods in the 1980s, your local
chemistry department has yet to shut down their diffractometer. Why?
Because they need it. And for macromolecular structures, the
systematic errors between refined coordinates and their corresponding
data are about 4-5x larger than experimental error. So, don't delete
your image data! Not for a while yet.
-James Holton
MAD Scientist
On 4/1/2023 7:57 AM, Subramanian, Ramaswamy wrote:
Ian,
Thank you. This is not an April fools..
Rams
subra...@purdue.edu
On Apr 1, 2023, at 10:46 AM, Ian Tickle <ianj...@gmail.com> wrote:
---- *External Email*: Use caution with attachments, links, or
sharing data ----
Hi Ramaswamy
I assume this is an April Fool's but it's still a serious question
because many reviewers who are not crystallographers or electron
microscopists may not fully appreciate the difference currently
between the precision of structures obtained by experimental and
predictive methods, though the latter are certainly catching up.
The answer of course lies in the mean co-ordinate precision, related
to the map resolution.
Quoting https://people.cryst.bbk.ac.uk/~ubcg05m/precgrant.html :
"The accuracy and precision required of an experimentally determined
model of a macromolecule depends on the biological questions being
asked of the structure. Questions involving the overall fold of a
protein, or its topological similarity to other proteins, can be
answered by structures of fairly low precision such as those
obtained from very low resolution X-ray crystal diffraction data [or
AlphaFold]. Questions involving reaction mechanisms require much
greater accuracy and precision as obtained from well-refined,
high-resolution X-ray structures, including proper statistical
analyses of the standard uncertainties (/s.u.'s/) of atomic
positions and bond lengths.".
According to https://www.nature.com/articles/s41586-021-03819-2 :
The accuracy of AlphaFold structures at the time of writing (2021)
was around 1.0 Ang. RMSD for main-chain and 1.5 Ang. RMSD for
side-chain atoms and probably hasn't changed much since. This is
described as "highly accurate"; however this only means that
AlphaFold's accuracy is much higher in comparison with other
prediction methods, not in comparison with experimental methods.
Also note that AlphaFold's accuracy is estimated by comparison with
the X-ray structure which remains the "gold standard"; there's no
way (AFAIK) of independently assessing AlphaFold's accuracy or
precision.
Quoting https://scripts.iucr.org/cgi-bin/paper?S0907444998012645 :
"Data of 0.94 A resolution for the 237-residue protein concanavalin
A are used in unrestrained and restrained full-matrix inversions to
provide standard uncertainties sigma(r) for positions and sigma(l)
for bond lengths. sigma(r) is as small as 0.01 A for atoms with low
Debye B values but increases strongly with B."
There's a yawning gap between 1.0 - 1.5 Ang. and 0.01 Ang.! Perhaps
AlphaFold structures should be deposited using James Holton's new
PDB format (now that is an April Fool's !).
One final suggestion for a reference in your grant application:
https://www.biorxiv.org/content/10.1101/2022.03.08.483439v2 .
Cheers
-- Ian
On Sat, 1 Apr 2023 at 13:06, Subramanian, Ramaswamy
<subra...@purdue.edu> wrote:
Dear All,
I am unsure if all other groups will get it - but I am sure this
group will understand the frustration.
My NIH grant did not get funded. A few genuine comments - they
make excellent sense. We will fix that.
One major comment is, “Structures can be predicted by alpfafold
and other software accurately, so the effort put on the grant to
get structures by X-ray crystallography/cryo-EM is not justified.”
The problem is when a company with billions of $$s develops a
method and blasts it everywhere - the message is so pervasive…
*Question: I*s there a canned consensus paragraph that one can
add with references to grants with structural biology
(especially if the review group is not a structural biology
group) to say why the most modern structure prediction programs
are not a substitute for structural work?
Thanks.
Rams
subra...@purdue.edu
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/