[ccp4bb] Validation of structure prediction

[dusan turk]

es in that thread should be
>> enough to illustrate the scale of the problem.
>> 
>> And that was *with* a map to fit! Take away the map, and run some MD
>> energy minimisation (perhaps with added Ramachandran and rotamer
>> restraints), and I think it would be easy to get your model to fool most
>> “simple” validation metrics (please don’t actually do this). The upshot is
>> that I still think validation of predicted models in the absence of at
>> least moderate-resolution experimental data is still a major challenge
>> requiring very careful thought.
>> 
>> — Tristan
>> 
>> On 13 Jan 2022, at 18:41, James Holton  wrote:
>> 
>> Agree with Pavel.
>> 
>> Something I think worth adding is a reminder that the MolProbity score
>> only looks at bad clashes, ramachandran and rotamer outliers.
>> 
>> 
>> MPscore=0.426∗ln(1+clashscore)+0.33∗ln(1+max(0,rota_out−1))+0.25∗ln(1+max(0,rama_iffy−2))+0.5
>> 
>> It pays no attention whatsoever to twisted peptide bonds, C-beta
>> deviations, and, for that matter, bond lengths and bond angles. If you
>> tweak your weights right you can get excellent MP scores, but horrible
>> "geometry" in the traditional bonds-and-angles sense. The logic behind this
>> kind of validation is that normally nonbonds and torsions are much softer
>> than bond and angle restraints and therefore fertile ground for detecting
>> problems.  Thus far, I am not aware of any "Grand Unified Score" that
>> combines all geometric considerations, but perhaps it is time for one?
>> 
>> Tristan's trivial solution aside, it is actually very hard to make all the
>> "geometry" ideal for a real-world fold, and especially difficult to do
>> without also screwing up the agreement with density (R factor).  I would
>> argue that if you don't have an R factor then you should get one, but I am
>> interested in opinions about alternatives.
>> 
>> I.E. What if we could train an AI to predict Rfree by looking at the
>> coordinates?
>> 
>> -James Holton
>> MAD Scientist
>> 
>> On 12/21/2021 9:25 AM, Pavel Afonine wrote:
>> 
>> Hi Reza,
>> 
>> If you think about it this way... Validation is making sure that the model
>> makes sense, data make sense and model-to-data fit make sense, then the
>> answer to your question is obvious: in your case you do not have
>> experimental data (at least in a way we used to think of it) and so then of
>> these three validation items you only have one, which, for example, means
>> you don’t have to report things like R-factors or completeness in
>> high-resolution shell.
>> 
>> Really, the geometry of an alpha helix does not depend on how you
>> determined it: using X-rays or cryo-EM or something else! So, most (if not
>> all) model validation tools still apply.
>> 
>> Pavel
>> 
>> On Mon, Dec 20, 2021 at 8:10 AM Reza Khayat  wrote:
>> 
>>> Hi,
>>> 
>>> 
>>> Can anyone suggest how to validate a predicted structure? Something
>>> similar to wwPDB validation without the need for refinement statistics. I
>>> realize this is a strange question given that the geometry of the model is
>>> anticipated to be fine if the structure was predicted by a server that
>>> minimizes the geometry to improve its statistics. Nonetheless, the journal
>>> has asked me for such a report. Thanks.
>>> 
>>> 
>>> Best wishes,
>>> 
>>> Reza
>>> 
>>> 
>>> Reza Khayat, PhD
>>> Associate Professor
>>> City College of New York
>>> Department of Chemistry and Biochemistry
>>> New York, NY 10031
>>> 
>>> --
>>> 
>>> To unsubscribe from the CCP4BB list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>>> 
>> 
>> --
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>> 
>> 
>> 
>> --
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>> 
>> 
>> 
>> --
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>> 
> 
> 
> -- 
> Jan Dohnalek, Ph.D
> Institute

Re: [ccp4bb] Validation of structure prediction

[CAVARELLI Jean (VIE)]

May be useful if not already mentioned 

Ten things I `hate' about refinement 
Pietro Roversi and Dale E. Tronrud 
https://journals.iucr.org/d/issues/2021/12/00/qt5008/index.html 

- 
Jean Cavarelli 
Professor of Structural Biology 
"Structural biology of epigenetic targets" 
Department of Integrated structural biology 
IGBMC,UMR7104 CNRS-UNISTRA, INSERM U 1258 
phone : +33 (0)3 69 48 52 74 


De: "Jan Dohnalek"  
À: "ccp4bb"  
Envoyé: Lundi 17 Janvier 2022 09:39:33 
Objet: Re: [ccp4bb] Validation of structure prediction 

I think quite a bit of this "inconsistency" with protein structures comes from 
the fact that with our larger globules it is much more true that our model is 
an approximate time and space average of something that could have the ideal 
geometry. 
I.e. the way we are trying to represent the density is actually not that 
appropriate. The only "improvement" to this I think is the multiple model 
approach. 

My 2 c. 

Jan 


On Sat, Jan 15, 2022 at 9:29 PM James Holton < [ mailto:jmhol...@lbl.gov | 
jmhol...@lbl.gov ] > wrote: 




On 1/13/2022 11:14 AM, Tristan Croll wrote: 

BQ_BEGIN
(please don’t actually do this) 


Too late! I've been doing that for years. What happens, of course, is the 
"geometry" improves, but the R factors go through the roof. This I expect comes 
as no surprise to anyone who has played with the "weight" parameters in 
refinement, but maybe it should? What is it about our knowledge of chemical 
bond lengths, angles, and radii that is inconsistent with the electron density 
of macromolecules, but not small molecules? Why do macro-models have a burning 
desire to leap away from the configuration we know they adopt in reality? If 
you zoom in on those "bad clashes" individually, they don't look like something 
that is supposed to happen. There is a LOT of energy stored up in those little 
springs. I have a hard time thinking that's for real. The molecule is no doubt 
doing something else and we're just not capturing it properly. There is 
information to be had here, a lot of information. 

This is why I too am looking for an all-encompassing "geometry score". Right 
now I'm multiplying other scores together: 

score = (1+Clashscore)*sin(worst_omega)*1./(1+worst_rama)*1/(1+worst_rota) 
*Cbetadev*worst_nonbond*worst_bond*worst_angle*worst_dihedral*worst_chir*worst_plane
 

where things like worst_rama is the "%score" given to the worst Ramachandran 
angle by phenix.ramalyze, and worst_bond is the largest "residual" reported 
among all the bonds in the structure by molprobity or 
phenix.geometry_minimization. For "worst_nonbond" I'm plugging the observed and 
ideal distances into a Leonard-Jones6-12 potential to convert it into an 
"energy" that is always positive. 

With x-ray data in hand, I've been multiplying this whole thing by Rwork and 
trying to find clever ways to minimize the product. Rfree is then, as always, 
the cross-check. 

Or does someone have a better idea? 

-James Holton 
MAD Scientist 


On 1/13/2022 11:14 AM, Tristan Croll wrote: 

BQ_BEGIN

Hard but not impossible - even when you *are* fitting to low-res density. See [ 
https://twitter.com/crolltristan/status/1381258326223290373?s=21 | 
https://twitter.com/crolltristan/status/1381258326223290373?s=21 ] for example 
- no Ramachandran outliers, 1.3% sidechain outliers, clashscore of 2... yet 
multiple regions out of register by anywhere up to 15 residues! I never 
publicly named the structure (although I did share my rebuilt model with the 
authors), but the videos and images in that thread should be enough to 
illustrate the scale of the problem. 

And that was *with* a map to fit! Take away the map, and run some MD energy 
minimisation (perhaps with added Ramachandran and rotamer restraints), and I 
think it would be easy to get your model to fool most “simple” validation 
metrics (please don’t actually do this). The upshot is that I still think 
validation of predicted models in the absence of at least moderate-resolution 
experimental data is still a major challenge requiring very careful thought. 

— Tristan 

On 13 Jan 2022, at 18:41, James Holton < [ mailto:jmhol...@lbl.gov | 
jmhol...@lbl.gov ] > wrote: 


BQ_BEGIN

Agree with Pavel. 

Something I think worth adding is a reminder that the MolProbity score only 
looks at bad clashes, ramachandran and rotamer outliers. 

MPscore=0.426∗ln(1+clashscore)+0.33∗ln(1+max(0,rota_out−1))+0.25∗ln(1+max(0,rama_iffy−2))+0.5
 

It pays no attention whatsoever to twisted peptide bonds, C-beta deviations, 
and, for that matter, bond lengths and bond angles. If you tweak your weights 
right you can get excellent MP scores, but horrible "geometry" in the 
traditional bonds-and-angles sense. The logic behind this kind of validation is 
that normally nonbonds and torsions ar

Re: [ccp4bb] Validation of structure prediction

[Jan Dohnalek]

I think quite a bit of this "inconsistency" with protein structures comes
from the fact that with our larger globules it is much more true that our
model is an approximate time and space average of something that could have
the ideal geometry.
I.e. the way we are trying to represent the density is actually not that
appropriate. The only "improvement" to this I think is the multiple model
approach.

My 2 c.

Jan


On Sat, Jan 15, 2022 at 9:29 PM James Holton  wrote:

>
> On 1/13/2022 11:14 AM, Tristan Croll wrote:
>
> (please don’t actually do this)
>
>
> Too late!  I've been doing that for years.  What happens, of course, is
> the "geometry" improves, but the R factors go through the roof.  This I
> expect comes as no surprise to anyone who has played with the "weight"
> parameters in refinement, but maybe it should?  What is it about our
> knowledge of chemical bond lengths, angles, and radii that is inconsistent
> with the electron density of macromolecules, but not small molecules?  Why
> do macro-models have a burning desire to leap away from the configuration
> we know they adopt in reality?  If you zoom in on those "bad clashes"
> individually, they don't look like something that is supposed to happen.
> There is a LOT of energy stored up in those little springs.  I have a hard
> time thinking that's for real. The molecule is no doubt doing something
> else and we're just not capturing it properly.  There is information to be
> had here, a lot of information.
>
> This is why I too am looking for an all-encompassing "geometry score".
> Right now I'm multiplying other scores together:
>
> score = (1+Clashscore)*sin(worst_omega)*1./(1+worst_rama)*1/(1+worst_rota)
>
> *Cbetadev*worst_nonbond*worst_bond*worst_angle*worst_dihedral*worst_chir*worst_plane
>
> where things like worst_rama is the "%score" given to the worst
> Ramachandran angle by phenix.ramalyze, and worst_bond is the largest
> "residual" reported among all the bonds in the structure by molprobity or
> phenix.geometry_minimization.  For "worst_nonbond" I'm plugging the
> observed and ideal distances into a Leonard-Jones6-12 potential to convert
> it into an "energy" that is always positive.
>
> With x-ray data in hand, I've been multiplying this whole thing by Rwork
> and trying to find clever ways to minimize the product.  Rfree is then, as
> always, the cross-check.
>
> Or does someone have a better idea?
>
> -James Holton
> MAD Scientist
>
>
> On 1/13/2022 11:14 AM, Tristan Croll wrote:
>
> Hard but not impossible - even when you *are* fitting to low-res density.
> See https://twitter.com/crolltristan/status/1381258326223290373?s=21 for
> example - no Ramachandran outliers, 1.3% sidechain outliers, clashscore of
> 2... yet multiple regions out of register by anywhere up to 15 residues! I
> never publicly named the structure (although I did share my rebuilt model
> with the authors), but the videos and images in that thread should be
> enough to illustrate the scale of the problem.
>
> And that was *with* a map to fit! Take away the map, and run some MD
> energy minimisation (perhaps with added Ramachandran and rotamer
> restraints), and I think it would be easy to get your model to fool most
> “simple” validation metrics (please don’t actually do this). The upshot is
> that I still think validation of predicted models in the absence of at
> least moderate-resolution experimental data is still a major challenge
> requiring very careful thought.
>
> — Tristan
>
> On 13 Jan 2022, at 18:41, James Holton  wrote:
>
> Agree with Pavel.
>
> Something I think worth adding is a reminder that the MolProbity score
> only looks at bad clashes, ramachandran and rotamer outliers.
>
>
> MPscore=0.426∗ln(1+clashscore)+0.33∗ln(1+max(0,rota_out−1))+0.25∗ln(1+max(0,rama_iffy−2))+0.5
>
>  It pays no attention whatsoever to twisted peptide bonds, C-beta
> deviations, and, for that matter, bond lengths and bond angles. If you
> tweak your weights right you can get excellent MP scores, but horrible
> "geometry" in the traditional bonds-and-angles sense. The logic behind this
> kind of validation is that normally nonbonds and torsions are much softer
> than bond and angle restraints and therefore fertile ground for detecting
> problems.  Thus far, I am not aware of any "Grand Unified Score" that
> combines all geometric considerations, but perhaps it is time for one?
>
> Tristan's trivial solution aside, it is actually very hard to make all the
> "geometry" ideal for a real-world fold, and especially difficult to do
> without also screwing up the agreement with density (R factor).  I would
> argue that if you don't have an R factor then you should get one, but I am
> interested in opinions about alternatives.
>
> I.E. What if we could train an AI to predict Rfree by looking at the
> coordinates?
>
> -James Holton
> MAD Scientist
>
> On 12/21/2021 9:25 AM, Pavel Afonine wrote:
>
> Hi Reza,
>
> If you think about it this way... Validation is making sure that the 

Re: [ccp4bb] Validation of structure prediction

[James Holton]

On 1/13/2022 11:14 AM, Tristan Croll wrote:

(please don’t actually do this)


Too late!  I've been doing that for years.  What happens, of course, is 
the "geometry" improves, but the R factors go through the roof. This I 
expect comes as no surprise to anyone who has played with the "weight" 
parameters in refinement, but maybe it should?  What is it about our 
knowledge of chemical bond lengths, angles, and radii that is 
inconsistent with the electron density of macromolecules, but not small 
molecules?  Why do macro-models have a burning desire to leap away from 
the configuration we know they adopt in reality?  If you zoom in on 
those "bad clashes" individually, they don't look like something that is 
supposed to happen. There is a LOT of energy stored up in those little 
springs.  I have a hard time thinking that's for real. The molecule is 
no doubt doing something else and we're just not capturing it properly.  
There is information to be had here, a lot of information.


This is why I too am looking for an all-encompassing "geometry score". 
Right now I'm multiplying other scores together:


score = (1+Clashscore)*sin(worst_omega)*1./(1+worst_rama)*1/(1+worst_rota)
*Cbetadev*worst_nonbond*worst_bond*worst_angle*worst_dihedral*worst_chir*worst_plane

where things like worst_rama is the "%score" given to the worst 
Ramachandran angle by phenix.ramalyze, and worst_bond is the largest 
"residual" reported among all the bonds in the structure by molprobity 
or phenix.geometry_minimization.  For "worst_nonbond" I'm plugging the 
observed and ideal distances into a Leonard-Jones6-12 potential to 
convert it into an "energy" that is always positive.


With x-ray data in hand, I've been multiplying this whole thing by Rwork 
and trying to find clever ways to minimize the product.  Rfree is then, 
as always, the cross-check.


Or does someone have a better idea?

-James Holton
MAD Scientist


On 1/13/2022 11:14 AM, Tristan Croll wrote:
Hard but not impossible - even when you *are* fitting to low-res 
density. See 
https://twitter.com/crolltristan/status/1381258326223290373?s=21 for 
example - no Ramachandran outliers, 1.3% sidechain outliers, 
clashscore of 2... yet multiple regions out of register by anywhere up 
to 15 residues! I never publicly named the structure (although I did 
share my rebuilt model with the authors), but the videos and images in 
that thread should be enough to illustrate the scale of the problem.


And that was *with* a map to fit! Take away the map, and run some MD 
energy minimisation (perhaps with added Ramachandran and rotamer 
restraints), and I think it would be easy to get your model to fool 
most “simple” validation metrics (please don’t actually do this). The 
upshot is that I still think validation of predicted models in the 
absence of at least moderate-resolution experimental data is still a 
major challenge requiring very careful thought.


— Tristan

On 13 Jan 2022, at 18:41, James Holton  wrote:


Agree with Pavel.

Something I think worth adding is a reminder that the MolProbity 
score only looks at bad clashes, ramachandran and rotamer outliers.


MPscore=0.426∗ln(1+clashscore)+0.33∗ln(1+max(0,rota_out−1))+0.25∗ln(1+max(0,rama_iffy−2))+0.5

 It pays no attention whatsoever to twisted peptide bonds, C-beta 
deviations, and, for that matter, bond lengths and bond angles. If 
you tweak your weights right you can get excellent MP scores, but 
horrible "geometry" in the traditional bonds-and-angles sense. The 
logic behind this kind of validation is that normally nonbonds and 
torsions are much softer than bond and angle restraints and therefore 
fertile ground for detecting problems.  Thus far, I am not aware of 
any "Grand Unified Score" that combines all geometric considerations, 
but perhaps it is time for one?


Tristan's trivial solution aside, it is actually very hard to make 
all the "geometry" ideal for a real-world fold, and especially 
difficult to do without also screwing up the agreement with density 
(R factor).  I would argue that if you don't have an R factor then 
you should get one, but I am interested in opinions about alternatives.


I.E. What if we could train an AI to predict Rfree by looking at the 
coordinates?


-James Holton
MAD Scientist

On 12/21/2021 9:25 AM, Pavel Afonine wrote:

Hi Reza,

If you think about it this way... Validation is making sure that the 
model makes sense, data make sense and model-to-data fit make sense, 
then the answer to your question is obvious: in your case you do not 
have experimental data (at least in a way we used to think of it) 
and so then of these three validation items you only have one, 
which, for example, means you don’t have to report things like 
R-factors or completeness in high-resolution shell.


Really, the geometry of an alpha helix does not depend on how you 
determined it: using X-rays or cryo-EM or something else! So, most 
(if not all) model validation tools still apply.


Pavel


On 

Re: [ccp4bb] Validation of structure prediction

[Alexandre Ourjoumtsev]

Hi, James, hi, everybody, 

somehow relevant to your, James, comments : 

>>> MPscore=0.426∗ln(1+clashscore)+0.33∗ln(1+max(0,rota_out−1))+0.25∗ln(1+max(0,rama_iffy−2))+0.5
>>>  
>>> I.E. What if we could train an AI to predict Rfree by looking at the 
>>> coordinates? 

if somebody missed, there is a couple of papers talking about a single / triple 
measure(s) of model quality : 

Shao et al ., 2017, Structure, 25, 458 

Brzezinski et al ., 2020, The FEBS Journal, 287, 2685 

Best regards, 

Sacha Urzhumtsev 

- Le 13 Jan 22, à 19:40, James Holton  a écrit : 

> Agree with Pavel.

> Something I think worth adding is a reminder that the MolProbity score only
> looks at bad clashes, ramachandran and rotamer outliers.

> MPscore=0.426∗ln(1+clashscore)+0.33∗ln(1+max(0,rota_out−1))+0.25∗ln(1+max(0,rama_iffy−2))+0.5

> It pays no attention whatsoever to twisted peptide bonds, C-beta deviations,
> and, for that matter, bond lengths and bond angles. If you tweak your weights
> right you can get excellent MP scores, but horrible "geometry" in the
> traditional bonds-and-angles sense. The logic behind this kind of validation 
> is
> that normally nonbonds and torsions are much softer than bond and angle
> restraints and therefore fertile ground for detecting problems. Thus far, I am
> not aware of any "Grand Unified Score" that combines all geometric
> considerations, but perhaps it is time for one?

> Tristan's trivial solution aside, it is actually very hard to make all the
> "geometry" ideal for a real-world fold, and especially difficult to do without
> also screwing up the agreement with density (R factor). I would argue that if
> you don't have an R factor then you should get one, but I am interested in
> opinions about alternatives.

> I.E. What if we could train an AI to predict Rfree by looking at the
> coordinates?

> -James Holton
> MAD Scientist

> On 12/21/2021 9:25 AM, Pavel Afonine wrote:

>> Hi Reza,

>> If you think about it this way... Validation is making sure that the model 
>> makes
>> sense, data make sense and model-to-data fit make sense, then the answer to
>> your question is obvious: in your case you do not have experimental data (at
>> least in a way we used to think of it) and so then of these three validation
>> items you only have one, which, for example, means you don’t have to report
>> things like R-factors or completeness in high-resolution shell.

>> Really, the geometry of an alpha helix does not depend on how you determined 
>> it:
>> using X-rays or cryo-EM or something else! So, most (if not all) model
>> validation tools still apply.

>> Pavel

>> On Mon, Dec 20, 2021 at 8:10 AM Reza Khayat < [ mailto:rkha...@ccny.cuny.edu 
>> |
>> rkha...@ccny.cuny.edu ] > wrote:

>>> Hi,

>>> Can anyone suggest how to validate a predicted structure? Something similar 
>>> to
>>> wwPDB validation without the need for refinement statistics. I realize this 
>>> is
>>> a strange question given that the geometry of the model is anticipated to be
>>> fine if the structure was predicted by a server that minimizes the geometry 
>>> to
>>> improve its statistics. Nonetheless, the journal has asked me for such a
>>> report. Thanks.

>>> Best wishes,

>>> Reza

>>> Reza Khayat, PhD
>>> Associate Professor
>>> City College of New York
>>> Department of Chemistry and Biochemistry
>>> New York, NY 10031

>>> To unsubscribe from the CCP4BB list, click the following link:
>>> [ https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 |
>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 ]
>> To unsubscribe from the CCP4BB list, click the following link:
>> [ https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 |
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 ]
> To unsubscribe from the CCP4BB list, click the following link:
> [ https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 |
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 ]



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Tristan Croll]

Hard but not impossible - even when you *are* fitting to low-res density. See 
https://twitter.com/crolltristan/status/1381258326223290373?s=21 for example - 
no Ramachandran outliers, 1.3% sidechain outliers, clashscore of 2... yet 
multiple regions out of register by anywhere up to 15 residues! I never 
publicly named the structure (although I did share my rebuilt model with the 
authors), but the videos and images in that thread should be enough to 
illustrate the scale of the problem.

And that was *with* a map to fit! Take away the map, and run some MD energy 
minimisation (perhaps with added Ramachandran and rotamer restraints), and I 
think it would be easy to get your model to fool most “simple” validation 
metrics (please don’t actually do this). The upshot is that I still think 
validation of predicted models in the absence of at least moderate-resolution 
experimental data is still a major challenge requiring very careful thought.

— Tristan

On 13 Jan 2022, at 18:41, James Holton 
mailto:jmhol...@lbl.gov>> wrote:

Agree with Pavel.

Something I think worth adding is a reminder that the MolProbity score only 
looks at bad clashes, ramachandran and rotamer outliers.

MPscore=0.426∗ln(1+clashscore)+0.33∗ln(1+max(0,rota_out−1))+0.25∗ln(1+max(0,rama_iffy−2))+0.5

 It pays no attention whatsoever to twisted peptide bonds, C-beta deviations, 
and, for that matter, bond lengths and bond angles. If you tweak your weights 
right you can get excellent MP scores, but horrible "geometry" in the 
traditional bonds-and-angles sense. The logic behind this kind of validation is 
that normally nonbonds and torsions are much softer than bond and angle 
restraints and therefore fertile ground for detecting problems.  Thus far, I am 
not aware of any "Grand Unified Score" that combines all geometric 
considerations, but perhaps it is time for one?

Tristan's trivial solution aside, it is actually very hard to make all the 
"geometry" ideal for a real-world fold, and especially difficult to do without 
also screwing up the agreement with density (R factor).  I would argue that if 
you don't have an R factor then you should get one, but I am interested in 
opinions about alternatives.

I.E. What if we could train an AI to predict Rfree by looking at the 
coordinates?

-James Holton
MAD Scientist

On 12/21/2021 9:25 AM, Pavel Afonine wrote:
Hi Reza,
If you think about it this way... Validation is making sure that the model 
makes sense, data make sense and model-to-data fit make sense, then the answer 
to your question is obvious: in your case you do not have experimental data (at 
least in a way we used to think of it) and so then of these three validation 
items you only have one, which, for example, means you don’t have to report 
things like R-factors or completeness in high-resolution shell.
Really, the geometry of an alpha helix does not depend on how you determined 
it: using X-rays or cryo-EM or something else! So, most (if not all) model 
validation tools still apply.
Pavel

On Mon, Dec 20, 2021 at 8:10 AM Reza Khayat 
mailto:rkha...@ccny.cuny.edu>> wrote:

Hi,

Can anyone suggest how to validate a predicted structure? Something similar to 
wwPDB validation without the need for refinement statistics. I realize this is 
a strange question given that the geometry of the model is anticipated to be 
fine if the structure was predicted by a server that minimizes the geometry to 
improve its statistics. Nonetheless, the journal has asked me for such a 
report. Thanks.

Best wishes,

Reza


Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[James Holton]

Agree with Pavel.

Something I think worth adding is a reminder that the MolProbity score 
only looks at bad clashes, ramachandran and rotamer outliers.


MPscore=0.426∗ln(1+clashscore)+0.33∗ln(1+max(0,rota_out−1))+0.25∗ln(1+max(0,rama_iffy−2))+0.5

 It pays no attention whatsoever to twisted peptide bonds, C-beta 
deviations, and, for that matter, bond lengths and bond angles. If you 
tweak your weights right you can get excellent MP scores, but horrible 
"geometry" in the traditional bonds-and-angles sense. The logic behind 
this kind of validation is that normally nonbonds and torsions are much 
softer than bond and angle restraints and therefore fertile ground for 
detecting problems.  Thus far, I am not aware of any "Grand Unified 
Score" that combines all geometric considerations, but perhaps it is 
time for one?


Tristan's trivial solution aside, it is actually very hard to make all 
the "geometry" ideal for a real-world fold, and especially difficult to 
do without also screwing up the agreement with density (R factor).  I 
would argue that if you don't have an R factor then you should get one, 
but I am interested in opinions about alternatives.


I.E. What if we could train an AI to predict Rfree by looking at the 
coordinates?


-James Holton
MAD Scientist

On 12/21/2021 9:25 AM, Pavel Afonine wrote:

Hi Reza,

If you think about it this way... Validation is making sure that the 
model makes sense, data make sense and model-to-data fit make sense, 
then the answer to your question is obvious: in your case you do not 
have experimental data (at least in a way we used to think of it) and 
so then of these three validation items you only have one, which, for 
example, means you don’t have to report things like R-factors or 
completeness in high-resolution shell.


Really, the geometry of an alpha helix does not depend on how you 
determined it: using X-rays or cryo-EM or something else! So, most (if 
not all) model validation tools still apply.


Pavel


On Mon, Dec 20, 2021 at 8:10 AM Reza Khayat  wrote:

Hi,


Can anyone suggest how to validate a predicted structure?
Something similar to wwPDB validation without the need for
refinement statistics. I realize this is a strange question given
that the geometry of the model is anticipated to be fine if the
structure was predicted by a server that minimizes the geometry to
improve its statistics. Nonetheless, the journal has asked me for
such a report. Thanks.


Best wishes,

Reza


Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 







To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Mukesh Kumar]

Hi,
try SAVES online tool,and ProSa.

On Mon, 20 Dec, 2021, 9:40 pm Reza Khayat,  wrote:

> ​Hi,
>
>
> Can anyone suggest how to validate a predicted structure? Something
> similar to wwPDB validation without the need for refinement statistics. I
> realize this is a strange question given that the geometry of the model is
> anticipated to be fine if the structure was predicted by a server that
> minimizes the geometry to improve its statistics. Nonetheless, the journal
> has asked me for such a report. Thanks.
>
>
> Best wishes,
>
> Reza
>
>
> Reza Khayat, PhD
> Associate Professor
> City College of New York
> Department of Chemistry and Biochemistry
> New York, NY 10031
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Pavel Afonine]

Hi Reza,

If you think about it this way... Validation is making sure that the model
makes sense, data make sense and model-to-data fit make sense, then the
answer to your question is obvious: in your case you do not have
experimental data (at least in a way we used to think of it) and so then of
these three validation items you only have one, which, for example, means
you don’t have to report things like R-factors or completeness in
high-resolution shell.

Really, the geometry of an alpha helix does not depend on how you
determined it: using X-rays or cryo-EM or something else! So, most (if not
all) model validation tools still apply.

Pavel

On Mon, Dec 20, 2021 at 8:10 AM Reza Khayat  wrote:

> Hi,
>
>
> Can anyone suggest how to validate a predicted structure? Something
> similar to wwPDB validation without the need for refinement statistics. I
> realize this is a strange question given that the geometry of the model is
> anticipated to be fine if the structure was predicted by a server that
> minimizes the geometry to improve its statistics. Nonetheless, the journal
> has asked me for such a report. Thanks.
>
>
> Best wishes,
>
> Reza
>
>
> Reza Khayat, PhD
> Associate Professor
> City College of New York
> Department of Chemistry and Biochemistry
> New York, NY 10031
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Javier Gonzalez]

Hello,
I think it is perfectly "valid" to run AlphaFold2 / RosettaFold / etc.
models through geometry validation servers (aside from reporting their
intrinsic quality indicators like TM-score and pLDDT distribution, as
mentioned above by Randy Read). Since most experimental structures have a
small percent of Ramachandran outlier residues, a good model ought to have
some too. I´ve run into crystal structures that display Ramachandran
outliers very well supported by the electron density, and that "strain" in
the backbone later on turned out to be favorable for the catalytic
mechanism, and so on... Therefore, if AF2 or RosettaFold algorithms are as
good as they are advertised, they should be able to capture and predict
those outliers too... and it would be very interesting to find out if they
do! A model with a perfect Ramachandran plot is probably as reliable as a
low resolution structure resulting from a heavily restrained geometry
refinement.
By the way this is true only for the polypeptide backbone conformation, I
don't think any algorithm can predict side chain conformations yet (indeed
AF2 offers an optional sidechain energy minimisation to tidy up side chains
in the final models, a least in ColabFold by Sergey Ovchinnikov).
Happy holidays,
Javier

On Tue, Dec 21, 2021 at 10:29 AM Reza Khayat  wrote:

> Hi,
>
>
> Thank you for the help. I've addressed some of the concerns raised here in
> another thread. "Validation" referred to checking geometric parameters;
> however, outstanding geometric parameters do not indicate a structure that
> is comparable to an experimentally determined structure. The structures
> were predicted with the Robetta server and all have, as expected, geometry
> better than most experimental structures.
>
>
> Best wishes,
> Reza
>
>
> Reza Khayat, PhD
> Associate Professor
> City College of New York
> Department of Chemistry and Biochemistry
> New York, NY 10031
> --
> *From:* CCP4 bulletin board  on behalf of Krieger,
> James M 
> *Sent:* Tuesday, December 21, 2021 7:14 AM
> *To:* CCP4BB@JISCMAIL.AC.UK
> *Subject:* [EXTERNAL] Re: [ccp4bb] Validation of structure prediction
>
> There are also dedicated homology modelling validation tools such as
> ANOLEA (ANOLEA (Atomic Non-Local Environment Assessment) (melolab.org)
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__melolab.org_anolea_=DwMF-g=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fzEbD5_hcb9Up0=wygUyYLV7I0tm87EBKcsmtMhtRx3xWnT36HOD04_BCo=Y379nBTcaulXSuY9Y7ZQJsQSkQtJGpI8jj-fuT7THVU=>
> ).
>
> Best wishes
> James
> --
> *From:* CCP4 bulletin board  on behalf of Nicholas
> Clark 
> *Sent:* Tuesday, December 21, 2021 11:57 AM
> *To:* CCP4BB@JISCMAIL.AC.UK 
> *Subject:* Re: [ccp4bb] Validation of structure prediction
>
> Reza,
>
> Thus far, it seems we’ve all assumed this was an AlphaFold or RobettaFold
> model. If this is not indeed the case, it may be worthwhile to “validate”
> your mode by running your sequence through one of these two and using the
> validation from them.
>
> The AlphaFold DB can be found here, with a number of predicted structures:
>
> https://alphafold.ebi.ac.uk
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__nam12.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Falphafold.ebi.ac.uk-252F-26data-3D04-257C01-257Ckriegerj-2540PITT.EDU-257Ca05aa26f114e4fb7e6c108d9c479140b-257C9ef9f489e0a04eeb87cc3a526112fd0d-257C1-257C0-257C637756847599764985-257CUnknown-257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0-253D-257C3000-26sdata-3D1szS78fj9859h8AR0Mt5YtgUXeW1JmsB2YU0j5Hhy5U-253D-26reserved-3D0=DwMF-g=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fzEbD5_hcb9Up0=wygUyYLV7I0tm87EBKcsmtMhtRx3xWnT36HOD04_BCo=OM7PDWl19Mz3PGjngC_uwF797ZeTV5gVNCg5RXNrUV8=>
>
> The AlphaFold colab can be found here, although the prediction is not as
> good as AlphaFold 2, as it does not use templates:
>
>
> https://colab.research.google.com/github/deepmind/alphafold/blob/main/notebooks/AlphaFold.ipynb
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__nam12.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fcolab.research.google.com-252Fgithub-252Fdeepmind-252Falphafold-252Fblob-252Fmain-252Fnotebooks-252FAlphaFold.ipynb-26data-3D04-257C01-257Ckriegerj-2540PITT.EDU-257Ca05aa26f114e4fb7e6c108d9c479140b-257C9ef9f489e0a04eeb87cc3a526112fd0d-257C1-257C0-257C637756847599764985-257CUnknown-257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0-253D-257C3000-26sdata-3Dd323My3Ix-252BQwDW3D7ZCe6RTlY7wvMqXl92ZGRiIiMSo-253D-26reserved-3D0=DwMF-g=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fz

Re: [ccp4bb] Validation of structure prediction

[Reza Khayat]

Hi,


Thank you for the help. I've addressed some of the concerns raised here in 
another thread. "Validation" referred to checking geometric parameters; 
however, outstanding geometric parameters do not indicate a structure that is 
comparable to an experimentally determined structure. The structures were 
predicted with the Robetta server and all have, as expected, geometry better 
than most experimental structures.


Best wishes,
Reza


Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031

From: CCP4 bulletin board  on behalf of Krieger, James M 

Sent: Tuesday, December 21, 2021 7:14 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [EXTERNAL] Re: [ccp4bb] Validation of structure prediction

There are also dedicated homology modelling validation tools such as ANOLEA 
(ANOLEA (Atomic Non-Local Environment Assessment) 
(melolab.org)<https://urldefense.proofpoint.com/v2/url?u=http-3A__melolab.org_anolea_=DwMF-g=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fzEbD5_hcb9Up0=wygUyYLV7I0tm87EBKcsmtMhtRx3xWnT36HOD04_BCo=Y379nBTcaulXSuY9Y7ZQJsQSkQtJGpI8jj-fuT7THVU=>).

Best wishes
James

From: CCP4 bulletin board  on behalf of Nicholas Clark 

Sent: Tuesday, December 21, 2021 11:57 AM
To: CCP4BB@JISCMAIL.AC.UK 
Subject: Re: [ccp4bb] Validation of structure prediction

Reza,

Thus far, it seems we’ve all assumed this was an AlphaFold or RobettaFold 
model. If this is not indeed the case, it may be worthwhile to “validate” your 
mode by running your sequence through one of these two and using the validation 
from them.

The AlphaFold DB can be found here, with a number of predicted structures:

https://alphafold.ebi.ac.uk<https://urldefense.proofpoint.com/v2/url?u=https-3A__nam12.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Falphafold.ebi.ac.uk-252F-26data-3D04-257C01-257Ckriegerj-2540PITT.EDU-257Ca05aa26f114e4fb7e6c108d9c479140b-257C9ef9f489e0a04eeb87cc3a526112fd0d-257C1-257C0-257C637756847599764985-257CUnknown-257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0-253D-257C3000-26sdata-3D1szS78fj9859h8AR0Mt5YtgUXeW1JmsB2YU0j5Hhy5U-253D-26reserved-3D0=DwMF-g=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fzEbD5_hcb9Up0=wygUyYLV7I0tm87EBKcsmtMhtRx3xWnT36HOD04_BCo=OM7PDWl19Mz3PGjngC_uwF797ZeTV5gVNCg5RXNrUV8=>

The AlphaFold colab can be found here, although the prediction is not as good 
as AlphaFold 2, as it does not use templates:

https://colab.research.google.com/github/deepmind/alphafold/blob/main/notebooks/AlphaFold.ipynb<https://urldefense.proofpoint.com/v2/url?u=https-3A__nam12.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fcolab.research.google.com-252Fgithub-252Fdeepmind-252Falphafold-252Fblob-252Fmain-252Fnotebooks-252FAlphaFold.ipynb-26data-3D04-257C01-257Ckriegerj-2540PITT.EDU-257Ca05aa26f114e4fb7e6c108d9c479140b-257C9ef9f489e0a04eeb87cc3a526112fd0d-257C1-257C0-257C637756847599764985-257CUnknown-257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0-253D-257C3000-26sdata-3Dd323My3Ix-252BQwDW3D7ZCe6RTlY7wvMqXl92ZGRiIiMSo-253D-26reserved-3D0=DwMF-g=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fzEbD5_hcb9Up0=wygUyYLV7I0tm87EBKcsmtMhtRx3xWnT36HOD04_BCo=iJvvBod4oNFHS9YMaLMNdy0edv1eh47A2txWBV5IuoQ=>


RobettaFold can be found here:
https://robetta.bakerlab.org<https://urldefense.proofpoint.com/v2/url?u=https-3A__nam12.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Frobetta.bakerlab.org-252F-26data-3D04-257C01-257Ckriegerj-2540PITT.EDU-257Ca05aa26f114e4fb7e6c108d9c479140b-257C9ef9f489e0a04eeb87cc3a526112fd0d-257C1-257C0-257C637756847599764985-257CUnknown-257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0-253D-257C3000-26sdata-3DT3foQ-252FDRiOinSJjqeKwkPkw1UtSXc4PvyJRCTSwtUiI-253D-26reserved-3D0=DwMF-g=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fzEbD5_hcb9Up0=wygUyYLV7I0tm87EBKcsmtMhtRx3xWnT36HOD04_BCo=8b0eA8ZDF8wlvyfoyNxzxxmGN0T58L9WsgHqEaxUoZY=>

Best,

Nick Clark

On Tue, Dec 21, 2021 at 6:20 AM Randy John Read 
mailto:rj...@cam.ac.uk>> wrote:
Just to add one point that I don’t think I’ve seen yet. If what the referee 
wants is a data-free assessment of the expected quality of the model, I think 
that the best assessment at the moment is the one done by AlphaFold2 (or indeed 
RoseTTAFold if you’re using one of their models). The machine-learning 
algorithm is pretty good at assessing how good of a job it has done, either 
overall (predicted TM score) or locally (predicted lDDT score for AlphaFold2 or 
predicted RMSD for RoseTTAFold). There are cases of false positives (poor 
models that think they’re good) and false negatives (good models that think 
they’re bad), but these are in the minority from what I’ve seen so

Re: [ccp4bb] [EXTERNAL] [ccp4bb] Fwd: [ccp4bb] Validation of structure prediction

[Reza Khayat]

​Dear all,


Thank you for the responses. My question regarding validation was not directed 
at weather the prediction was correct (i.e. sufficiently comparable to the 
experimentally determined structure). I realize this question in itself is not 
easily answered. I also agree with what everyone has written below. My question 
was regarding the geometry of the structure. I also understand that outstanding 
geometry does not indicate an accurately predicted structure -the helix example 
indicated by Tristian. Nonetheless, the server links sent earlier are extremely 
helpful.


Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031

From: CCP4 bulletin board  on behalf of F.Xavier 
Gomis-Rüth 
Sent: Tuesday, December 21, 2021 5:04 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [EXTERNAL] [ccp4bb] Fwd: [ccp4bb] Validation of structure prediction

Dear all,
this is by far not the general case in our hands. Depending on which AlphaFold 
protocol is used, the resulting models have locally disfavourable
geometries–including clashes–, impossible chain crossovers, etc. I would 
definitively recommend everybody to go through the model in detail and perform
a final geometry minimization with Coot and/or Phenix/Refmac. And in these 
cases, general geometry validation as provided by MolProbity
provides a final proof of the computational model.
Best,
Xavier


 Forwarded Message 
Subject:Re: [ccp4bb] Validation of structure prediction
Date:   Tue, 21 Dec 2021 09:43:37 +
From:   Vollmar, Melanie (DLSLtd,RAL,LSCI) 
<64fe7ccc6b4d-dmarc-requ...@jiscmail.ac.uk><mailto:64fe7ccc6b4d-dmarc-requ...@jiscmail.ac.uk>
Reply-To:   Vollmar, Melanie (DLSLtd,RAL,LSCI) 
<mailto:melanie.voll...@diamond.ac.uk>
To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>


Tristan is spot on. All the predicted structures have near perfect geometry, so 
commonly used validation tools like MolProbity can no longer be applied.

What you need to consider is biological relevance of the predicted model. Does 
the model correctly reflect residue arrangement in the active site? Are domains 
in correct relative orientation to allow for interactions and movements, 
perhaps found by some other assay? Is there appropriate room to fit a 
ligand/cofactor? Are transmembrane helices, if there are any, correctly found?

You need to map the knowledge you have of your protein to the structure and see 
if the atom positions and what you know support each other.

Cheers

M

From: CCP4 bulletin board <mailto:CCP4BB@JISCMAIL.AC.UK> 
on behalf of Tristan Croll <mailto:ti...@cam.ac.uk>
Sent: 21 December 2021 08:28
To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK> 
<mailto:CCP4BB@JISCMAIL.AC.UK>
Subject: Re: [ccp4bb] Validation of structure prediction

I agree with Dale. Tools like MolProbity are not the right approach to 
validating a structure prediction. To understand why, just consider that all 
you need to do to get a perfect MolProbity score is predict every structure as 
a single long alpha helix with ideal rotamers, with a kink at each proline.

To validate a predicted structure will require a completely different toolset - 
one that I’m not sure fully exists yet.

— Tristan

> On 20 Dec 2021, at 18:47, Dale Tronrud 
> <mailto:de...@daletronrud.com> wrote:
>
>   I don't see any reason to believe that software designed to validate 
> crystallographic or NMR models would have any utility validating AlphaFold 
> predicted models.  Doesn't the prediction software already ensure that all 
> the indicators used by Molprobity are obeyed?  I'm afraid that the tools to 
> validate any new technique must be designed specifically for that technique. 
> (And when they become available they will be useless for validating 
> crystallographic models!)
>
> Dale E. Tronrud
>
>> On 12/20/2021 10:28 AM, Nicholas Clark wrote:
>> The Molprobity server can be run online and only requires the coordinates in 
>> PDB format: 
>> http://molprobity.biochem.duke.edu/<https://urldefense.proofpoint.com/v2/url?u=http-3A__molprobity.biochem.duke.edu_=DwMDaQ=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fzEbD5_hcb9Up0=Ore7pZl_g57-Bha3m6bv3ayt12QpXPz1lbBlJYIx0rY=ekPRcBIEvpMpSOCQ3Iwa3WeIz5hew6AEXPmRUbWF9eg=>
>>  
>> <http://molprobity.biochem.duke.edu/<https://urldefense.proofpoint.com/v2/url?u=http-3A__molprobity.biochem.duke.edu_=DwMDaQ=4NmamNZG3KTnUCoC6InoLJ6KV1tbVKrkZXHRwtIMGmo=1DzJFW0v6TgEhkW1gy_-ke-RbtvS1fzEbD5_hcb9Up0=Ore7pZl_g57-Bha3m6bv3ayt12QpXPz1lbBlJYIx0rY=ekPRcBIEvpMpSOCQ3Iwa3WeIz5hew6AEXPmRUbWF9eg=>>.
>> Best,
>> Nick Clark
>> On Mon, Dec 20, 2021 at 11:10 AM Reza Khayat 
>> mailto:rkha...@ccny.cuny.edu> 
>

Re: [ccp4bb] Validation of structure prediction

[Krieger, James M]

There are also dedicated homology modelling validation tools such as ANOLEA 
(ANOLEA (Atomic Non-Local Environment Assessment) 
(melolab.org)<http://melolab.org/anolea/>).

Best wishes
James

From: CCP4 bulletin board  on behalf of Nicholas Clark 

Sent: Tuesday, December 21, 2021 11:57 AM
To: CCP4BB@JISCMAIL.AC.UK 
Subject: Re: [ccp4bb] Validation of structure prediction

Reza,

Thus far, it seems we’ve all assumed this was an AlphaFold or RobettaFold 
model. If this is not indeed the case, it may be worthwhile to “validate” your 
mode by running your sequence through one of these two and using the validation 
from them.

The AlphaFold DB can be found here, with a number of predicted structures:

https://alphafold.ebi.ac.uk<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Falphafold.ebi.ac.uk%2F=04%7C01%7Ckriegerj%40PITT.EDU%7Ca05aa26f114e4fb7e6c108d9c479140b%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637756847599764985%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=1szS78fj9859h8AR0Mt5YtgUXeW1JmsB2YU0j5Hhy5U%3D=0>

The AlphaFold colab can be found here, although the prediction is not as good 
as AlphaFold 2, as it does not use templates:

https://colab.research.google.com/github/deepmind/alphafold/blob/main/notebooks/AlphaFold.ipynb<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcolab.research.google.com%2Fgithub%2Fdeepmind%2Falphafold%2Fblob%2Fmain%2Fnotebooks%2FAlphaFold.ipynb=04%7C01%7Ckriegerj%40PITT.EDU%7Ca05aa26f114e4fb7e6c108d9c479140b%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637756847599764985%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=d323My3Ix%2BQwDW3D7ZCe6RTlY7wvMqXl92ZGRiIiMSo%3D=0>


RobettaFold can be found here:
https://robetta.bakerlab.org<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Frobetta.bakerlab.org%2F=04%7C01%7Ckriegerj%40PITT.EDU%7Ca05aa26f114e4fb7e6c108d9c479140b%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637756847599764985%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=T3foQ%2FDRiOinSJjqeKwkPkw1UtSXc4PvyJRCTSwtUiI%3D=0>

Best,

Nick Clark

On Tue, Dec 21, 2021 at 6:20 AM Randy John Read 
mailto:rj...@cam.ac.uk>> wrote:
Just to add one point that I don’t think I’ve seen yet. If what the referee 
wants is a data-free assessment of the expected quality of the model, I think 
that the best assessment at the moment is the one done by AlphaFold2 (or indeed 
RoseTTAFold if you’re using one of their models). The machine-learning 
algorithm is pretty good at assessing how good of a job it has done, either 
overall (predicted TM score) or locally (predicted lDDT score for AlphaFold2 or 
predicted RMSD for RoseTTAFold). There are cases of false positives (poor 
models that think they’re good) and false negatives (good models that think 
they’re bad), but these are in the minority from what I’ve seen so far.

Of the tools mentioned earlier, I think ProQ2 is the only one that is really 
trained to assess predicted model quality, but I suspect it’s not as good at 
assessing the quality of the latest generation of models as the tools 
generating those models are.

Best wishes,

Randy Read

> On 21 Dec 2021, at 11:12, Kay Diederichs 
> mailto:kay.diederi...@uni-konstanz.de>> wrote:
>
> Hi Reza,
>
> the term "validation" as used by e.g. crystallographers - namely by checking 
> geometric parameters of a structure derived from experiment(s) - is 
> euphemistic since realistic geometry is a required but not sufficient 
> property of a model - it can be completely wrong even if it has good 
> geometry. This type of validation should rather be called "checked for 
> geometric consistency" or similar.
>
> In general, the validation of a prediction should be performed by 
> experiment(s). Ideally, that would be X-ray structure solution or cryo-EM or 
> NMR. But it could also be e.g. a set of binding or cross-linking experiments, 
> with suitable positive and negative controls.
>
> That a computational prediction can be "validated" by another calculation may 
> be possible if the prediction is based on a set of facts that is disjoint 
> from that of the "validation calculation". But this is not the case e.g. for 
> models from AlphaFold or RoseTTaFold; MolProbility or similar programs could 
> at most indicate that the prediction has failed, but not that the prediction 
> is correct. Experiments are needed for this - and even these could be 
> inconclusive.
>
> Best wishes,
> Kay
>
> On Mon, 20 Dec 2021 16:10:02 +, Reza Khayat 
> mailto:rkha...@ccny.cuny.edu>> wrote:
>
>> ?Hi,
>>
>> Can anyone suggest how to validate a predicted structure? Something similar

Re: [ccp4bb] Validation of structure prediction

[Nicholas Clark]

Reza,

Thus far, it seems we’ve all assumed this was an AlphaFold or RobettaFold
model. If this is not indeed the case, it may be worthwhile to “validate”
your mode by running your sequence through one of these two and using the
validation from them.

The AlphaFold DB can be found here, with a number of predicted structures:

https://alphafold.ebi.ac.uk

The AlphaFold colab can be found here, although the prediction is not as
good as AlphaFold 2, as it does not use templates:

https://colab.research.google.com/github/deepmind/alphafold/blob/main/notebooks/AlphaFold.ipynb


RobettaFold can be found here:
https://robetta.bakerlab.org

Best,

Nick Clark

On Tue, Dec 21, 2021 at 6:20 AM Randy John Read  wrote:

> Just to add one point that I don’t think I’ve seen yet. If what the
> referee wants is a data-free assessment of the expected quality of the
> model, I think that the best assessment at the moment is the one done by
> AlphaFold2 (or indeed RoseTTAFold if you’re using one of their models). The
> machine-learning algorithm is pretty good at assessing how good of a job it
> has done, either overall (predicted TM score) or locally (predicted lDDT
> score for AlphaFold2 or predicted RMSD for RoseTTAFold). There are cases of
> false positives (poor models that think they’re good) and false negatives
> (good models that think they’re bad), but these are in the minority from
> what I’ve seen so far.
>
> Of the tools mentioned earlier, I think ProQ2 is the only one that is
> really trained to assess predicted model quality, but I suspect it’s not as
> good at assessing the quality of the latest generation of models as the
> tools generating those models are.
>
> Best wishes,
>
> Randy Read
>
> > On 21 Dec 2021, at 11:12, Kay Diederichs 
> wrote:
> >
> > Hi Reza,
> >
> > the term "validation" as used by e.g. crystallographers - namely by
> checking geometric parameters of a structure derived from experiment(s) -
> is euphemistic since realistic geometry is a required but not sufficient
> property of a model - it can be completely wrong even if it has good
> geometry. This type of validation should rather be called "checked for
> geometric consistency" or similar.
> >
> > In general, the validation of a prediction should be performed by
> experiment(s). Ideally, that would be X-ray structure solution or cryo-EM
> or NMR. But it could also be e.g. a set of binding or cross-linking
> experiments, with suitable positive and negative controls.
> >
> > That a computational prediction can be "validated" by another
> calculation may be possible if the prediction is based on a set of facts
> that is disjoint from that of the "validation calculation". But this is not
> the case e.g. for models from AlphaFold or RoseTTaFold; MolProbility or
> similar programs could at most indicate that the prediction has failed, but
> not that the prediction is correct. Experiments are needed for this - and
> even these could be inconclusive.
> >
> > Best wishes,
> > Kay
> >
> > On Mon, 20 Dec 2021 16:10:02 +, Reza Khayat 
> wrote:
> >
> >> ?Hi,
> >>
> >> Can anyone suggest how to validate a predicted structure? Something
> similar to wwPDB validation without the need for refinement statistics. I
> realize this is a strange question given that the geometry of the model is
> anticipated to be fine if the structure was predicted by a server that
> minimizes the geometry to improve its statistics. Nonetheless, the journal
> has asked me for such a report. Thanks.
> >>
> >> Best wishes,
> >>
> >> Reza
> >>
> >>
> >> Reza Khayat, PhD
> >> Associate Professor
> >> City College of New York
> >> Department of Chemistry and Biochemistry
> >> New York, NY 10031
> >>
> >> 
> >>
> >> To unsubscribe from the CCP4BB list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> >>
> >> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>
> >
> > 
> >
> > To unsubscribe from the CCP4BB list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> >
> > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> available at https://www.jiscmail.ac.uk/policyandsecurity/
>
> -
> Randy J. Read
> Department of Haematology, University of Cambridge
> Cambridge Institute for Medical Research Tel: +44 1223 336500
> The Keith Peters Building   Fax: +44 1223
> 336827
> Hills Road   E-mail:
> rj...@cam.ac.uk
> Cambridge CB2 0XY, U.K.
> www-structmed.cimr.cam.ac.uk
>
>
> 

Re: [ccp4bb] Validation of structure prediction

[Randy John Read]

Just to add one point that I don’t think I’ve seen yet. If what the referee 
wants is a data-free assessment of the expected quality of the model, I think 
that the best assessment at the moment is the one done by AlphaFold2 (or indeed 
RoseTTAFold if you’re using one of their models). The machine-learning 
algorithm is pretty good at assessing how good of a job it has done, either 
overall (predicted TM score) or locally (predicted lDDT score for AlphaFold2 or 
predicted RMSD for RoseTTAFold). There are cases of false positives (poor 
models that think they’re good) and false negatives (good models that think 
they’re bad), but these are in the minority from what I’ve seen so far.

Of the tools mentioned earlier, I think ProQ2 is the only one that is really 
trained to assess predicted model quality, but I suspect it’s not as good at 
assessing the quality of the latest generation of models as the tools 
generating those models are.

Best wishes,

Randy Read

> On 21 Dec 2021, at 11:12, Kay Diederichs  
> wrote:
> 
> Hi Reza,
> 
> the term "validation" as used by e.g. crystallographers - namely by checking 
> geometric parameters of a structure derived from experiment(s) - is 
> euphemistic since realistic geometry is a required but not sufficient 
> property of a model - it can be completely wrong even if it has good 
> geometry. This type of validation should rather be called "checked for 
> geometric consistency" or similar.
> 
> In general, the validation of a prediction should be performed by 
> experiment(s). Ideally, that would be X-ray structure solution or cryo-EM or 
> NMR. But it could also be e.g. a set of binding or cross-linking experiments, 
> with suitable positive and negative controls. 
> 
> That a computational prediction can be "validated" by another calculation may 
> be possible if the prediction is based on a set of facts that is disjoint 
> from that of the "validation calculation". But this is not the case e.g. for 
> models from AlphaFold or RoseTTaFold; MolProbility or similar programs could 
> at most indicate that the prediction has failed, but not that the prediction 
> is correct. Experiments are needed for this - and even these could be 
> inconclusive.
> 
> Best wishes,
> Kay
> 
> On Mon, 20 Dec 2021 16:10:02 +, Reza Khayat  wrote:
> 
>> ?Hi,
>> 
>> Can anyone suggest how to validate a predicted structure? Something similar 
>> to wwPDB validation without the need for refinement statistics. I realize 
>> this is a strange question given that the geometry of the model is 
>> anticipated to be fine if the structure was predicted by a server that 
>> minimizes the geometry to improve its statistics. Nonetheless, the journal 
>> has asked me for such a report. Thanks.
>> 
>> Best wishes,
>> 
>> Reza
>> 
>> 
>> Reza Khayat, PhD
>> Associate Professor
>> City College of New York
>> Department of Chemistry and Biochemistry
>> New York, NY 10031
>> 
>> 
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>> 
>> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing 
>> list hosted by www.jiscmail.ac.uk, terms & conditions are available at 
>> https://www.jiscmail.ac.uk/policyandsecurity/
>> 
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> 
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing 
> list hosted by www.jiscmail.ac.uk, terms & conditions are available at 
> https://www.jiscmail.ac.uk/policyandsecurity/

-
Randy J. Read
Department of Haematology, University of Cambridge
Cambridge Institute for Medical Research Tel: +44 1223 336500
The Keith Peters Building   Fax: +44 1223 336827
Hills Road   E-mail: 
rj...@cam.ac.uk
Cambridge CB2 0XY, U.K.  
www-structmed.cimr.cam.ac.uk




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Kay Diederichs]

Hi Reza,

the term "validation" as used by e.g. crystallographers - namely by checking 
geometric parameters of a structure derived from experiment(s) - is euphemistic 
since realistic geometry is a required but not sufficient property of a model - 
it can be completely wrong even if it has good geometry. This type of 
validation should rather be called "checked for geometric consistency" or 
similar.

In general, the validation of a prediction should be performed by 
experiment(s). Ideally, that would be X-ray structure solution or cryo-EM or 
NMR. But it could also be e.g. a set of binding or cross-linking experiments, 
with suitable positive and negative controls. 

That a computational prediction can be "validated" by another calculation may 
be possible if the prediction is based on a set of facts that is disjoint from 
that of the "validation calculation". But this is not the case e.g. for models 
from AlphaFold or RoseTTaFold; MolProbility or similar programs could at most 
indicate that the prediction has failed, but not that the prediction is 
correct. Experiments are needed for this - and even these could be inconclusive.

Best wishes,
Kay

On Mon, 20 Dec 2021 16:10:02 +, Reza Khayat  wrote:

>?Hi,
>
>Can anyone suggest how to validate a predicted structure? Something similar to 
>wwPDB validation without the need for refinement statistics. I realize this is 
>a strange question given that the geometry of the model is anticipated to be 
>fine if the structure was predicted by a server that minimizes the geometry to 
>improve its statistics. Nonetheless, the journal has asked me for such a 
>report. Thanks.
>
>Best wishes,
>
>Reza
>
>
>Reza Khayat, PhD
>Associate Professor
>City College of New York
>Department of Chemistry and Biochemistry
>New York, NY 10031
>
>
>
>To unsubscribe from the CCP4BB list, click the following link:
>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>
>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing 
>list hosted by www.jiscmail.ac.uk, terms & conditions are available at 
>https://www.jiscmail.ac.uk/policyandsecurity/
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Fwd: [ccp4bb] Validation of structure prediction

[Vollmar, Melanie (DLSLtd,RAL,LSCI)]

I also should have added that if you use a predicted structure and you run MR 
with it and then modify it to fit the data of your novel structure then, for 
sure, MolProbity applies or other tools as you will find in the various 
packages and on the PDB site.

The most current predicted structures usually do have an energy minimisation 
step at the end but nevertheless you can always add it yourself as Xavier 
pointed out.

In general, your prediction is a very good, educated guess on what your protein 
might look like. However,  the algorithm has no clue about your crystallisation 
condition or even the true biological environment in the cell and hence cannot 
take this chemical information into account when arranging the atoms. The 
algorithm also doesn't know that you have a membrane protein or different 
domains that need to be arranged relative to each other. The artefacts 
mentioned by Xavier are most likely a result for this lack of knowledge by the 
algorithm. Or just poor performance after all, even the best predictor can't do 
magic...

Look at the pLDDT score for your prediction, a local measure for the confidence 
with which each residue was placed into 3D space. A low score (<50) means high 
uncertainty and these residues should be removed anyway.

So, open your model in Coot, look at it and remove the rubbish...

M

From: CCP4 bulletin board  on behalf of F.Xavier 
Gomis-Rüth 
Sent: 21 December 2021 10:04
To: CCP4BB@JISCMAIL.AC.UK 
Subject: [ccp4bb] Fwd: [ccp4bb] Validation of structure prediction

Dear all,
this is by far not the general case in our hands. Depending on which AlphaFold 
protocol is used, the resulting models have locally disfavourable
geometries–including clashes–, impossible chain crossovers, etc. I would 
definitively recommend everybody to go through the model in detail and perform
a final geometry minimization with Coot and/or Phenix/Refmac. And in these 
cases, general geometry validation as provided by MolProbity
provides a final proof of the computational model.
Best,
Xavier


 Forwarded Message 
Subject:Re: [ccp4bb] Validation of structure prediction
Date:   Tue, 21 Dec 2021 09:43:37 +
From:   Vollmar, Melanie (DLSLtd,RAL,LSCI) 
<64fe7ccc6b4d-dmarc-requ...@jiscmail.ac.uk><mailto:64fe7ccc6b4d-dmarc-requ...@jiscmail.ac.uk>
Reply-To:   Vollmar, Melanie (DLSLtd,RAL,LSCI) 
<mailto:melanie.voll...@diamond.ac.uk>
To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>


Tristan is spot on. All the predicted structures have near perfect geometry, so 
commonly used validation tools like MolProbity can no longer be applied.

What you need to consider is biological relevance of the predicted model. Does 
the model correctly reflect residue arrangement in the active site? Are domains 
in correct relative orientation to allow for interactions and movements, 
perhaps found by some other assay? Is there appropriate room to fit a 
ligand/cofactor? Are transmembrane helices, if there are any, correctly found?

You need to map the knowledge you have of your protein to the structure and see 
if the atom positions and what you know support each other.

Cheers

M

From: CCP4 bulletin board <mailto:CCP4BB@JISCMAIL.AC.UK> 
on behalf of Tristan Croll <mailto:ti...@cam.ac.uk>
Sent: 21 December 2021 08:28
To: CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK> 
<mailto:CCP4BB@JISCMAIL.AC.UK>
Subject: Re: [ccp4bb] Validation of structure prediction

I agree with Dale. Tools like MolProbity are not the right approach to 
validating a structure prediction. To understand why, just consider that all 
you need to do to get a perfect MolProbity score is predict every structure as 
a single long alpha helix with ideal rotamers, with a kink at each proline.

To validate a predicted structure will require a completely different toolset - 
one that I’m not sure fully exists yet.

— Tristan

> On 20 Dec 2021, at 18:47, Dale Tronrud 
> <mailto:de...@daletronrud.com> wrote:
>
>   I don't see any reason to believe that software designed to validate 
> crystallographic or NMR models would have any utility validating AlphaFold 
> predicted models.  Doesn't the prediction software already ensure that all 
> the indicators used by Molprobity are obeyed?  I'm afraid that the tools to 
> validate any new technique must be designed specifically for that technique. 
> (And when they become available they will be useless for validating 
> crystallographic models!)
>
> Dale E. Tronrud
>
>> On 12/20/2021 10:28 AM, Nicholas Clark wrote:
>> The Molprobity server can be run online and only requires the coordinates in 
>> PDB format: http://molprobity.biochem.duke.edu/ 
>> <http://molprobity.biochem.duke.edu/>.
>> Best,
>> Nick Clark
>> On Mon, Dec 20, 2021 at 11:10

[ccp4bb] Fwd: [ccp4bb] Validation of structure prediction

[F . Xavier Gomis-Rüth]

Dear all,
this is by far not the general case in our hands. Depending on which 
AlphaFold protocol is used, the resulting models have locally disfavourable
geometries–including clashes–, impossible chain crossovers, etc. I would 
definitively recommend everybody to go through the model in detail and 
perform
a final geometry minimization with Coot and/or Phenix/Refmac. And in 
these cases, general geometry validation as provided by MolProbity

provides a final proof of the computational model.
Best,
Xavier


 Forwarded Message 
Subject:Re: [ccp4bb] Validation of structure prediction
Date:   Tue, 21 Dec 2021 09:43:37 +
From: 	Vollmar, Melanie (DLSLtd,RAL,LSCI) 
<64fe7ccc6b4d-dmarc-requ...@jiscmail.ac.uk>
Reply-To: 	Vollmar, Melanie (DLSLtd,RAL,LSCI) 


To: CCP4BB@JISCMAIL.AC.UK



Tristan is spot on. All the predicted structures have near perfect 
geometry, so commonly used validation tools like MolProbity can no 
longer be applied.


What you need to consider is biological relevance of the predicted 
model. Does the model correctly reflect residue arrangement in the 
active site? Are domains in correct relative orientation to allow for 
interactions and movements, perhaps found by some other assay? Is there 
appropriate room to fit a ligand/cofactor? Are transmembrane helices, if 
there are any, correctly found?


You need to map the knowledge you have of your protein to the structure 
and see if the atom positions and what you know support each other.


Cheers

M

*From:* CCP4 bulletin board  on behalf of Tristan 
Croll 

*Sent:* 21 December 2021 08:28
*To:* CCP4BB@JISCMAIL.AC.UK 
*Subject:* Re: [ccp4bb] Validation of structure prediction
I agree with Dale. Tools like MolProbity are not the right approach to 
validating a structure prediction. To understand why, just consider that 
all you need to do to get a perfect MolProbity score is predict every 
structure as a single long alpha helix with ideal rotamers, with a kink 
at each proline.


To validate a predicted structure will require a completely different 
toolset - one that I’m not sure fully exists yet.


— Tristan

> On 20 Dec 2021, at 18:47, Dale Tronrud  wrote:
>
>   I don't see any reason to believe that software designed to 
validate crystallographic or NMR models would have any utility 
validating AlphaFold predicted models. Doesn't the prediction software 
already ensure that all the indicators used by Molprobity are obeyed?  
I'm afraid that the tools to validate any new technique must be designed 
specifically for that technique. (And when they become available they 
will be useless for validating crystallographic models!)

>
> Dale E. Tronrud
>
>> On 12/20/2021 10:28 AM, Nicholas Clark wrote:
>> The Molprobity server can be run online and only requires the 
coordinates in PDB format: http://molprobity.biochem.duke.edu/ 
<http://molprobity.biochem.duke.edu/>.

>> Best,
>> Nick Clark
>> On Mon, Dec 20, 2021 at 11:10 AM Reza Khayat <mailto:rkha...@ccny.cuny.edu <mailto:rkha...@ccny.cuny.edu>>> wrote:

>>    ​Hi,
>>    Can anyone suggest how to validate a predicted structure? Something
>>    similar to wwPDB validation without the need for refinement
>>    statistics. I realize this is a strange question given that the
>>    geometry of the model is anticipated to be fine if the structure was
>>    predicted by a server that minimizes the geometry to improve its
>>    statistics. Nonetheless, the journal has asked me for such a report.
>>    Thanks.
>>    Best wishes,
>>    Reza
>>    Reza Khayat, PhD
>>    Associate Professor
>>    City College of New York
>>    Department of Chemistry and Biochemistry
>>    New York, NY 10031
>> 
>>    To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1>
>>    <https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1>>

>> --
>> Nicholas D. Clark
>> PhD Candidate
>> Malkowski Lab
>> University at Buffalo
>> Department of Structural Biology
>> Jacob's School of Medicine & Biomedical Sciences
>> 955 Main Street, RM 5130
>> Buffalo, NY 14203
>> Cell: 716-830-1908
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CC

Re: [ccp4bb] Validation of structure prediction

[Vollmar, Melanie (DLSLtd,RAL,LSCI)]

Tristan is spot on. All the predicted structures have near perfect geometry, so 
commonly used validation tools like MolProbity can no longer be applied.

What you need to consider is biological relevance of the predicted model. Does 
the model correctly reflect residue arrangement in the active site? Are domains 
in correct relative orientation to allow for interactions and movements, 
perhaps found by some other assay? Is there appropriate room to fit a 
ligand/cofactor? Are transmembrane helices, if there are any, correctly found?

You need to map the knowledge you have of your protein to the structure and see 
if the atom positions and what you know support each other.

Cheers

M

From: CCP4 bulletin board  on behalf of Tristan Croll 

Sent: 21 December 2021 08:28
To: CCP4BB@JISCMAIL.AC.UK 
Subject: Re: [ccp4bb] Validation of structure prediction

I agree with Dale. Tools like MolProbity are not the right approach to 
validating a structure prediction. To understand why, just consider that all 
you need to do to get a perfect MolProbity score is predict every structure as 
a single long alpha helix with ideal rotamers, with a kink at each proline.

To validate a predicted structure will require a completely different toolset - 
one that I’m not sure fully exists yet.

— Tristan

> On 20 Dec 2021, at 18:47, Dale Tronrud  wrote:
>
>   I don't see any reason to believe that software designed to validate 
> crystallographic or NMR models would have any utility validating AlphaFold 
> predicted models.  Doesn't the prediction software already ensure that all 
> the indicators used by Molprobity are obeyed?  I'm afraid that the tools to 
> validate any new technique must be designed specifically for that technique. 
> (And when they become available they will be useless for validating 
> crystallographic models!)
>
> Dale E. Tronrud
>
>> On 12/20/2021 10:28 AM, Nicholas Clark wrote:
>> The Molprobity server can be run online and only requires the coordinates in 
>> PDB format: http://molprobity.biochem.duke.edu/ 
>> <http://molprobity.biochem.duke.edu/>.
>> Best,
>> Nick Clark
>> On Mon, Dec 20, 2021 at 11:10 AM Reza Khayat > <mailto:rkha...@ccny.cuny.edu>> wrote:
>>​Hi,
>>Can anyone suggest how to validate a predicted structure? Something
>>similar to wwPDB validation without the need for refinement
>>statistics. I realize this is a strange question given that the
>>geometry of the model is anticipated to be fine if the structure was
>>predicted by a server that minimizes the geometry to improve its
>>statistics. Nonetheless, the journal has asked me for such a report.
>>Thanks.
>>Best wishes,
>>Reza
>>Reza Khayat, PhD
>>Associate Professor
>>City College of New York
>>Department of Chemistry and Biochemistry
>>New York, NY 10031
>>
>>To unsubscribe from the CCP4BB list, click the following link:
>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>><https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1>
>> --
>> Nicholas D. Clark
>> PhD Candidate
>> Malkowski Lab
>> University at Buffalo
>> Department of Structural Biology
>> Jacob's School of Medicine & Biomedical Sciences
>> 955 Main Street, RM 5130
>> Buffalo, NY 14203
>> Cell: 716-830-1908
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 
>> <https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1>
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>
> This message was issued to members of 
> www.jiscmail.ac.uk/CCP4BB<http://www.jiscmail.ac.uk/CCP4BB>, a mailing list 
> hosted by www.jiscmail.ac.uk<http://www.jiscmail.ac.uk>, terms & conditions 
> are available at https://www.jiscmail.ac.uk/policyandsecurity/



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of 
www.jiscmail.ac.uk/CCP4BB<http://www.jiscmail.ac.uk/CCP4BB>, a mailing list 
hosted by www.jiscmail.ac.uk<http://www.jiscmail.ac.uk>, terms & conditions are 
available at https://www.jiscmail.ac.uk/policyandsecurity/

-- 
This e

Re: [ccp4bb] Validation of structure prediction

[Tristan Croll]

I agree with Dale. Tools like MolProbity are not the right approach to 
validating a structure prediction. To understand why, just consider that all 
you need to do to get a perfect MolProbity score is predict every structure as 
a single long alpha helix with ideal rotamers, with a kink at each proline. 

To validate a predicted structure will require a completely different toolset - 
one that I’m not sure fully exists yet. 

— Tristan 

> On 20 Dec 2021, at 18:47, Dale Tronrud  wrote:
> 
>   I don't see any reason to believe that software designed to validate 
> crystallographic or NMR models would have any utility validating AlphaFold 
> predicted models.  Doesn't the prediction software already ensure that all 
> the indicators used by Molprobity are obeyed?  I'm afraid that the tools to 
> validate any new technique must be designed specifically for that technique. 
> (And when they become available they will be useless for validating 
> crystallographic models!)
> 
> Dale E. Tronrud
> 
>> On 12/20/2021 10:28 AM, Nicholas Clark wrote:
>> The Molprobity server can be run online and only requires the coordinates in 
>> PDB format: http://molprobity.biochem.duke.edu/ 
>> .
>> Best,
>> Nick Clark
>> On Mon, Dec 20, 2021 at 11:10 AM Reza Khayat > > wrote:
>>​Hi,
>>Can anyone suggest how to validate a predicted structure? Something
>>similar to wwPDB validation without the need for refinement
>>statistics. I realize this is a strange question given that the
>>geometry of the model is anticipated to be fine if the structure was
>>predicted by a server that minimizes the geometry to improve its
>>statistics. Nonetheless, the journal has asked me for such a report.
>>Thanks.
>>Best wishes,
>>Reza
>>Reza Khayat, PhD
>>Associate Professor
>>City College of New York
>>Department of Chemistry and Biochemistry
>>New York, NY 10031
>>
>>To unsubscribe from the CCP4BB list, click the following link:
>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>>
>> -- 
>> Nicholas D. Clark
>> PhD Candidate
>> Malkowski Lab
>> University at Buffalo
>> Department of Structural Biology
>> Jacob's School of Medicine & Biomedical Sciences
>> 955 Main Street, RM 5130
>> Buffalo, NY 14203
>> Cell: 716-830-1908
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 
>> 
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> 
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing 
> list hosted by www.jiscmail.ac.uk, terms & conditions are available at 
> https://www.jiscmail.ac.uk/policyandsecurity/



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Anat Bashan]

Dear Reza,
You may run it through the Moleprobity server for validation:
http://molprobity.manchester.ac.uk/

Good luck, Anat.

From: CCP4 bulletin board  On Behalf Of Reza Khayat
Sent: Monday, 20 December 2021 18:10
To: CCP4BB@jiscmail.ac.uk
Subject: [ccp4bb] Validation of structure prediction


​Hi,

Can anyone suggest how to validate a predicted structure? Something similar to 
wwPDB validation without the need for refinement statistics. I realize this is 
a strange question given that the geometry of the model is anticipated to be 
fine if the structure was predicted by a server that minimizes the geometry to 
improve its statistics. Nonetheless, the journal has asked me for such a 
report. Thanks.

Best wishes,

Reza


Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Dale Tronrud]

   I don't see any reason to believe that software designed to validate 
crystallographic or NMR models would have any utility validating 
AlphaFold predicted models.  Doesn't the prediction software already 
ensure that all the indicators used by Molprobity are obeyed?  I'm 
afraid that the tools to validate any new technique must be designed 
specifically for that technique. (And when they become available they 
will be useless for validating crystallographic models!)


Dale E. Tronrud

On 12/20/2021 10:28 AM, Nicholas Clark wrote:
The Molprobity server can be run online and only requires the 
coordinates in PDB format: http://molprobity.biochem.duke.edu/ 
.


Best,

Nick Clark

On Mon, Dec 20, 2021 at 11:10 AM Reza Khayat > wrote:


​Hi,


Can anyone suggest how to validate a predicted structure? Something
similar to wwPDB validation without the need for refinement
statistics. I realize this is a strange question given that the
geometry of the model is anticipated to be fine if the structure was
predicted by a server that minimizes the geometry to improve its
statistics. Nonetheless, the journal has asked me for such a report.
Thanks.


Best wishes,

Reza


Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1




--
Nicholas D. Clark
PhD Candidate
Malkowski Lab
University at Buffalo
Department of Structural Biology
Jacob's School of Medicine & Biomedical Sciences
955 Main Street, RM 5130
Buffalo, NY 14203

Cell: 716-830-1908



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 







To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Nicholas Clark]

The Molprobity server can be run online and only requires the coordinates
in PDB format: http://molprobity.biochem.duke.edu/.

Best,

Nick Clark

On Mon, Dec 20, 2021 at 11:10 AM Reza Khayat  wrote:

> ​Hi,
>
>
> Can anyone suggest how to validate a predicted structure? Something
> similar to wwPDB validation without the need for refinement statistics. I
> realize this is a strange question given that the geometry of the model is
> anticipated to be fine if the structure was predicted by a server that
> minimizes the geometry to improve its statistics. Nonetheless, the journal
> has asked me for such a report. Thanks.
>
>
> Best wishes,
>
> Reza
>
>
> Reza Khayat, PhD
> Associate Professor
> City College of New York
> Department of Chemistry and Biochemistry
> New York, NY 10031
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>


-- 
Nicholas D. Clark
PhD Candidate
Malkowski Lab
University at Buffalo
Department of Structural Biology
Jacob's School of Medicine & Biomedical Sciences
955 Main Street, RM 5130
Buffalo, NY 14203

Cell: 716-830-1908



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[mesters]

Hello,

another one . http://bioinfo.ifm.liu.se/ProQ2/

Best regards

Jeroen

Am 20.12.21 um 17:39 schrieb Weidenhausen, Jonas:

Hi Reza,

Could you use MolProbity (also from within Phenix), which will give 
you statistics about Ramachandran/rotamer outliers, clash score etc?

Besides, maybe ModFOLD? (https://www.reading.ac.uk/bioinf/ModFOLD/)
Which journal, if I may ask?

Best,
Jonas

--
Jonas Weidenhausen
PhD Student
AG Sinning

BZH Heidelberg University, room: 524
INF 328, 69120 Heidelberg
Phone: +49 6221 54-4786
jonas.weidenhau...@bzh.uni-heidelberg.de


On 20. Dec 2021, at 17:10, Reza Khayat  wrote:

​Hi,

Can anyone suggest how to validate a predicted structure? Something 
similar to wwPDB validation without the need for refinement 
statistics. I realize this is a strange question given that the 
geometry of the model is anticipated to be fine if the structure was 
predicted by a server that minimizes the geometry to improve its 
statistics. Nonetheless, the journal has asked me for such a report. 
Thanks.


Best wishes,
Reza

Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031


To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 






To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 





--
signature.html *Dr. /math. et dis. nat./ Jeroen R. Mesters
*Deputy, Lecturer, Program Coordinator Infection Biology 

Visiting Professorship (South Bohemian University 
) in Biophysics


*University of Lübeck*
Center for Structural and Cell Biology in Medicine*
Institute of Biochemistry*

Tel  +49 451 3101 3105 (Secretariate 3101)
Fax +49 451 3101 3104 *
*jeroen.mest...@uni-luebeck.de
https://www.biochem.uni-luebeck.de
https://orcid.org/-0001-8532-6699

*Ratzeburger Allee 160
23538 Lübeck, Schleswig-Holstein
Germany*



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] Validation of structure prediction

[Weidenhausen, Jonas]

Hi Reza,

Could you use MolProbity (also from within Phenix), which will give you 
statistics about Ramachandran/rotamer outliers, clash score etc?
Besides, maybe ModFOLD? (https://www.reading.ac.uk/bioinf/ModFOLD/)
Which journal, if I may ask?

Best,
Jonas

--
Jonas Weidenhausen
PhD Student
AG Sinning

BZH Heidelberg University, room: 524
INF 328, 69120 Heidelberg
Phone: +49 6221 54-4786
jonas.weidenhau...@bzh.uni-heidelberg.de

On 20. Dec 2021, at 17:10, Reza Khayat 
mailto:rkha...@ccny.cuny.edu>> wrote:

​Hi,

Can anyone suggest how to validate a predicted structure? Something similar to 
wwPDB validation without the need for refinement statistics. I realize this is 
a strange question given that the geometry of the model is anticipated to be 
fine if the structure was predicted by a server that minimizes the geometry to 
improve its statistics. Nonetheless, the journal has asked me for such a 
report. Thanks.

Best wishes,
Reza

Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031


To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

[ccp4bb] Validation of structure prediction

[Reza Khayat]

?Hi,

Can anyone suggest how to validate a predicted structure? Something similar to 
wwPDB validation without the need for refinement statistics. I realize this is 
a strange question given that the geometry of the model is anticipated to be 
fine if the structure was predicted by a server that minimizes the geometry to 
improve its statistics. Nonetheless, the journal has asked me for such a 
report. Thanks.

Best wishes,

Reza


Reza Khayat, PhD
Associate Professor
City College of New York
Department of Chemistry and Biochemistry
New York, NY 10031



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/