[ccp4bb] diverging Rcryst and Rfree
Hi all, Can anyone comment, in general, on diverging Rcryst and Rfree values(say>7%) for structures with kind of low resolutions(2.5-2.9 angstroms)? Thanks RJ
Re: [ccp4bb] diverging Rcryst and Rfree
Hi, I have seen this happening when I had NCS and did not include it in refinement. Rwork drops and Rfree increases. In this case the difference became small when I included the NCS. Also if your Rmerge is high and you include all reflections in refinement, Rfree is high. In my experience, by excluding F < sigma reflections you drop Rfree a lot. My limited experience suggests errors in the data and/.or in the way you handle the data. Jackie Vitali On Mon, Oct 25, 2010 at 5:10 PM, Rakesh Joshi wrote: > Hi all, > > Can anyone comment, in general, on diverging Rcryst and Rfree > values(say>7%) for > structures with kind of low resolutions(2.5-2.9 angstroms)? > > Thanks > RJ >
Re: [ccp4bb] diverging Rcryst and Rfree
I had a similar problem. It dissappeared when I switched the refinement to phenix. The R factors dropped and the difference between them became acceptable.
Re: [ccp4bb] diverging Rcryst and Rfree
On Monday, October 25, 2010 03:16:28 pm Maia Cherney wrote: > I had a similar problem. It dissappeared when I switched the > refinement to phenix. The R factors dropped and the difference between > them became acceptable. It would be useful to pin down what the two programs did differently. It is likely that some particular default setting is the relevant quantity. - Geometric restraint weight? - B-factor restraint weight? - NCS? - hydrogen treatment? - Did you use the same set of R-free reflections in both programs? I'm currently running a bunch of parallel runs in phenix and refmac to test something else. What I notice to be the biggest difference in their default settings is that the default restraint on B factors is looser by a factor of 2-3 in phenix, whereas the default restraint on geometry is looser by a factor of ~2 in refmac. I need to adjust those for each refinement pair so that I can get comparable runs. Ethan -- Ethan A Merritt Biomolecular Structure Center, K-428 Health Sciences Bldg University of Washington, Seattle 98195-7742
Re: [ccp4bb] diverging Rcryst and Rfree
http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg04677.html as well as some notes in the older posts :) As a very basic rule of thumb, Rfree-Rwork tends to be around Rmerge for the dataset for refinements that are not overfitted. Artem On Mon, Oct 25, 2010 at 4:10 PM, Rakesh Joshi wrote: > Hi all, > > Can anyone comment, in general, on diverging Rcryst and Rfree > values(say>7%) for > structures with kind of low resolutions(2.5-2.9 angstroms)? > > Thanks > RJ >
Re: [ccp4bb] diverging Rcryst and Rfree
And the rationale for that rule being exactly what? For stats, see figures 12-23, 12-24 http://www.ruppweb.org/garland/gallery/Ch12/index_2.htm br From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Artem Evdokimov Sent: Monday, October 25, 2010 6:36 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] diverging Rcryst and Rfree http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg04677.html as well as some notes in the older posts :) As a very basic rule of thumb, Rfree-Rwork tends to be around Rmerge for the dataset for refinements that are not overfitted. Artem On Mon, Oct 25, 2010 at 4:10 PM, Rakesh Joshi wrote: Hi all, Can anyone comment, in general, on diverging Rcryst and Rfree values(say>7%) for structures with kind of low resolutions(2.5-2.9 angstroms)? Thanks RJ
Re: [ccp4bb] diverging Rcryst and Rfree
Not that rules of thumb always have to have a rationale, nor that they're always correct - but it would seem that noise in the data (of which Rmerge is an indicator) should have a significant relationship with the R:Rfree difference, since Rfree is not (should not be, if selected correctly) subject to noise fitting. This rule is easily broken if one refines against very noisy data (e.g. "that last shell with Rmerge of 55% and ratio of 1.3 is still good, right?") or if the structure is overfit. The rule is only an indicative one (i.e. one should get really worried if R-Rfree looks very different from Rmerge) and it breaks down at very high and very low resolution (more complete picture by GK and shown in BR's book). Since selection of data and refinement procedures is subject to the decisions of the practitioner, I suspect that the extreme divergence shown in the figures that you refer to is probably the result of our own collective decisions. I have no proof, but I suspect that if a large enough section of the PDB were to be re-refined using the same methods and the same data trimming practices, the spread would be considerably more narrow. That'd be somewhat hard to do - but may be doable now given the abundance of auto-building and auto-correcting algorithms. Artem On Mon, Oct 25, 2010 at 9:07 PM, Bernhard Rupp (Hofkristallrat a.D.) < hofkristall...@gmail.com> wrote: > And the rationale for that rule being exactly what? > > > > For stats, see figures 12-23, 12-24 > > http://www.ruppweb.org/garland/gallery/Ch12/index_2.htm > > > > br > > > > *From:* CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] *On Behalf Of > *Artem > Evdokimov > *Sent:* Monday, October 25, 2010 6:36 PM > *To:* CCP4BB@JISCMAIL.AC.UK > *Subject:* Re: [ccp4bb] diverging Rcryst and Rfree > > > > http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg04677.html > > as well as some notes in the older posts :) > > As a very basic rule of thumb, Rfree-Rwork tends to be around Rmerge for > the dataset for refinements that are not overfitted. > > Artem > > On Mon, Oct 25, 2010 at 4:10 PM, Rakesh Joshi wrote: > > Hi all, > > Can anyone comment, in general, on diverging Rcryst and Rfree > values(say>7%) for > structures with kind of low resolutions(2.5-2.9 angstroms)? > > Thanks > RJ > > >
Re: [ccp4bb] diverging Rcryst and Rfree
Dear Rakesh, dear Artem, Since the initial question is not precise (which kind of comments are expected?) I may mention that the most frequent values of R, Rfree and DeltaR (that is asked about) are given in our work published in 2009 in Acta Cryst., D65, 1283-1291. Interestingly, they are practically linear functions of log(resolution). The plots show also the statistics of deviation from these lines. Best regards, Sacha Urzhumtsev Universities of Strasbourg & Nancy De : CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] De la part de Artem Evdokimov Envoyé : mardi 26 octobre 2010 03:36 À : CCP4BB@JISCMAIL.AC.UK Objet : Re: [ccp4bb] diverging Rcryst and Rfree http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg04677.html as well as some notes in the older posts :) As a very basic rule of thumb, Rfree-Rwork tends to be around Rmerge for the dataset for refinements that are not overfitted. Artem On Mon, Oct 25, 2010 at 4:10 PM, Rakesh Joshi mailto:rjo...@purdue.edu>> wrote: Hi all, Can anyone comment, in general, on diverging Rcryst and Rfree values(say>7%) for structures with kind of low resolutions(2.5-2.9 angstroms)? Thanks RJ
Re: [ccp4bb] diverging Rcryst and Rfree
Rakesh Looking at the http://www.pdbe.org/statistics Structure Statistics Then for all structures a we see that Rdiff of > 0.7 is not that uncommon with this about 1 sigma away from the mean value of 0.4 for all structutures and 0.45 for your resolution range For structures with Rdiff range of 0.7-0.8 and resolution 2.7 - 2.9 we see that there 212 structures. If I edit the query in PDBeDatabase to your exact requirement ranges then there are 1353 example structures out of 53616 examples where this data exists. My comment is that this is little worse than average, but not particularly a problem. Tom Hi all, Can anyone comment, in general, on diverging Rcryst and Rfree values(say>7%) for structures with kind of low resolutions(2.5-2.9 angstroms)? Thanks RJ
Re: [ccp4bb] diverging Rcryst and Rfree
I'm not sure that "a lttle worse" is the appropriate description - I think all you can say is that it deviates from the average when only resolution is used to define what is "average". My point is that a number of other factors are known to be involved and so you can't say that this deviation is "worse" until you have taken them all into account. Specifically, it's known that the expected value of the ratio Rfree/Rwork (i.e. expected on the basis of the null hypothesis that the structure is correct and complete and the only errors are random experimental errors) is directly related to the ratio (no of significant experimental observations) / (effective no of refined parameters), where "effective" here means "taking into account the restraints". The number of significant observations will clearly depend on a number of factors, not only resolution, but also solvent content (which you obviously can't control unless you use a different crystal form), and data completeness (which you can control up to a point by optimising the data collection strategy). The effective no of parameters obviously depends only on the parameter/restraint model and the weights, both of which you have full control of, and therefore the effective no of parameters should be completely determined if the parameter/restraint model that has been selected is optimal. Finally, in order to compute Rfree/Rwork from Rdiff = Rfree-Rwork, Rwork itself must be specified, i.e.: Rfree/Rwork = (Rfree-Rwork) / Rwork - 1 = Rdiff/Rwork - 1 It's actually much easier to work with Rfree/Rwork instead of Rdiff, because then you don't need to specify a particular value of Rwork, and you have one less variable to worry about in the factor analysis. So assuming the model is optimal, the major factors in addition to resolution which control the expected value of Rdiff are the solvent content, the data completeness and Rwork. The value of Rwork obtained for an optimal model on convergence is obviously related to the data quality (e.g. mean I/sig(I)), and of course the resolution. The bottom line is that unless we are given a lot more information it's not possible to say whether a specific value of Rdiff deviates significantly from the expected value. Cheers -- Ian > > Then for all structures a we see that Rdiff of > 0.7 is not that uncommon > with this about 1 sigma away from the mean value of 0.4 for all structutures > and 0.45 for your resolution range > > For structures with Rdiff range of 0.7-0.8 and resolution 2.7 - 2.9 we see > that there 212 structures. > > If I edit the query in PDBeDatabase to your exact requirement ranges then > there > are 1353 example structures out of 53616 examples where this data exists. > > My comment is that this is little worse than average, but not particularly a > problem. > > Tom >> >> Hi all, >> >> Can anyone comment, in general, on diverging Rcryst and Rfree >> values(say>7%) for structures with kind of low resolutions(2.5-2.9 >> angstroms)? >> >> Thanks >> RJ >> >
Re: [ccp4bb] diverging Rcryst and Rfree
I would expect such a difference with lowish resolution data. Your model will be biased towards the restraints - ie the geometry willbe good, but there is bearly enough observations to fit the actualmodel properly. eg - it will be hard to position solvent, and to recognise any deviaions from NCS. So dont be too surprised or worried.. Look at the maps - if they look "clean" then things are probably OK Eleanor On 10/25/2010 10:44 PM, Jacqueline Vitali wrote: Hi, I have seen this happening when I had NCS and did not include it in refinement. Rwork drops and Rfree increases. In this case the difference became small when I included the NCS. Also if your Rmerge is high and you include all reflections in refinement, Rfree is high. In my experience, by excluding F< sigma reflections you drop Rfree a lot. My limited experience suggests errors in the data and/.or in the way you handle the data. Jackie Vitali On Mon, Oct 25, 2010 at 5:10 PM, Rakesh Joshi wrote: Hi all, Can anyone comment, in general, on diverging Rcryst and Rfree values(say>7%) for structures with kind of low resolutions(2.5-2.9 angstroms)? Thanks RJ
Re: [ccp4bb] diverging Rcryst and Rfree
Jackie, please note that (at least imho) the desire to obtain "better" R-factors does not justify excluding data from analysis. Weak reflections that you suggest should be rejected contain information, and excluding them will indeed artificially lower the R-factors while reducing the accuracy of your model. Cheers, Ed. On Mon, 2010-10-25 at 17:44 -0400, Jacqueline Vitali wrote: > Also if your Rmerge is high and you include all reflections in > refinement, Rfree is high. In my experience, by excluding F < sigma > reflections you drop Rfree a lot. -- "I'd jump in myself, if I weren't so good at whistling." Julian, King of Lemurs
Re: [ccp4bb] diverging Rcryst and Rfree
Jackie I agree completely with Ed (for once!), not only for the reasons he gave, but also that it's valid to compare statistics such as likelihood and R factors ONLY if only the model is varied. Such a comparison is not valid if the data used are varied (in this case you are changing the data by deleting some of them). Cheers -- Ian On Tue, Oct 26, 2010 at 2:37 PM, Ed Pozharski wrote: > Jackie, > > please note that (at least imho) the desire to obtain "better" R-factors > does not justify excluding data from analysis. Weak reflections that > you suggest should be rejected contain information, and excluding them > will indeed artificially lower the R-factors while reducing the accuracy > of your model. > > Cheers, > > Ed. > > On Mon, 2010-10-25 at 17:44 -0400, Jacqueline Vitali wrote: >> Also if your Rmerge is high and you include all reflections in >> refinement, Rfree is high. In my experience, by excluding F < sigma >> reflections you drop Rfree a lot. > > > > -- > "I'd jump in myself, if I weren't so good at whistling." > Julian, King of Lemurs >
Re: [ccp4bb] diverging Rcryst and Rfree
b) very large Rmerge values: Rmerge Rwork Rfree Rfree-Rwork Resolution - 0.9990 0.1815 0.20860.0271 1.80<<< SG center, unpublished 0.8700 0.1708 0.22700.0562 1.96<<< unpublished 0.7700 0.1870 0.22970.0428 1.56 0.7600 0.2380 0.26800.0300 2.50<<< SG center, unpublished 0.7000 0.1700 0.22530.0553 1.71 0.6400 0.2179 0.27150.0536 2.75<<< SG center, unpublished The most disturbing to me is that of those with very large overall Rmerge values, 3 come from structural genomics centers. Is that less or more disturbing than that the other 50% come from not-SG centers? Of course, the authors themselves may be willing to help correct the obvious typos -- which will presumably disappear forever once we can finally upload log files upon deposition (coming soon, I'm told). On an unrelated note, it's reassuring to see sound statistical principles -- averages, large N, avoidance of small number-anecdotes, and such rot -- continue not to be abandoned in the politics of science funding, he said airily. phx
Re: [ccp4bb] diverging Rcryst and Rfree
Yes! - the critical piece of information that we're missing is the proportion of *all* structures that come from SG centres. Only knowing that can we do any serious statistics ... -- Ian On Tue, Oct 26, 2010 at 5:07 PM, Frank von Delft wrote: >> b) very large Rmerge values: >> >> Rmerge Rwork Rfree Rfree-Rwork Resolution >> - >> 0.9990 0.1815 0.2086 0.0271 1.80<<< SG center, unpublished >> 0.8700 0.1708 0.2270 0.0562 1.96<<< unpublished >> 0.7700 0.1870 0.2297 0.0428 1.56 >> 0.7600 0.2380 0.2680 0.0300 2.50<<< SG center, unpublished >> 0.7000 0.1700 0.2253 0.0553 1.71 >> 0.6400 0.2179 0.2715 0.0536 2.75<<< SG center, unpublished >> >> The most disturbing to me is that of those with very large overall >> Rmerge values, 3 come from structural genomics centers. > > Is that less or more disturbing than that the other 50% come from not-SG > centers? > > Of course, the authors themselves may be willing to help correct the obvious > typos -- which will presumably disappear forever once we can finally upload > log files upon deposition (coming soon, I'm told). > > On an unrelated note, it's reassuring to see sound statistical principles -- > averages, large N, avoidance of small number-anecdotes, and such rot -- > continue not to be abandoned in the politics of science funding, he said > airily. > > phx >
Re: [ccp4bb] diverging Rcryst and Rfree
I found a practical solution to a similar problem. When I get large gap between Rf/R in refmac I repeat the refinement in PHENIX using the same model and the same mtz file, It has always worked for me. And I have no theory for that observation, but the tables in publications looked better. Maia Quoting "Ian Tickle" : Jackie I agree completely with Ed (for once!), not only for the reasons he gave, but also that it's valid to compare statistics such as likelihood and R factors ONLY if only the model is varied. Such a comparison is not valid if the data used are varied (in this case you are changing the data by deleting some of them). Cheers -- Ian On Tue, Oct 26, 2010 at 2:37 PM, Ed Pozharski wrote: Jackie, please note that (at least imho) the desire to obtain "better" R-factors does not justify excluding data from analysis. Weak reflections that you suggest should be rejected contain information, and excluding them will indeed artificially lower the R-factors while reducing the accuracy of your model. Cheers, Ed. On Mon, 2010-10-25 at 17:44 -0400, Jacqueline Vitali wrote: Also if your Rmerge is high and you include all reflections in refinement, Rfree is high. In my experience, by excluding F < sigma reflections you drop Rfree a lot. -- "I'd jump in myself, if I weren't so good at whistling." Julian, King of Lemurs
Re: [ccp4bb] diverging Rcryst and Rfree
Dear Ian, On Tue, Oct 26, 2010 at 05:15:50PM +0100, Ian Tickle wrote: > Yes! - the critical piece of information that we're missing is the > proportion of *all* structures that come from SG centres. Only > knowing that can we do any serious statistics ... The point I was trying to make was not to blame SG centres (or comparing them with other groups) - I was concerned about technology mainly. Clearly, there was some problem at the point of deposition. I would have thought that especially in structures coming from SG centres there would be technology in place on both sides (at the SG centre and at the PDB site) to catch unusual values like an overall Rmerge of 0.99? Cheers Clemens PS: according to some simple search out of those 3026 entries 1473 are from SG centres and 1553 not. > > -- Ian > > On Tue, Oct 26, 2010 at 5:07 PM, Frank von Delft > wrote: > >> b) very large Rmerge values: > >> > >> Rmerge Rwork Rfree Rfree-Rwork Resolution > >> - > >> 0.9990 0.1815 0.2086 0.0271 1.80<<< SG center, unpublished > >> 0.8700 0.1708 0.2270 0.0562 1.96<<< unpublished > >> 0.7700 0.1870 0.2297 0.0428 1.56 > >> 0.7600 0.2380 0.2680 0.0300 2.50<<< SG center, unpublished > >> 0.7000 0.1700 0.2253 0.0553 1.71 > >> 0.6400 0.2179 0.2715 0.0536 2.75<<< SG center, unpublished > >> > >> The most disturbing to me is that of those with very large overall > >> Rmerge values, 3 come from structural genomics centers. > > > > Is that less or more disturbing than that the other 50% come from not-SG > > centers? > > > > Of course, the authors themselves may be willing to help correct the obvious > > typos -- which will presumably disappear forever once we can finally upload > > log files upon deposition (coming soon, I'm told). > > > > On an unrelated note, it's reassuring to see sound statistical principles -- > > averages, large N, avoidance of small number-anecdotes, and such rot -- > > continue not to be abandoned in the politics of science funding, he said > > airily. > > > > phx > > -- *** * Clemens Vonrhein, Ph.D. vonrhein AT GlobalPhasing DOT com * * Global Phasing Ltd. * Sheraton House, Castle Park * Cambridge CB3 0AX, UK *-- * BUSTER Development Group (http://www.globalphasing.com) ***
Re: [ccp4bb] diverging Rcryst and Rfree [SEC=UNCLASSIFIED]
One "rule of thumb" based on R and R-free divergence that I impress onto crystallography students is this: If a change in refinement strategy or parameters (eg loosening restraints, introducing TLS) or a round of addition of unimportant water molecules results in a reduction of R that is more than double the reduction in R-free, then don't do it. This rule of thumb has proven successful in providing a defined end point for building and refining a structure. The rule works on the differential of R - R-free divergence. I've noticed that some structures begin with a bigger divergence than others. Different Rmerge might explain. Has anyone else found a student in a dark room carefully adding large numbers of partially occupied water molecules? Anthony Anthony DuffTelephone: 02 9717 3493 Mob: 043 189 1076 From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Artem Evdokimov Sent: Tuesday, 26 October 2010 1:45 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] diverging Rcryst and Rfree Not that rules of thumb always have to have a rationale, nor that they're always correct - but it would seem that noise in the data (of which Rmerge is an indicator) should have a significant relationship with the R:Rfree difference, since Rfree is not (should not be, if selected correctly) subject to noise fitting. This rule is easily broken if one refines against very noisy data (e.g. "that last shell with Rmerge of 55% and ratio of 1.3 is still good, right?") or if the structure is overfit. The rule is only an indicative one (i.e. one should get really worried if R-Rfree looks very different from Rmerge) and it breaks down at very high and very low resolution (more complete picture by GK and shown in BR's book). Since selection of data and refinement procedures is subject to the decisions of the practitioner, I suspect that the extreme divergence shown in the figures that you refer to is probably the result of our own collective decisions. I have no proof, but I suspect that if a large enough section of the PDB were to be re-refined using the same methods and the same data trimming practices, the spread would be considerably more narrow. That'd be somewhat hard to do - but may be doable now given the abundance of auto-building and auto-correcting algorithms. Artem On Mon, Oct 25, 2010 at 9:07 PM, Bernhard Rupp (Hofkristallrat a.D.) wrote: And the rationale for that rule being exactly what? For stats, see figures 12-23, 12-24 http://www.ruppweb.org/garland/gallery/Ch12/index_2.htm br From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Artem Evdokimov Sent: Monday, October 25, 2010 6:36 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] diverging Rcryst and Rfree http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg04677.html as well as some notes in the older posts :) As a very basic rule of thumb, Rfree-Rwork tends to be around Rmerge for the dataset for refinements that are not overfitted. Artem On Mon, Oct 25, 2010 at 4:10 PM, Rakesh Joshi wrote: Hi all, Can anyone comment, in general, on diverging Rcryst and Rfree values(say>7%) for structures with kind of low resolutions(2.5-2.9 angstroms)? Thanks RJ
Re: [ccp4bb] diverging Rcryst and Rfree [SEC=UNCLASSIFIED]
> This rule of thumb has proven successful in providing a defined end point for building and refining a structure. Hmmm. I always thought things like no more significant explainable (difference) density define endpoints in model building and not R-values. This strategy has proven successful in nailing ligand structures where R-value rules of thumb were used to define the end points. Cheers, BR
Re: [ccp4bb] diverging Rcryst and Rfree [SEC=UNCLASSIFIED]
Anthony, Your rule actually works on the difference (Rfree - Rwork/2), not (Rfree - Rwork) as you said, so is rather different from what most people seem to be using. For example let's say the current values are Rwork = 20, Rfree = 30, so your current test value is (30 - 20/2) = 20. Then according to your rule Rwork = 18, Rfree = 29 is equally acceptable (29 - 18/2 = 20, i.e. same test value), whereas Rwork = 16, Rfree = 29 would not be acceptable by your rule (29 - 16/2 = 21, so the test value is higher). Rwork = 18, Rfree = 28 would represent an improvement by your rule (28 - 18/2 = 19, i.e. a lower test value). You say this criterion "provides a defined end-point", i.e. a minimum in the test value above. However wouldn't other linear combinations of Rwork & Rfree also have a defined minimum value? In particular Rfree itself always has a defined minimum with respect to adding parameters or changing the weights, so would also satisfy your criterion. There has to be some additional criterion that you are relying on to select the particular linear combination (Rfree - Rwork.2) over any of the other possible ones? Cheers -- Ian On Tue, Oct 26, 2010 at 6:33 AM, DUFF, Anthony wrote: > > > One “rule of thumb” based on R and R-free divergence that I impress onto > crystallography students is this: > > > > If a change in refinement strategy or parameters (eg loosening restraints, > introducing TLS) or a round of addition of unimportant water molecules > results in a reduction of R that is more than double the reduction in > R-free, then don’t do it. > > > > This rule of thumb has proven successful in providing a defined end point > for building and refining a structure. > > > > The rule works on the differential of R – R-free divergence. I’ve noticed > that some structures begin with a bigger divergence than others. Different > Rmerge might explain. > > > > Has anyone else found a student in a dark room carefully adding large > numbers of partially occupied water molecules? > > > > > > > > Anthony > > Anthony Duff Telephone: 02 9717 3493 Mob: 043 189 1076 > > > > > > From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Artem > Evdokimov > Sent: Tuesday, 26 October 2010 1:45 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] diverging Rcryst and Rfree > > > > Not that rules of thumb always have to have a rationale, nor that they're > always correct - but it would seem that noise in the data (of which Rmerge > is an indicator) should have a significant relationship with the R:Rfree > difference, since Rfree is not (should not be, if selected correctly) > subject to noise fitting. This rule is easily broken if one refines against > very noisy data (e.g. "that last shell with Rmerge of 55% and > ratio of 1.3 is still good, right?") or if the structure is overfit. The > rule is only an indicative one (i.e. one should get really worried if > R-Rfree looks very different from Rmerge) and it breaks down at very high > and very low resolution (more complete picture by GK and shown in BR's > book). > > Since selection of data and refinement procedures is subject to the > decisions of the practitioner, I suspect that the extreme divergence shown > in the figures that you refer to is probably the result of our own > collective decisions. I have no proof, but I suspect that if a large enough > section of the PDB were to be re-refined using the same methods and the same > data trimming practices, the spread would be considerably more narrow. > That'd be somewhat hard to do - but may be doable now given the abundance of > auto-building and auto-correcting algorithms. > > Artem > > On Mon, Oct 25, 2010 at 9:07 PM, Bernhard Rupp (Hofkristallrat a.D.) > wrote: > > And the rationale for that rule being exactly what? > > > > For stats, see figures 12-23, 12-24 > > http://www.ruppweb.org/garland/gallery/Ch12/index_2.htm > > > > br > > > > From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Artem > Evdokimov > Sent: Monday, October 25, 2010 6:36 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] diverging Rcryst and Rfree > > > > http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg04677.html > > as well as some notes in the older posts :) > > As a very basic rule of thumb, Rfree-Rwork tends to be around Rmerge for the > dataset for refinements that are not overfitted. > > Artem > > On Mon, Oct 25, 2010 at 4:10 PM, Rakesh Joshi wrote: > > Hi all, > > Can anyone comment, in general, on diverging Rcryst and Rfree values(say>7%) > for > structures with kind of low resolutions(2.5-2.9 angstroms)? > > Thanks > RJ > > > >
Re: [ccp4bb] diverging Rcryst and Rfree [SEC=UNCLASSIFIED]
Hi Ian, Yes, I guess my rule does work as you say. If, starting the day from (Rwork = 20, Rfree = 30) abbreviate (20,30), you do something to get (18,29), yes this means that that something was a bare minimum acceptable thing to do. If you do something to get (16,29) (decreased R by 4, Rfree by 1), then I would immediately suspect that that thing that was done introduced excessive over-fitting. If you do something to get (18,28) (decreased R by 2, Rfree by 2), then I would say that the thing that was done was a good thing. Yes, other arbitrary linear combinations could work. Not great analysis of this method was performed. I considered that it came to a question of what degree of over-fitting is acceptable. In practice, this rule stopped endless additions of water molecules and further alternate conformations, and for that purpose the precise point seemed unimportant. However, I also used this rule to determine preferred parameters for BFAC and the matrix weight. Do you think this is a bad rule, and can you point me to a better rule? Replying to BR: >> This rule of thumb has proven successful in providing a defined end point >>for building and refining a structure. >Hmmm. I always thought things like no more significant explainable >(difference) density define endpoints > >in model building and not R-values. This strategy has proven successful in >nailing ligand structures where > >R-value rules of thumb were used to define the end points. Of course, there are other rules. One has to explain all significant residual density. But this tends to be a finite task. The above rule was not applicable to building active sites, or other things that would be discussed directly in a paper. The problem I attempt to address is endless fiddling with features of every-diminishing importance. Apologies if I have missed a recent relevant thread, but are lists of rules of thumb for model building and refinement? Anthony Anthony DuffTelephone: 02 9717 3493 Mob: 043 189 1076 -Original Message- From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Ian Tickle Sent: Wednesday, 27 October 2010 12:53 AM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] diverging Rcryst and Rfree [SEC=UNCLASSIFIED] Anthony, Your rule actually works on the difference (Rfree - Rwork/2), not (Rfree - Rwork) as you said, so is rather different from what most people seem to be using. For example let's say the current values are Rwork = 20, Rfree = 30, so your current test value is (30 - 20/2) = 20. Then according to your rule Rwork = 18, Rfree = 29 is equally acceptable (29 - 18/2 = 20, i.e. same test value), whereas Rwork = 16, Rfree = 29 would not be acceptable by your rule (29 - 16/2 = 21, so the test value is higher). Rwork = 18, Rfree = 28 would represent an improvement by your rule (28 - 18/2 = 19, i.e. a lower test value). You say this criterion "provides a defined end-point", i.e. a minimum in the test value above. However wouldn't other linear combinations of Rwork & Rfree also have a defined minimum value? In particular Rfree itself always has a defined minimum with respect to adding parameters or changing the weights, so would also satisfy your criterion. There has to be some additional criterion that you are relying on to select the particular linear combination (Rfree - Rwork.2) over any of the other possible ones? Cheers -- Ian On Tue, Oct 26, 2010 at 6:33 AM, DUFF, Anthony wrote: > > > One "rule of thumb" based on R and R-free divergence that I impress onto > crystallography students is this: > > > > If a change in refinement strategy or parameters (eg loosening restraints, > introducing TLS) or a round of addition of unimportant water molecules > results in a reduction of R that is more than double the reduction in > R-free, then don't do it. > > > > This rule of thumb has proven successful in providing a defined end point > for building and refining a structure. > > > > The rule works on the differential of R - R-free divergence. I've noticed > that some structures begin with a bigger divergence than others. Different > Rmerge might explain. > > > > Has anyone else found a student in a dark room carefully adding large > numbers of partially occupied water molecules? > > > > > > > > Anthony > > Anthony DuffTelephone: 02 9717 3493 Mob: 043 189 1076 > > > > ________ > > From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Artem > Evdokimov > Sent: Tuesday, 26 October 2010 1:45 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] diverging
Re: [ccp4bb] diverging Rcryst and Rfree [SEC=UNCLASSIFIED]
ddition of unimportant water molecules >> results in a reduction of R that is more than double the reduction in >> R-free, then don’t do it. >> >> >> >> This rule of thumb has proven successful in providing a defined end point >> for building and refining a structure. >> >> >> >> The rule works on the differential of R – R-free divergence. I’ve noticed >> that some structures begin with a bigger divergence than others. Different >> Rmerge might explain. >> >> >> >> Has anyone else found a student in a dark room carefully adding large >> numbers of partially occupied water molecules? >> >> >> >> >> >> >> >> Anthony >> >> Anthony Duff Telephone: 02 9717 3493 Mob: 043 189 1076 >> >> >> >> >> >> From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Artem >> Evdokimov >> Sent: Tuesday, 26 October 2010 1:45 PM >> To: CCP4BB@JISCMAIL.AC.UK >> Subject: Re: [ccp4bb] diverging Rcryst and Rfree >> >> >> >> Not that rules of thumb always have to have a rationale, nor that they're >> always correct - but it would seem that noise in the data (of which Rmerge >> is an indicator) should have a significant relationship with the R:Rfree >> difference, since Rfree is not (should not be, if selected correctly) >> subject to noise fitting. This rule is easily broken if one refines against >> very noisy data (e.g. "that last shell with Rmerge of 55% and >> ratio of 1.3 is still good, right?") or if the structure is overfit. The >> rule is only an indicative one (i.e. one should get really worried if >> R-Rfree looks very different from Rmerge) and it breaks down at very high >> and very low resolution (more complete picture by GK and shown in BR's >> book). >> >> Since selection of data and refinement procedures is subject to the >> decisions of the practitioner, I suspect that the extreme divergence shown >> in the figures that you refer to is probably the result of our own >> collective decisions. I have no proof, but I suspect that if a large enough >> section of the PDB were to be re-refined using the same methods and the same >> data trimming practices, the spread would be considerably more narrow. >> That'd be somewhat hard to do - but may be doable now given the abundance of >> auto-building and auto-correcting algorithms. >> >> Artem >> >> On Mon, Oct 25, 2010 at 9:07 PM, Bernhard Rupp (Hofkristallrat a.D.) >> wrote: >> >> And the rationale for that rule being exactly what? >> >> >> >> For stats, see figures 12-23, 12-24 >> >> http://www.ruppweb.org/garland/gallery/Ch12/index_2.htm >> >> >> >> br >> >> >> >> From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Artem >> Evdokimov >> Sent: Monday, October 25, 2010 6:36 PM >> To: CCP4BB@JISCMAIL.AC.UK >> Subject: Re: [ccp4bb] diverging Rcryst and Rfree >> >> >> >> http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg04677.html >> >> as well as some notes in the older posts :) >> >> As a very basic rule of thumb, Rfree-Rwork tends to be around Rmerge for the >> dataset for refinements that are not overfitted. >> >> Artem >> >> On Mon, Oct 25, 2010 at 4:10 PM, Rakesh Joshi wrote: >> >> Hi all, >> >> Can anyone comment, in general, on diverging Rcryst and Rfree values(say>7%) >> for >> structures with kind of low resolutions(2.5-2.9 angstroms)? >> >> Thanks >> RJ >> >> >> >> >
