This is an interesting thread, and perhaps I should not dive in on such a heady topic, BUT, I do want to point out my own particular bias regarding FOM that is not entirely consistent with James' point of view. In my experience, the FOM obtained after density modification runs are almost always extremely optimistic - I have seen relatively high apparent FOM after density modification runs (0.7) that had nearly uninterpretable maps. As such, I am much more interested in knowing about the FOM from the phasing calculations themselves and NOT after density modification.
That said, applying arbitrary cut-offs to what would be deemed an acceptable FOM, after phasing calculations, to generate maps that are "interpretable" is not really a good thing to do. For instance, I just had a structure where the FOM was 0.35 after phasing (a rubbish structure perhaps?), BUT, the data are highly redundant and the solvent content in the high 70% range. The post density modified maps are stunningly good. One could easily imagine many other scenarios (e.g. NCS) where the modified maps and apparent FOM would be decoupled. So, I do agree with James's suggestion that perhaps we should be retrospectively calculating a "real" FOM between the final model and the actual maps you built into (after whatever you did to get them). This seems like a very good idea indeed. More personal biases revealed: I actually look at, and use, the Cullis R values on anomalous and isomorphous data to help determine how much signal is in the data. Simplified, these numbers are your Average estimated lack-of-closure divided by your average observed difference. That's an important thing to know, and I find them quite useful. Steve -----Original Message----- From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of James Holton Sent: Tuesday, April 13, 2010 1:48 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] Phasing statistics Probably the only phasing stat that I pay any attention to these days is the Figure of Merit (FOM). This is because, the _definition_ of FOM is that it is the cosine of the phase error (or at least your best estimate of it). FOM=1 is perfect phases and FOM=0 is random phases, and a reasonable cutoff value for FOM is 0.5 (see Lunin & Woolfson, Acta D, 1993). Yes, there are ways to get various programs to report very inaccurate values for FOM (such as running DM for thousands of cycles), and yes, there are often legitimate reasons to run these programs in this way. But, there are also very wrong things one can do to get low Rmerge, Rcryst, and especially Rfree. It is simply a matter of knowing (and reporting) what you are doing. If you are worried that your favorite estimate of FOM is inaccurate, then you can always turn to your most accurate phases: those of your final, refined model (the one that you have convinced yourself is "right" and ready to publish). Taking these as the "true phases", the "true FOM" can always be obtained by comparing the final-model phases to those of your initial map (using PHISTATS or SFTOOLS). This is by no means standard practice, but perhaps it should be? Anyway, FOM is _supposed_ to be the cosine of the phase error, and is therefore the most relevant statistic when it comes to how good your phases were when you started building. This is why it is important as a reviewer to know what it is. If I am faced with a structure that was built into a MAD map with initial FOM = 0.8 to 2 A resolution, then I am already convinced that the structure is "right" because I know they had a very clear map to build into. It is hard to do something egregiously wrong with such a map (such as tracing it backwards), so I would even excuse a high R/Rfree in this case, especially if the map has large absent (disordered) regions that the authors were honest enough to not build. On the other hand, if the initial solvent-flattened SAD map had FOM=0.3 to 2 A, you are really pushing it. It is possible to get a correct structure from such a map, but extremely difficult. One might combine some MR phases with the SAD phases to improve them somewhat, but how does one evaluate such a result? I'd say that if FOM < 0.5, then the phases don't make you right. You need to look at other statistics (like R/Rfree). The extreme case, of course, is MR, where the "starting" FOM=0. The author then makes an assumption about the starting phases (based on prior knowledge such as homology with PDB ID = xxxx), and that assumption is then borne out by an "acceptable" R/Rfree (Kleywegt & Brunger, 1996). The "true FOM" (comparing final refined phases to those of the initial MR hit) in this case might still be interesting because it tells you a lot about how much rebuilding had to be done. To answer Frank's question about a 4 A structure with anisotropic diffraction (which I assume means that 4 A is in the best direction, and the other(s) are 5 A or so), I would first ask that the "true" resolution limit be denoted by the point where the average I/sigma drops to ~2 (this is _without_ an anisotropic resolution cutoff!). Then we probably have a 4.5A structure. The "metrics by which we then judge the results?" then depends on the bigger question: "Does the evidence presented justify the conclusions drawn?". If the conclusion is that bond lengths in the active site are "strained", then the answer is obviously "no". Indeed, if the conclusions rely on the helicies in a 4.5 A map being traced in the right direction, then I would also answer "no". This is because at 4.5 A the image of a backward-traced helix looks a _lot_ like the correctly-traced one (see http://bl831.als.lbl.gov/~jamesh/movies/index.html#reso). To put it another way, the R-factors alone are not convincing evidence of a correct trace at 4.5A, and corroborating evidence must be presented to make the helix direction convincing. By "presented", I mean spelled out in the text, and by "corroborating evidence" I mean something as simple as a clear connectivity with enough big side chains placed to deduce the register of the sequence. Barring that, something like "SeMet scanning" can also clarify tracing ambiguities (for a relevant example, see Chen et al. (2007) PNAS 104 p 18999). I am not saying that every 4.5 A structure needs to do this, but I am saying that the number of alternative explanations (models) for a given observation (map) increases as the map gets blurrier, and if a plausible alternative model could change the conclusions of the paper, then it must be eliminated with controls. You know, basic science stuff. It is a common misconception that MAD/SAD/MIR phasing depends on resolution, but nowhere on the Harker diagram does one see the "resolution" of the vectors. The accuracy of the phase depends entirely on the magnitude of the signal (delta-F) and the magnitude of the noise (sigma(F)). This is why you only get experimental phases for strong spots, and never all the way out to your "resolution limit". True, this is a "resolution dependence", but it is actually the signal-to-noise ratio itself that is important. The only part of experimental phasing that seems to be reproducibly resolution-dependent is the density modification used to clean it up. This seems to be limited to pushing your "good phases" out by ~ 1 A in most cases (i.e. from 4A to 3A or from 3.5A to 2.5A, etc.), but I'm not sure why that is. Probably something in histogram matching. Unfortunately, I am not aware of a good comprehensive review of the resolution dependence of phase extension, possibly because one cannot do such an analysis with the data currently available in the PDB (initial phases are not deposited). I would finally like to note that I am highly uncomfortable with the idea of excusing the reporting of data processing statistics if the structure is deemed "correct". Formally, no protein structure is intrinsically "correct" if it does not explain the data (Fobs) to withing experimental error (~5%). In the "small molecule world" models with Rcryst > Rmerge are rejected out-of-hand (and for good reason). The only reason protein structures are "excused" from this rule is because they have a good "track record" of agreeing with experimentally-phased maps. -James Holton MAD Scientist Frank von Delft wrote: > I fully agree, for high quality data. > > What though if the data are not impeccable and the structure > necessarily ropey? E.g. 4A phases and anisotropic diffraction. By > what metrics do we then judge the results? > > (I don't know the answer, btw, but our membranous colleagues surely > spend quite a bit of time with that question...) > > phx. > > > On 12/04/2010 12:10, Anastassis Perrakis wrote: >> Hi - >> >> A year or so ago, I have asked as a referee somebody to provide for a >> paper the statistics for their heavy atom derivative dataset, >> and for the phasing statistics. For some good reasons, they were >> unable to do that, and they (politely) asked me >> 'what would it change if you knew these, isn't the structure we >> present impeccable?'. Well, I think they were right. >> Their structure was surely correct, surely high quality. After that >> incident and giving it some thought, >> I fail to see why should one report e.g. PP or Rcullis, or why will I >> care what they were if the structure has a convincing Rfree and is >> properly validated. >> If someone wants to cheat at the end of the day, its easy to provide >> two numbers, but its hard to provide a good validated model that >> agrees with the data. >> (and, yes, you can also make up the data, but we have been there, >> haven't we?!?) >> >> So, my question to that referee, likely being a ccp4bb aficionado >> that is reading this email, or to anyone else really, is: >> >> "What would it help to judge the quality of the structure or the >> paper if you know PP, Rcullis and FOM?" >> >> Best - >> >> A. >> >> PS Especially since you used SHELXE for phasing these statistics are >> utterly irrelevant, and possibly you could advice the referee to read >> a bit about how SHELXE works ... or go to one of the nice courses >> that George teaches ... >> >> On Apr 12, 2010, at 10:37, Eleanor Dodson wrote: >> >>> You can feed the SHELX sites into phaser_er or CRANK both of which will >>> give this sort of information. >>> >>> Or mlphare if you know how to set it up.. >>> >>> Eleanor >>> >>> >>> Harmer, Nicholas wrote: >>>> Dear CCP4ers, >>>> >>>> I've been asked by a referee to provide the phasing statistics for >>>> a SAD dataset that I used to solve a recent structure. Whilst I >>>> have been able to find a figure-of-merit for the data after >>>> phasing, I can't work out how to get any other statistics (e.g. >>>> phasing power or an equivalent or Rcullis). Does anyone know a good >>>> route to obtaining useful statistics to put in the paper for SAD data? >>>> >>>> The structure solution was carried out using SHELX C/D/E and then >>>> ARP/wARP. >>>> >>>> Thanks in advance, >>>> >>>> Nic Harmer >>>> >>>> ===================== >>>> Dr. Nic Harmer >>>> School of Biosciences >>>> University of Exeter >>>> tel: +44 1392 725179 >>>> >> >> *P** **please don't print this e-mail unless you really need to* >> Anastassis (Tassos) Perrakis, Principal Investigator / Staff Member >> Department of Biochemistry (B8) >> Netherlands Cancer Institute, >> Dept. B8, 1066 CX Amsterdam, The Netherlands >> Tel: +31 20 512 1951 Fax: +31 20 512 1954 Mobile / SMS: +31 6 28 597791 >> >> >> >> Notice: This e-mail message, together with any attachments, contains information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, New Jersey, USA 08889), and/or its affiliates Direct contact information for affiliates is available at http://www.merck.com/contact/contacts.html) that may be confidential, proprietary copyrighted and/or legally privileged. It is intended solely for the use of the individual or entity named on this message. If you are not the intended recipient, and have received this message in error, please notify us immediately by reply e-mail and then delete it from your system.
