Dear Gerard, Many thanks for these useful clarifications.I see your points clearly.
Just to mention that one remark in James's posting regarding photon counting versus read noise caught my attention. I will follow up on this ASAP , which like fine phi slicing gets to the heart of the measurement physics. Greetings, John Prof John R Helliwell DSc FInstP CPhys FRSC CChem F Soc Biol. Chair School of Chemistry, University of Manchester, Athena Swan Team. http://www.chemistry.manchester.ac.uk/aboutus/athena/index.html On 19 May 2013, at 21:06, Gerard Bricogne <g...@globalphasing.com> wrote: > Dear John, > > Thank you for your message. I do realise that what I wrote may > have sounded like a "categoric blanket endorsement of XDS". Perhaps I > should have slept on my draft message a little longer, but James's > long e-mail made me wake up and it seemed appropriate to say what I > wanted to say, in the form I had written it, withouth further delay. > > My main point was to try and shake people out of habits that > are, in part at least, linked to the use of integration programs still > based on a 2D analysis of diffraction images, and to getting misled > into thinking that fine slicing doesn't help because it doesn't help > these programs. What I most wanted to put across is that the merits of > fine slicing and low-exposure, high-multiplicity collection protocols > will emerge faster if people are encouraged to evaluate the data they > yield through processing with XDS - hence my strong endorsement of it > in this context. > > I do not consider XDS to be beyond perfectibility by any means, > but I was reluctant to make my e-mail longer by analysing its possible > improvements. Once its use has spread enough to give users of Pilatus > detectors the best data they can hope for in the current state of the > art, then its limitations will make concrete sense to enough people > for their discussion to move spontaneously to the top of the agenda. > It is indeed because I think there is so much scope for improvement on > the current diffraction image processing software, including XDS, that > I have been strongly (some would perhaps say: stridently) advocating > the deposition of raw diffraction image data, as both an incentive and > a testing ground for such developments in the future. > > > With best wishes, > > Gerard. > > > > On Sun, May 19, 2013 at 08:01:34PM +0100, John R Helliwell wrote: >> Dear Gerard, >> Thank you for sharing these extensive details which I feel sure everyone >> will appreciate. >> Just one aspect I wondered about namely your categorical blanket >> endorsement of XDS. Indeed a very fine program and eg most recently >> evaluated and discussed at CCP4 2011 I think it was, where it emerged 'the >> winner'. You probably guess though that I am thinking of our mutually >> emphasized point that one key reason for raw diffraction data images >> availability is to see such software improve. Is XDS already perfection? Is >> its use by users already guaranteed to yield processed data as good as it >> can get? >> Greetings, >> John >> Prof John R Helliwell DSc >> >> >> On Thu, May 16, 2013 at 6:03 PM, Gerard Bricogne >> <g...@globalphasing.com>wrote: >> >>> Dear James, >>> >>> A week ago I wrote what I thought was a perhaps excessively long and >>> overly dense message in reply to Theresa's initial query, then I thought I >>> should sleep on it before sending it, and got distracted by other things. >>> >>> I guess you may well have used that whole week composing yours ;-) and >>> reading it just now makes the temptation of sending mine irresistible. I am >>> largely in agreement with you about the need to change mental habits in >>> this >>> field, and hope that the emphasis on various matters in my message below is >>> sufficiently different from yours to make a distinct contribution to this >>> very important discussion. Your analysis of pile-up effects goes well >>> beyond >>> anything I have ever looked at. However, in line with Theresa's initial >>> question, I would say that, while I agree with you that the best strategy >>> for collecting "native data" is no strategy at all, this isn't the case >>> when >>> collecting data for phasing. In that case one needs to go back and consider >>> how to measure accurate differences of intensities, not just accurate >>> intensities on their own. That is another subject, on which I was going to >>> follow up so as to fully answer Theresa's message - but perhaps that should >>> come in another installment! >>> >>> >>> With best wishes, >>> >>> Gerard. >>> >>> -- >>> On Tue, May 07, 2013 at 12:04:33AM +0100, Theresa Hsu wrote: >>>> Dear crystallographers >>>> >>>> Is there a good source/review/software to obtain tips for good data >>> collection strategy using PILATUS detectors at synchrotron? Do we need to >>> collect sweeps of high and low resolution data separately? For anomalous >>> phasing (MAD), does the order of wavelengths used affect structure solution >>> or limit radiation damage? >>>> >>>> Thank you. >>>> >>>> Theresa >>> -- >>> >>> Dear Theresa, >>> >>> You have had several excellent replies to your question. Perhaps I >>> could venture to add a few more comments, remarks and suggestions, which >>> can >>> be summarised as follows: with a Pilatus, (1) use fine slicing, (2) use >>> strategies combining low exposure with high multiplicity, and (3) use XDS! >>> >>> As the use of Pilatus detectors has spread widely, it has been rather >>> puzzling to come across so many instances when these detectors are misused, >>> sometimes on the basis of explicit expert advice that is simply misguided. >>> A >>> typical example will be to see images collected on a Pilatus 6M with an >>> image width of 1 degree and an exposure time of 1 second. When you see >>> this, >>> you know that there is some erroneous thinking (or habit) behind it. >>> >>> When talking to various users who have ended up with such datasets, >>> and >>> with people who advocate this kind of strategy, it seems clear that a >>> number >>> of irrational concerns about fine-slicing and >>> low-exposure+high-multiplicity >>> strategies have tended to override published rational arguments in favour >>> of >>> those strategies: there is a fear that if the images being collected do not >>> show spots discernible by the naked eye to the resolution limit that is >>> being aimed for, the integration software will then somehow not be able to >>> find those spots in order to integrate them, and the final data resolution >>> will be lower than expected. Perhaps this may be of concern in relation >>> with >>> the use of some integration programs, but if you use XDS, which implements >>> a >>> full 3D approach to image integration, this is simply not the case: XDS >>> will >>> collect all the counts belonging to a given reflection, whether those >>> counts >>> are all from a spot on a single 1-degree image exposed for 1 second, or >>> from >>> 10 consecutive images of 0.1 degree width exposed for 0.1 second each, or >>> from 100 images obtained by grouping together the same 10 images as >>> previously collected in 10 successive passes with a 10-fold attenuated >>> beam. >>> The hallmark of the Pilatus detector is to lead to equivalent signal/noise >>> ratios for the last two ways of measuring that reflection, because it is a >>> photon counter and has zero readout noise: therefore the combination >>> Pilatus+XDS is a powerful one. >>> >>> What is different between these three strategies, however, is the >>> quality of the overall dataset they will produce. There is nothing new in >>> what I am describing below: it is all in the references that Bob Sweet gave >>> you in his reply, or is an obvious consequence of what is found in these >>> references. >>> >>> In case 1 (1-degree, 1 second - "coarse slicing") you would presumably >>> also be (mis-)advised to use a strategy aiming at collecting a complete >>> dataset in the minimum number of images. These strategies used to make >>> sense >>> in the days of films, of image plates, and even of CCDs because of the >>> image >>> readout noise, but they have no place any longer in the context of Pilatus >>> detectors. First of all, using 1-degree image widths can only degrade the >>> precision with which 2D spots on images are lifted to 3D reciprocal space >>> for indexing, and hence worsen the quality of that indexing and therefore >>> the accuracy with which the spot locations will be predicted (unless you >>> carefully "post-refine") - then the integration step perhaps does need to >>> "hunt" for those spots locally, and needs them to be somewhat visible. >>> Secondly, 1 degree is usually greater than the angular width of a typical >>> reflection: the integration process will therefore pick up more background >>> noise (variance) than it would have done with a smaller image width. >>> Thirdly, by collecting only enough images to reach completeness you will >>> have substantial radiation damage in your late images compared to the early >>> ones (if you don't, it means you have under-exposed your crystal) and will >>> therefore end up with internal inconsistencies in your dataset, as well as >>> perhaps some extra, spurious anisotropy of diffraction limits as a result >>> of >>> having to impose increasingly stringent resolution cut-offs in the later >>> images. This will affect the internal scaling of that dataset and the final >>> quality of the merged data. >>> >>> In case 2 (0.1 degree, 0.1 second - "fine slicing") you will have a >>> more precise sampling of the 3D shape of each spot, hence more accurate >>> indexing and prediction of spot positions if you use a genuinely 3D >>> integration program like XDS. Thanks to that increased precision, spots can >>> be integrated "blind", even if they are not terribly visible in the images, >>> and the same number of photons will be collected with no penalty in terms >>> of >>> noise level, thanks to the photon-counting noiseless-readout nature of the >>> Pilatus detector. An improvement will be that the finely sampled 3D shape >>> of >>> the spots will be used by XDS to minimise the impact of background variance >>> on the integrated intensities. On the other hand, the differential >>> radiation >>> damage between early and late images will still be the same as in case 1 if >>> you have chosen one of those old-style strategies (and associated beam >>> intensity setting) that aim at just about exhausting the useful lifetime of >>> the crystal by the time you reach completeness. >>> >>> In case 3 (like case 2, but collecting n times more images with an >>> n-fold attenuated beam once you have collected a few "characterisation >>> images" without that attenuation to carry out the initial indexing) you >>> still have the two advantages of case 2 (the same total number of photons >>> will be picked up by XDS, even if the individual images are now so weak >>> that >>> you can't see anything) but you are spreading the radiation damage so >>> thinly >>> over multiple successive complete datasets that you can choose to later >>> apply a cut-off on image number at the processing stage, when the >>> statistics >>> tell you that diffraction quality has become degraded beyond some critical >>> level. This is much preferable to having to apply different resolution >>> cut-offs to different images towards the end of a barely complete dataset, >>> as in cases 1 and 2. The impact of radiation damage will be quite smoothly >>> and uniformly distributed across the final unique reflections, and your >>> scaling problems (as well as any spurious anisotropy in your diffraction >>> limits) will be minimised. >>> >>> >>> This is becoming quite a long message: you can see why I included a >>> summary of it at the beginning! Returning to it for a conclusion: Pilatus >>> detectors, fine-slicing with low-exposure and high-multiplicity strategies, >>> and XDS are a unique winning combination. If fears that another integration >>> program may not perform as well as XDS on fine-sliced data make you feel >>> tempted to revert to old-fashioned strategies (case 1) because it >>> supposedly >>> makes no difference: resist the temptation! Switch to those Pilatus-adapted >>> strategies and to XDS, and enjoy the very real difference in the results! >>> >>> >>> With best wishes, >>> >>> Gerard >>> >>> and colleagues at Global Phasing. >>> >>> -- >>> On Tue, May 07, 2013 at 12:04:33AM +0100, Theresa Hsu wrote: >>>> Dear crystallographers >>>> >>>> Is there a good source/review/software to obtain tips for good data >>> collection strategy using PILATUS detectors at synchrotron? Do we need to >>> collect sweeps of high and low resolution data separately? For anomalous >>> phasing (MAD), does the order of wavelengths used affect structure solution >>> or limit radiation damage? >>>> >>>> Thank you. >>>> >>>> Theresa >>> >>> -- >>> >>> =============================================================== >>> * * >>> * Gerard Bricogne g...@globalphasing.com * >>> * * >>> * Global Phasing Ltd. * >>> * Sheraton House, Castle Park Tel: +44-(0)1223-353033 * >>> * Cambridge CB3 0AX, UK Fax: +44-(0)1223-366889 * >>> * * >>> =============================================================== >>> >> >> >> >> -- >> Professor John R Helliwell DSc > > -- > > =============================================================== > * * > * Gerard Bricogne g...@globalphasing.com * > * * > * Global Phasing Ltd. * > * Sheraton House, Castle Park Tel: +44-(0)1223-353033 * > * Cambridge CB3 0AX, UK Fax: +44-(0)1223-366889 * > * * > ===============================================================
