I'll add it at some stage Phil On 2 Nov 2010, at 09:38, Ian Tickle wrote:
> I would say that being able to take in phases & H-L coeffs would be a > useful option, because sometimes (I accept it's not mandatory) phases > are available in SF files from PDB which allows us to see exactly the > same map that was interpreted by the depositor. In some cases there's > some doubt that the density for an inhibitor or whatever claimed by > the depositor is actually there (usually of course that's the whole > point of their paper!!!). If you recalculate the phases there's > always a nagging doubt that you've done something to change the > density (e.g. different solvent model or whatever). The depositor's > own density is damning evidence! > > Cheers > > -- Ian > > On Tue, Nov 2, 2010 at 8:56 AM, Phil Evans <p...@mrc-lmb.cam.ac.uk> wrote: >> As Ian says, it doesn't modify the reference set. >> >> Yes it is a subsidiary function of Pointless, aimed at replacing the program >> REINDEX (since much of the functionality was already in there, it seemed >> sensible to add this option). It ought to do appropriate phase changes, but >> like Reindex, it doesn't at present. My feeling was that you can always >> recalculate the phases after a reindexing. >> >> Phil >> >> On 2 Nov 2010, at 08:28, Ian Tickle wrote: >> >>> Re-indexing of an input (either merged or unmerged) dataset to a (also >>> merged or unmerged) reference dataset is one of the 'subsidiary' >>> functions of pointless, according to Phil's man page: >>> >>> This mode is selected if an HKLREF dataset is specified. >>> Given a test dataset, merged or unmerged (file(s) HKLIN), and a merged >>> or unmerged reference dataset in a known space group (file HKLREF), the >>> program tests all possible alternative indexing schemes of the test >>> dataset to find which one best matches the reference set. >>> >>> Not sure what you mean by "Why would you need to modify the reference >>> data?" ? The reference data is not modified, who said it was? >>> >>> -- Ian >>> >>> On Tue, Nov 2, 2010 at 7:03 AM, Eleanor Dodson <c...@ysbl.york.ac.uk> wrote: >>>> CAD and Kevins phasematch correctly change phases etc when you change >>>> symmetry operator. >>>> >>>> I cant think that this is the job for pointless.. it is responsible for >>>> intensities, and surely only needs to use a merged file to decide on the >>>> appropriate choice of axes - eg getting your new PG3 data set on the same >>>> indexing convention? Why would you need to modify the reference data? >>>> Eleanor >>>> >>>> On 11/01/2010 03:41 PM, Ian Tickle wrote: >>>>> >>>>> Dealing with the phases (and therefore also the Hendrickson-Lattman >>>>> coefficients) on re-indexing is trivial: the phases are not changed by >>>>> re-indexing because the inverse transformation must simultaneously be >>>>> applied to the co-ordinates. This is because in general the >>>>> re-indexing transformation is not necessarily a symmetry operator >>>>> (think of P1), so you can't rely on being able to compensate for the >>>>> effect on the co-ordinates by using a symmetry operator. So the >>>>> effect on the phases cancels out ... unless of course your >>>>> re-indexing operator inverts the hand, in which case you almost >>>>> certainly don't want also to invert the hand of your co-ordinates, so >>>>> in that case you must compensate by transforming the structure factors >>>>> to their complex conjugates (i.e. multiply phases by -1). I guess >>>>> you're thinking of the subsequent necessary transformation of the >>>>> indices to the asymmetric unit, where the phases& H-L coeffs do in >>>>> general change (because then you are only changing the indices, not >>>>> the co-ordinates); however CAD will do that transformation for you. >>>>> >>>>> Incidentally this is a neat illustration of the difference between a >>>>> vector and a complex number. The re-indexing transformation is a >>>>> transformation of the reference frame, which as long as it doesn't >>>>> invert the hand, leaves the complex structure factors invariant, so >>>>> they must be complex scalars (except in centric zones where they can >>>>> sometimes be represented by real scalars). The indices (whether >>>>> reflection or Miller!) obviously form a 3-D vector with integer >>>>> elements (unless of course you're interested in diffuse scattering >>>>> when they have to be reals). Either way, this is a vector because in >>>>> the general case (there will be exceptions for reflections on symmetry >>>>> axes) its elements change on re-indexing (that's what re-indexing >>>>> means!). If the structure were in 1-D or 2-D exactly the same would >>>>> apply: the 1- or 2-D elements would still in general change on >>>>> transforming the reference frame so would be represented by 1- or 2-D >>>>> vectors; the structure factors would still be invariant, thus >>>>> illustrating the important difference between a real scalar and a 1-D >>>>> vector, and between a complex scalar and a 2-D vector. >>>>> >>>>> Cheers >>>>> >>>>> --Ian >>>>> >>>>> On Mon, Nov 1, 2010 at 1:28 PM, Phil Evans<p...@mrc-lmb.cam.ac.uk> wrote: >>>>>> >>>>>> I can see we need to make sure that data can come in at any point, as Is >>>>>> of Fs >>>>>> >>>>>> Pointless can do automatic reindexing to a reference, and will preserve >>>>>> all columns from a merged file, but can't cope with phases, as I've not >>>>>> got >>>>>> round to working out appropriate phase shifts on reindexing >>>>>> >>>>>> Phil >>>>>> >>>>>> >>>>>> On 1 Nov 2010, at 13:17, Ian Tickle wrote: >>>>>> >>>>>>> Phil >>>>>>> >>>>>>> Yes our processing pipeline absolutely has to be able to take in data >>>>>>> from any internal (in-house X-ray or synchrotron) or external (PDB or >>>>>>> collaborator's) source, including those where I, sigI and freeR flag >>>>>>> are present. One of the first things I did was to modify truncate so >>>>>>> it would pass through the freeR flag column. If the I/sigI are >>>>>>> present I always strip out the F/sigF columns. So it seemed logical >>>>>>> to run truncate as the very last step, e.g.: >>>>>>> >>>>>>> 1. sortmtz >>>>>>> 2. scala Steps 1& 2 only for internally collected or unmerged >>>>>>> data. >>>>>>> 3. refindex External merged data enters pipeline here: >>>>>>> auto-re-index to reference. >>>>>>> 4. cad Sort; put into standard a.u.; add freeR column from >>>>>>> reference if not already present. >>>>>>> 5. rescut My own prog for auto-determination of resolution cutoff >>>>>>> based on shell<I/sigI> & completeness. >>>>>>> 6. truncate Apply resolution cutoff; if Is available convert to Fs. >>>>>>> >>>>>>> I always run steps 3-6 in that order. I always check that the >>>>>>> resolution cutoff is sensible& if Is are available I always run >>>>>>> truncate to ensure it's done properly (i.e. correct cell contents are >>>>>>> specified). I'm still using truncate because AFAICS ctruncate >>>>>>> couldn't handle freeR flags (maybe that's fixed now, maybe not). Also >>>>>>> truncate produces a more informative N(Z) plot which shows the >>>>>>> expected distribution for a twinned crystal (I believe this feature >>>>>>> has now been added to ctruncate). >>>>>>> >>>>>>> Cheers >>>>>>> >>>>>>> -- Ian >>>>>>> >>>>>>> On Fri, Oct 29, 2010 at 1:05 PM, Phil Evans<p...@mrc-lmb.cam.ac.uk> >>>>>>> wrote: >>>>>>>> >>>>>>>> The normal use of [c]truncate is to take intensities from Scala, so it >>>>>>>> wouldn't expect FreeR flags in the file. >>>>>>>> >>>>>>>> I suppose this should be added for other uses of the program >>>>>>>> >>>>>>>> Is this something that is often used? Do people import intensities into >>>>>>>> CCP4 to convert them to Fs? >>>>>>>> >>>>>>>> Phil >>>>>>>> >>>>>>>> >>>>>>>> On 29 Oct 2010, at 13:01, herman.schreu...@sanofi-aventis.com wrote: >>>>>>>> >>>>>>>>> Dear Peter, >>>>>>>>> >>>>>>>>> Since I did not hear that your problem is solved here my two cents. I >>>>>>>>> did some tests using the ccp4i option "Convert Intensities to SFs" and >>>>>>>>> found that here ctruncate completely ignored the FreeRflags. So my >>>>>>>>> conclusion is that ctruncate does not need FreeRflags and you can use >>>>>>>>> the following procedure: >>>>>>>>> >>>>>>>>> 1) convert your hkl file (including FreeRflags) into an mtz with f2mtz >>>>>>>>> without any special SHELX options. --> mtz 1 >>>>>>>>> Careful: a FreeRflag of 1 means an unfree reflection and the free >>>>>>>>> reflections have a FreeRflag of zero. >>>>>>>>> 2) run ctruncate with the "Convert Intensities to SFs". You will loose >>>>>>>>> your FreeRflags in this stage. --> mtz 2 >>>>>>>>> 3) add the FreeRflags from mtz 1 to mtz 2 using cad. >>>>>>>>> >>>>>>>>> If you wish, I can give you a command file which will do this. It is a >>>>>>>>> somewhat roundabout procedure and I hope that this bug (or feature) >>>>>>>>> will >>>>>>>>> be fixed by the next release of ccp4. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Herman >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of >>>>>>>>> George M. Sheldrick >>>>>>>>> Sent: Friday, October 29, 2010 12:30 PM >>>>>>>>> To: CCP4BB@JISCMAIL.AC.UK >>>>>>>>> Subject: Re: [ccp4bb] Bug in c_truncate? >>>>>>>>> >>>>>>>>> Tim, >>>>>>>>> >>>>>>>>> Although I always like to advocate XPREP, that would not work because >>>>>>>>> the .sca format - most unfortunately - does not know about free R >>>>>>>>> flags. >>>>>>>>> >>>>>>>>> George >>>>>>>>> >>>>>>>>> Prof. George M. Sheldrick FRS >>>>>>>>> Dept. Structural Chemistry, >>>>>>>>> University of Goettingen, >>>>>>>>> Tammannstr. 4, >>>>>>>>> D37077 Goettingen, Germany >>>>>>>>> Tel. +49-551-39-3021 or -3068 >>>>>>>>> Fax. +49-551-39-22582 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, 29 Oct 2010, Tim Gruene wrote: >>>>>>>>> >>>>>>>>>> Hello Peter, >>>>>>>>>> >>>>>>>>>> the easiest way to overcome the problem might be to use xprep to >>>>>>>>>> export to sca-format and use scalepack2mtz for the conversion. That >>>>>>>>>> seems to be the least hasslesome way, although I am not totally sure >>>>>>>>>> that this procedure preserves the R-free flags set by xprep. >>>>>>>>>> >>>>>>>>>> Tim >>>>>>>>>> >>>>>>>>>> On Thu, Oct 28, 2010 at 12:48:14PM -0400, Peter Chan wrote: >>>>>>>>>>> >>>>>>>>>>> Hello Tim, >>>>>>>>>>> >>>>>>>>>>> Thank you for the suggestion. I have now tagged the working set as >>>>>>>>> >>>>>>>>> "1" and test set as "0". Unfortunately, it still gives the same error >>>>>>>>> about all Rfree being the same, and only in c-truncate but not >>>>>>>>> old-truncate. Perhaps I should install 6.1.3 and see if the problem >>>>>>>>> still persist. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Peter >>>>>>>>>>> >>>>>>>>>>>> Date: Thu, 28 Oct 2010 16:29:31 +0200 >>>>>>>>>>>> From: t...@shelx.uni-ac.gwdg.de >>>>>>>>>>>> Subject: Re: [ccp4bb] Bug in c_truncate? >>>>>>>>>>>> To: CCP4BB@JISCMAIL.AC.UK >>>>>>>>>>>> >>>>>>>>>>>> Hello Peter, >>>>>>>>>>>> >>>>>>>>>>>> I faintly rememeber a similar kind of problem, and think that if >>>>>>>>>>>> you replace "-1" with "0", the problem should go away. It seemed >>>>>>>>>>>> that "-1" is not an allowed flag for (some) ccp4 programs. >>>>>>>>>>>> >>>>>>>>>>>> Please let us know if this resolves the issue. >>>>>>>>>>>> >>>>>>>>>>>> Tim >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Oct 28, 2010 at 10:21:20AM -0400, Peter Chan wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Dear Crystallographers, >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you all for the emails. Below are some details of the >>>>>>>>> >>>>>>>>> procedures I performed leading up to the problem. >>>>>>>>>>>>> >>>>>>>>>>>>> The reflection file is my own data, processed in XDS and then >>>>>>>>> >>>>>>>>> flagging FreeR's in XPREP in thin resolution shells. I am using CCP4i >>>>>>>>> version 6.1.2. I tried looking for known/resolved issues/updates in >>>>>>>>> version 6.1.3 but could not find any so I assumed it is the same >>>>>>>>> version >>>>>>>>> of f2mtz/ctruncate/uniqueify. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I used the GUI version of F2MTZ, with the settings below: >>>>>>>>>>>>> >>>>>>>>>>>>> - import file in SHELX format >>>>>>>>>>>>> >>>>>>>>>>>>> - "keep existing FreeR flags" >>>>>>>>>>>>> >>>>>>>>>>>>> - fortran format (3F4.0,2F8.3,F4.0) >>>>>>>>>>>>> >>>>>>>>>>>>> - added data label "I other integer" // FreeRflag >>>>>>>>>>>>> >>>>>>>>>>>>> The hkl file, in SHELX format, output by XPREP look something >>>>>>>>> >>>>>>>>> like this: >>>>>>>>>>>>> >>>>>>>>>>>>> -26 -3 1 777.48 39.19 >>>>>>>>>>>>> 26 -3 -1 800.83 36.31 >>>>>>>>>>>>> -26 3 -1 782.67 37.97 >>>>>>>>>>>>> 27 -3 1 45.722 25.711 -1 >>>>>>>>>>>>> -27 3 1 -14.20 31.69 -1 >>>>>>>>>>>>> >>>>>>>>>>>>> Notice the test set is flagged "-1" and the working set is not >>>>>>>>> >>>>>>>>> flagged at all. This actually lead to another error message in f2mtz >>>>>>>>> about missing FreeR flags. From my understanding, the SHELX flagging >>>>>>>>> convention is "1" for working and "-1" for test. So I manually tagged >>>>>>>>> the working set with "1" using vi: >>>>>>>>>>>>> >>>>>>>>>>>>> -26 -3 1 777.48 39.19 1 >>>>>>>>>>>>> 26 -3 -1 800.83 36.31 1 >>>>>>>>>>>>> -26 3 -1 782.67 37.97 1 >>>>>>>>>>>>> 27 -3 1 45.722 25.711 -1 >>>>>>>>>>>>> -27 3 1 -14.20 31.69 -1 >>>>>>>>>>>>> >>>>>>>>>>>>> This is the file which gives me the error message: "Problem with >>>>>>>>> >>>>>>>>> FREE column in input file. All flags apparently identical. Check input >>>>>>>>> file.". Apparently, import to mtz works ok when I use old-truncate >>>>>>>>> instead of c-truncate. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Peter >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> -- >>>>>>>>>>>> Tim Gruene >>>>>>>>>>>> Institut fuer anorganische Chemie >>>>>>>>>>>> Tammannstr. 4 >>>>>>>>>>>> D-37077 Goettingen >>>>>>>>>>>> >>>>>>>>>>>> phone: +49 (0)551 39 22149 >>>>>>>>>>>> >>>>>>>>>>>> GPG Key ID = A46BEE1A >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> Tim Gruene >>>>>>>>>> Institut fuer anorganische Chemie >>>>>>>>>> Tammannstr. 4 >>>>>>>>>> D-37077 Goettingen >>>>>>>>>> >>>>>>>>>> phone: +49 (0)551 39 22149 >>>>>>>>>> >>>>>>>>>> GPG Key ID = A46BEE1A >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>>> >>
