Re: [ccp4bb] MTZ to Shel-X?
PS Meant to say, I am aware the GUI doesn't give you the option to specify I for SHELX. Will fix in due course (meanwhile RunView Com) m On Tue, 2007-03-06 at 13:43 +, Martyn Winn wrote: Hi Ian, Thanks for your helpful analysis. There are now updated versions of the source code and the documentation at: ftp://ftp.ccp4.ac.uk/ccp4/6.0.2/patches/mtz2various-r1_130.f ftp://ftp.ccp4.ac.uk/ccp4/6.0.2/patches/mtz2various-r1_39.html or via the Problems Pages. In particular: 1. The foo.ins stub at the top of the output has been removed, as have the additional FREEs. 2. The automatic scaling that was used for the FSQUARED option has been extended to the I option. This can be overridden by the SCALe keyword. 3. The option LABIN I=I SIGI=SIGI does in fact write out the intensities, and can be used instead of the FSQUARED option. The output was mis-labelled HKLF 3 and I've corrected that. 4. Some tidying of the documentation. Cheers Martyn
Re: [ccp4bb] MTZ to Shel-X?
Of course, if the refinement is not too far along, you could start over and first import the scaled data into SHELX, generate the R-free flags using SHELXPro, and THEN use mtz2various to convert the data to MTZ format whilst taking care to keep the original R-free flags. Worked very well for me. Art On Mar 1, 2007, at 12:41 PM, Martin Hallberg wrote: I would probably prioritize keeping the same R-free set (thus using the F^2 output by mtz2various) over going through the scalepack format and loosing track of it. You can however use XPREP to transfer the R-free set from the inferior F^2 HKLF4 file output by mtz2various to the proper intensities read from the scalepack file generated by mtz2sca. Then you can write out an HKLF4 file for refinement in SHELXL-97. That HKLF4 file will have proper intensities and the same Rfree set that you have used in Refmac5 (or perhaps Restrain?). But then one would wish that mtz2various did this from the beginning... Best regards, Martin On Mar 1, 2007, at 7:58 PM, Ethan Merritt wrote: On Thursday 01 March 2007 10:17, Ian Tickle wrote: All, I thought this would be a simple task, but for the life of me I can't see how to do it! All I want to do is convert an MTZ file to Shel-X format for refinement. I thought it would take me 2 secs, but it's taken me at least 5 attempts, and it's still not right! Do the conversion on the shelx side, rather than the CCP4 side. http://shelx.uni-ac.gwdg.de/~tg/mtz2sca/mtz2sca.html . B. Martin Hallberg, PhD Molecular Cell Biology Program Department of Cell and Molecular Biology Karolinska Institute Nobels väg 3 SE-171 77 Stockholm
Re: [ccp4bb] MTZ to Shel-X?
You are absolutely right! The difficulty in getting from MTZ to any other format or back is unacceptable. Expecting working crystallographers to write Fortran format statements is ridiculous. I've been trying to address this by adding support for other formats to clipper, but my pace has been glacial, owing to the limited academic rewards for such work. Even then, it needs someone to put together an application and GUI to use it with. I will try and remember to raise this issue at the CCP4 developers meeting this year. If you prod me in April and I'll report back to the BB. Kevin George M. Sheldrick wrote: Dear Ian, It is all part of a diabolical CCP4 plot to make it as inconvenient as possible to move from a REFMAC refinement to SHELXL! I hope that i do not get excommunicated like DVD for this comment. To summarize your and Martin's suggestions, I know of only two ways to move from mtz to hkl: = 1] run mtz2various with the keywords: LABIN FP=FP SIGFP=SIGFP FREE=FreeR_flag OUTPUT SHELX FSQUAR then edit the resulting .hkl file with any text editor to remove the header (which SHELXL cannot read) and all occurences of the word 'FREE'. I have from time to time suggested that mtz2various be changed so that it no longer outputs the extra junk, but it is still there (at least in CCP4-6.0.2). This produces an 'inferior' SHELXL HKLF4 format file because the intensities and their standard deviations have been converted to F and back to I, which requires the assumption of an intensity distribution function that may not be completely valid (e.g. if NCS is present) and so produces inferior sigma(I) values. In practice one can live with this, but it is not very scientific. = 2] Use Tim Gruene's mtz2sca to convert from .mtz to .sca. mtz2sca name reads name.mtz and writes name.sca. This program is avaliable from Tim or from the SHELX download area (Linux only). This has the advantage that it uses intensities if it can find them in the .mtz (e.g. if they are still there from SCALA) and otherwise uses F's. mtz2sca is fine if you want to do exprimental phasing (e.g. read the data into SHELXC either directly or via Thomas Schneider's hkl2map) but unfortunately .sca format does not know about Free-R flags. So, as Martin suggested, one can read the .sca file into xprep (start xprep without a file name and give the full name of the .sca file when prompted) and then use the xprep option to transfer the Free-R flags from the 'inferior' .hkl file from mtz2various and write it out in SHELXL HKLF4 format. This produces a slightly better file for refinement when the .mtz file contains intensities, but requires xprep. George Martin Hallberg wrote: I would probably prioritize keeping the same R-free set (thus using the F^2 output by mtz2various) over going through the scalepack format and loosing track of it. You can however use XPREP to transfer the R-free set from the inferior F^2 HKLF4 file output by mtz2various to the proper intensities read from the scalepack file generated by mtz2sca. Then you can write out an HKLF4 file for refinement in SHELXL-97. That HKLF4 file will have proper intensities and the same Rfree set that you have used in Refmac5 (or perhaps Restrain?). But then one would wish that mtz2various did this from the beginning... Best regards, Martin On Mar 1, 2007, at 7:58 PM, Ethan Merritt wrote: On Thursday 01 March 2007 10:17, Ian Tickle wrote: All, I thought this would be a simple task, but for the life of me I can't see how to do it! All I want to do is convert an MTZ file to Shel-X format for refinement. I thought it would take me 2 secs, but it's taken me at least 5 attempts, and it's still not right! Do the conversion on the shelx side, rather than the CCP4 side. http://shelx.uni-ac.gwdg.de/~tg/mtz2sca/mtz2sca.html . B. Martin Hallberg, PhD Molecular Cell Biology Program Department of Cell and Molecular Biology Karolinska Institute Nobels väg 3 SE-171 77 Stockholm
Re: [ccp4bb] MTZ to Shel-X?
[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] From: Ian Tickle [EMAIL PROTECTED] To: [EMAIL PROTECTED], [EMAIL PROTECTED] Cc: CCP4BB@JISCMAIL.AC.UK Return-Path: [EMAIL PROTECTED] X-OriginalArrivalTime: 05 Mar 2007 13:15:51.0648 (UTC) FILETIME=[65E97E00:01C75F28] Martyn, sorry yes you're quite right, I should have submitted a proper bug report, consider this to be it. What's needed is a way to read an MTZ file with h, k, l, I, SIGI, FREE and to write same in _strict_ Shel-X format: - There must be NO header info written to the file (i.e. first line is first refln), and NO additional text (e.g. 'FREE') on the lines (same applies to all options that write Shel-X format output). - I would suggest multiplying I SIGI by 10 (or user-supplied scale factor) and writing out as nearest integers; this will reduce the chance of overflowing the format (leaving out the dec. pt. gives you one more column to play with which might make all the difference!). All other problems with mtz2various I think lie with the documentation: - The program uses labels I SIGI not IP SIGIP as stated (standard output also needs to be changed). - The program states that if I SIGI is input then the same is output; this is what is desired but not what currently happens (F's are written). It's possible of course that the option to write F's is still needed, in which case there needs to be a way of specifying this. - The user should be very strongly discouraged from re-squaring the F's from Truncate (possibly even by removing the FSQUARED option completely: but at present this is the only way of getting the desired output). - The bit about having the FREE text there to allow extraction of test set reflns, should be changed to use 'grep -e -1$ file' to extract - Possibly additional suggestions from others. Cheers -- Ian -Original Message- From: CCP4 bulletin board [mailto:[EMAIL PROTECTED] On Behalf Of Martyn Winn Sent: 05 March 2007 12:17 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] MTZ to Shel-X? A glance through the CVS history of mtz2various and f2mtz shows that there has been a lot of work keeping these up-to-date for various formats, work that is largely unrewarded and unacknowledged. But they are indeed still deficient in places. The required code writing is relatively trivial. The hard-bit is sorting through the available snippets of information, separating fact from fiction, dealing with different versions of software (backwards compatibility), dealing with different scenarios (I present or not, anomalous present or not, Rfree present or not, etc), and dealing with certain working practices (see e.g. Eleanor's email). When I get something working for myself, I'm usually less than halfway to something that is robust enough for general use. The way forward is to submit proper bug reports to [EMAIL PROTECTED] saying I want to convert from/to version X of software Y. I expect this script to use/produce this reflection file. If column Z is missing, then this happens. This header is optional, this one isn't. Etc, etc. I suggest this as a practical realistic way forward. Of course, one can use ccp4bb to debate what would happen in an ideal world. We (the ones that actually have the power to do so) don't throw people off the BB. Of course, patching mtz2various and f2mtz is not the best thing to do. Someone should write a modern import/export utility. Yes, we should discuss this at the March meeting. Regards Martyn On Mon, 2007-03-05 at 10:53 +, Kevin Cowtan wrote: You are absolutely right! The difficulty in getting from MTZ to any other format or back is unacceptable. Expecting working crystallographers to write Fortran format statements is ridiculous. I've been trying to address this by adding support for other formats to clipper, but my pace has been glacial, owing to the limited academic rewards for such work. Even then, it needs someone to put together an application and GUI to use it with. I will try and remember to raise this issue at the CCP4 developers meeting this year. If you prod me in April and I'll report back to the BB. Kevin George M. Sheldrick wrote: Dear Ian, It is all part of a diabolical CCP4 plot to make it as inconvenient as possible to move from a REFMAC refinement to SHELXL! I hope that i do not get excommunicated like DVD for this comment. To summarize your and Martin's suggestions, I know of only two ways to move from mtz to hkl: = 1] run mtz2various with the keywords: LABIN FP=FP SIGFP=SIGFP FREE=FreeR_flag OUTPUT SHELX FSQUAR then edit the resulting .hkl file with any text editor to remove the header
Re: [ccp4bb] MTZ to Shel-X?
I would like to second Ian's bug report, and suggest one minor improvement. Rather than multiplying I and sigI by 10, one could find the largest intensity value I(max) and multiply all the I and sigI values by (say) .99/I(max) to avoid any possibility of overflowing the format. An additional refinement, useful for datasets extending to very high resolution that can have a high dynamic range, would be to change the format from F8.2 to F8.3 or F8.4 when writing out small intensities. XPREP and other programs do all this. To avoid any confusion, the SHELX HKLF4 format (last changed when Axel introduced the free R in 1993) is: h k l I sigI nf '(3I4,2F8.2,I4)' (but Fortran can read any F8.X instead of F8.2) where the free R flag nf is 1 for the working set and -1 for the free R set. The file may be terminated either by a reflection with indices 0 0 0 or by the end of file. The data may be in any order, merged or unmerged, and systematic absences if present are removed as required by all the SHELX programs. The scale of the data is arbitrary. George Ian J. Tickle wrote: [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] From: Ian Tickle [EMAIL PROTECTED] To: [EMAIL PROTECTED], [EMAIL PROTECTED] Cc: CCP4BB@JISCMAIL.AC.UK Return-Path: [EMAIL PROTECTED] X-OriginalArrivalTime: 05 Mar 2007 13:15:51.0648 (UTC) FILETIME=[65E97E00:01C75F28] Martyn, sorry yes you're quite right, I should have submitted a proper bug report, consider this to be it. What's needed is a way to read an MTZ file with h, k, l, I, SIGI, FREE and to write same in _strict_ Shel-X format: - There must be NO header info written to the file (i.e. first line is first refln), and NO additional text (e.g. 'FREE') on the lines (same applies to all options that write Shel-X format output). - I would suggest multiplying I SIGI by 10 (or user-supplied scale factor) and writing out as nearest integers; this will reduce the chance of overflowing the format (leaving out the dec. pt. gives you one more column to play with which might make all the difference!). All other problems with mtz2various I think lie with the documentation: - The program uses labels I SIGI not IP SIGIP as stated (standard output also needs to be changed). - The program states that if I SIGI is input then the same is output; this is what is desired but not what currently happens (F's are written). It's possible of course that the option to write F's is still needed, in which case there needs to be a way of specifying this. - The user should be very strongly discouraged from re-squaring the F's from Truncate (possibly even by removing the FSQUARED option completely: but at present this is the only way of getting the desired output). - The bit about having the FREE text there to allow extraction of test set reflns, should be changed to use 'grep -e -1$ file' to extract - Possibly additional suggestions from others. Cheers -- Ian -Original Message- From: CCP4 bulletin board [mailto:[EMAIL PROTECTED] On Behalf Of Martyn Winn Sent: 05 March 2007 12:17 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] MTZ to Shel-X? A glance through the CVS history of mtz2various and f2mtz shows that there has been a lot of work keeping these up-to-date for various formats, work that is largely unrewarded and unacknowledged. But they are indeed still deficient in places. The required code writing is relatively trivial. The hard-bit is sorting through the available snippets of information, separating fact from fiction, dealing with different versions of software (backwards compatibility), dealing with different scenarios (I present or not, anomalous present or not, Rfree present or not, etc), and dealing with certain working practices (see e.g. Eleanor's email). When I get something working for myself, I'm usually less than halfway to something that is robust enough for general use. The way forward is to submit proper bug reports to [EMAIL PROTECTED] saying I want to convert from/to version X of software Y. I expect this script to use/produce this reflection file. If column Z is missing, then this happens. This header is optional, this one isn't. Etc, etc. I suggest this as a practical realistic way forward. Of course, one can use ccp4bb to debate what would happen in an ideal world. We (the ones that actually have the power to do so) don't throw people off the BB. Of course, patching mtz2various and f2mtz is not the best thing to do. Someone should write a modern import/export utility. Yes, we should discuss this at the March meeting. Regards Martyn On Mon, 2007-03-05 at 10:53 +, Kevin Cowtan wrote: You are absolutely right! The difficulty in getting from MTZ to any other format or back is unacceptable. Expecting working crystallographers to write Fortran format statements is ridiculous. I've
Re: [ccp4bb] MTZ to Shel-X?
That is done I believe in mtz2various.. Eleanor George M. Sheldrick wrote: I would like to second Ian's bug report, and suggest one minor improvement. Rather than multiplying I and sigI by 10, one could find the largest intensity value I(max) and multiply all the I and sigI values by (say) .99/I(max) to avoid any possibility of overflowing the format. An additional refinement, useful for datasets extending to very high resolution that can have a high dynamic range, would be to change the format from F8.2 to F8.3 or F8.4 when writing out small intensities. XPREP and other programs do all this. To avoid any confusion, the SHELX HKLF4 format (last changed when Axel introduced the free R in 1993) is: h k l I sigI nf '(3I4,2F8.2,I4)' (but Fortran can read any F8.X instead of F8.2) where the free R flag nf is 1 for the working set and -1 for the free R set. The file may be terminated either by a reflection with indices 0 0 0 or by the end of file. The data may be in any order, merged or unmerged, and systematic absences if present are removed as required by all the SHELX programs. The scale of the data is arbitrary. George Ian J. Tickle wrote: [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] From: Ian Tickle [EMAIL PROTECTED] To: [EMAIL PROTECTED], [EMAIL PROTECTED] Cc: CCP4BB@JISCMAIL.AC.UK Return-Path: [EMAIL PROTECTED] X-OriginalArrivalTime: 05 Mar 2007 13:15:51.0648 (UTC) FILETIME=[65E97E00:01C75F28] Martyn, sorry yes you're quite right, I should have submitted a proper bug report, consider this to be it. What's needed is a way to read an MTZ file with h, k, l, I, SIGI, FREE and to write same in _strict_ Shel-X format: - There must be NO header info written to the file (i.e. first line is first refln), and NO additional text (e.g. 'FREE') on the lines (same applies to all options that write Shel-X format output). - I would suggest multiplying I SIGI by 10 (or user-supplied scale factor) and writing out as nearest integers; this will reduce the chance of overflowing the format (leaving out the dec. pt. gives you one more column to play with which might make all the difference!). All other problems with mtz2various I think lie with the documentation: - The program uses labels I SIGI not IP SIGIP as stated (standard output also needs to be changed). - The program states that if I SIGI is input then the same is output; this is what is desired but not what currently happens (F's are written). It's possible of course that the option to write F's is still needed, in which case there needs to be a way of specifying this. - The user should be very strongly discouraged from re-squaring the F's from Truncate (possibly even by removing the FSQUARED option completely: but at present this is the only way of getting the desired output). - The bit about having the FREE text there to allow extraction of test set reflns, should be changed to use 'grep -e -1$ file' to extract - Possibly additional suggestions from others. Cheers -- Ian -Original Message- From: CCP4 bulletin board [mailto:[EMAIL PROTECTED] On Behalf Of Martyn Winn Sent: 05 March 2007 12:17 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] MTZ to Shel-X? A glance through the CVS history of mtz2various and f2mtz shows that there has been a lot of work keeping these up-to-date for various formats, work that is largely unrewarded and unacknowledged. But they are indeed still deficient in places. The required code writing is relatively trivial. The hard-bit is sorting through the available snippets of information, separating fact from fiction, dealing with different versions of software (backwards compatibility), dealing with different scenarios (I present or not, anomalous present or not, Rfree present or not, etc), and dealing with certain working practices (see e.g. Eleanor's email). When I get something working for myself, I'm usually less than halfway to something that is robust enough for general use. The way forward is to submit proper bug reports to [EMAIL PROTECTED] saying I want to convert from/to version X of software Y. I expect this script to use/produce this reflection file. If column Z is missing, then this happens. This header is optional, this one isn't. Etc, etc. I suggest this as a practical realistic way forward. Of course, one can use ccp4bb to debate what would happen in an ideal world. We (the ones that actually have the power to do so) don't throw people off the BB. Of course, patching mtz2various and f2mtz is not the best thing to do. Someone should write a modern import/export utility. Yes, we should discuss this at the March meeting. Regards Martyn On Mon, 2007-03-05 at 10:53 +, Kevin Cowtan wrote: You are absolutely right! The difficulty in getting from MTZ to any other format or back is unacceptable. Expecting
Re: [ccp4bb] MTZ to Shel-X?
Yes, there is: C input FP(+) IF(LOOKUP(23).GT.0 .AND. IFSQ.NE.0) + SCCHK = 9.00/(RANGES(2,LOOKUP(23))* RANGES(2,LOOKUP(23))) C IF (SCAL.GE.SCCHK .AND.IFSQ.NE.0) SCAL = SCCHK IF(IFSQ.NE.0) WRITE(6,'(/,a,/,a,F8.4,/)') + ' *** You are inputting Fs and requesting Is to be output.', + ' *** The Fs will be squared and scaled by:',SCAL which works for me. That has been there for years. The caveat is that it is only applied for the FSQUARED option. I believe Ian and George want it for the I, SIGI route too. I've taken out the header and FREE text. This will apply to all SHELX output from mtz2various, whether for SHELXD, SHELXL, 3rd party apps. If anyone uses this info they'd better let me know asap. I also notice that the mtz2various output starts: Data line--- LABIN FP=FP SIGFP=SIGFP FREE=FreeR_flag Data line--- FSQUARED Fs are squared on output - better to use TRUNCATE output Data line--- OUTPUT SHELX Data line--- END but I guess no-one reads log files these days ... (in case that's not clear, TRUNCATE outputs the Is as well as Fs) m On Mon, 2007-03-05 at 16:01 +, Eleanor Dodson wrote: That is done I believe in mtz2various.. Eleanor George M. Sheldrick wrote: I would like to second Ian's bug report, and suggest one minor improvement. Rather than multiplying I and sigI by 10, one could find the largest intensity value I(max) and multiply all the I and sigI values by (say) .99/I(max) to avoid any possibility of overflowing the format. An additional refinement, useful for datasets extending to very high resolution that can have a high dynamic range, would be to change the format from F8.2 to F8.3 or F8.4 when writing out small intensities. XPREP and other programs do all this. To avoid any confusion, the SHELX HKLF4 format (last changed when Axel introduced the free R in 1993) is: h k l I sigI nf '(3I4,2F8.2,I4)' (but Fortran can read any F8.X instead of F8.2) where the free R flag nf is 1 for the working set and -1 for the free R set. The file may be terminated either by a reflection with indices 0 0 0 or by the end of file. The data may be in any order, merged or unmerged, and systematic absences if present are removed as required by all the SHELX programs. The scale of the data is arbitrary. George Ian J. Tickle wrote: [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] From: Ian Tickle [EMAIL PROTECTED] To: [EMAIL PROTECTED], [EMAIL PROTECTED] Cc: CCP4BB@JISCMAIL.AC.UK Return-Path: [EMAIL PROTECTED] X-OriginalArrivalTime: 05 Mar 2007 13:15:51.0648 (UTC) FILETIME=[65E97E00:01C75F28] Martyn, sorry yes you're quite right, I should have submitted a proper bug report, consider this to be it. What's needed is a way to read an MTZ file with h, k, l, I, SIGI, FREE and to write same in _strict_ Shel-X format: - There must be NO header info written to the file (i.e. first line is first refln), and NO additional text (e.g. 'FREE') on the lines (same applies to all options that write Shel-X format output). - I would suggest multiplying I SIGI by 10 (or user-supplied scale factor) and writing out as nearest integers; this will reduce the chance of overflowing the format (leaving out the dec. pt. gives you one more column to play with which might make all the difference!). All other problems with mtz2various I think lie with the documentation: - The program uses labels I SIGI not IP SIGIP as stated (standard output also needs to be changed). - The program states that if I SIGI is input then the same is output; this is what is desired but not what currently happens (F's are written). It's possible of course that the option to write F's is still needed, in which case there needs to be a way of specifying this. - The user should be very strongly discouraged from re-squaring the F's from Truncate (possibly even by removing the FSQUARED option completely: but at present this is the only way of getting the desired output). - The bit about having the FREE text there to allow extraction of test set reflns, should be changed to use 'grep -e -1$ file' to extract - Possibly additional suggestions from others. Cheers -- Ian -Original Message- From: CCP4 bulletin board [mailto:[EMAIL PROTECTED] On Behalf Of Martyn Winn Sent: 05 March 2007 12:17 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] MTZ to Shel-X? A glance through the CVS history of mtz2various and f2mtz shows that there has been a lot of work keeping these up-to-date for various formats, work that is largely unrewarded and unacknowledged. But they are indeed still deficient in places. The required code writing is relatively trivial. The hard-bit
Re: [ccp4bb] MTZ to Shel-X?
Martyn Winn wrote: Yes, there is: C input FP(+) IF(LOOKUP(23).GT.0 .AND. IFSQ.NE.0) + SCCHK = 9.00/(RANGES(2,LOOKUP(23))* RANGES(2,LOOKUP(23))) C IF (SCAL.GE.SCCHK .AND.IFSQ.NE.0) SCAL = SCCHK IF(IFSQ.NE.0) WRITE(6,'(/,a,/,a,F8.4,/)') + ' *** You are inputting Fs and requesting Is to be output.', + ' *** The Fs will be squared and scaled by:',SCAL which works for me. That has been there for years. The caveat is that it is only applied for the FSQUARED option. I believe Ian and George want it for the I, SIGI route too. Precisely! Now this needs to be applied to output I and sigI directly without going via F and sigF I and sigI output in this way (plus the free R flag) should be the default for the 'SHELX' keyword, it has after all been the standard SHELX format since 1993 (and without the free R flag, since 1969). I've taken out the header and FREE text. This will apply to all SHELX output from mtz2various, whether for SHELXD, SHELXL, 3rd party apps. If anyone uses this info they'd better let me know asap. Good. With the junk removed all SHELX programs (and all 3rd party programs that are SHELX-compatible) should work fine. Maybe CCP4i can make sure that the intensities don't get lost from the mtz file somewhere on the way. George
Re: [ccp4bb] MTZ to Shel-X?
On Mon, Mar 05, 2007 at 10:53:55AM +, Kevin Cowtan wrote: You are absolutely right! The difficulty in getting from MTZ to any other format or back is unacceptable. Expecting working crystallographers to write Fortran format statements is ridiculous. I've been trying to address this by adding support for other formats to clipper, but my pace has been glacial, owing to the limited academic rewards for such work. Even then, it needs someone to put together an application and GUI to use it with. I posit that if the CCP4 format was open (i.e. the CCP4C/F libraries were covered by an OSI/FSF approved license) and deposited in a reasonable code repository, then there exists the potential assistance of the entire public domain to standardize/extend/improve the library functions and ABIs (with, of course, oversight by the CCP4 folks). I know I would be far more inclined to help if such was the case. Regards, Tim
Re: [ccp4bb] MTZ to Shel-X?
On Thursday 01 March 2007 10:17, Ian Tickle wrote: All, I thought this would be a simple task, but for the life of me I can't see how to do it! All I want to do is convert an MTZ file to Shel-X format for refinement. I thought it would take me 2 secs, but it's taken me at least 5 attempts, and it's still not right! Do the conversion on the shelx side, rather than the CCP4 side. http://shelx.uni-ac.gwdg.de/~tg/mtz2sca/mtz2sca.html First I tried mtz2various with this command line: echo 'labi FP=F SIGFP=SIGF FREE=FreeR_flag\noutp shelx' | mtz2various hklin 16294 hklout 16294.hkl with 'HKLF 3' in my .ins file. This works fine, but then shelxl complains with: -48 0 27 37.49 19.74 -1 ?? ** REFLECTION 30 HAS WRONG FORMAT ** OK fair enough I should have read the Shel-X manual first, because it appears that 'HKLF 3' doesn't accept free R flags, it has to be 'HKLF 4' (i.e. I's not F's). So when all else fails read the mtz2various documentation (at least read it more thoroughly): * If only FP SIGFP or IP SIGIP are assigned on LABIN, then h k l FP SIGFP or h k l IP SIGIP is output. Great, precisely what I want! So for my second try let's go with: echo 'labi IP=IMEAN SIGIP=SIGIMEAN FREE=FreeR_flag\noutp shelx' | mtz2various hklin 16294 hklout 16294.hkl but no: Data line--- labi IP=IMEAN SIGIP=SIGIMEAN FREE=FreeR_flag MtzParseLabin: neither label recognised: IP IMEAN MTZ2VARIOUS: Error in label assignments in LABIN Yet the output also says: LABIN program label=file label ... Valid program labels are: FP, SIGFP and optionally FPH, SIGFPH, IP, SIGIP, FC, PHIC, W, FPART, PHIPART, PHIB, FOM, HLA HLB HLC HLD FREE My first thought was I've got the IMEAN/SIGIMEAN label wrong, but no, mtzdump says: Col SortMinMaxNum % Mean Mean Resolution Type Column num order Missing complete abs. LowHigh label 1 ASC-49 43 0 100.00 -8.7 18.6 48.75 1.33 H H 2 NONE 0 32 0 100.00 12.0 12.0 48.75 1.33 H K 3 NONE 0 41 0 100.00 15.7 15.7 48.75 1.33 H L 4 NONE4.2 854.8 314 99.0182.2682.26 34.92 1.33 F F 5 NONE0.7 241.2 314 99.01 3.38 3.38 34.92 1.33 Q SIGF 6 NONE -58.6 8600.6 314 99.01 146.23 146.25 34.92 1.33 J IMEAN 7 NONE0.5 5370.6 314 99.01 5.88 5.88 34.92 1.33 Q SIGIMEAN 8 NONE0.019.0 0 100.00 9.48 9.48 48.75 1.33 I FreeR_flag Then I look further down in mtz2various.doc: Input labels accepted are: H, K, L Indices FP, SIGFP F and Sigma for native FPH, SIGFPH F and Sigma for derivative FC, PHIC F and Phase from model FPART, PHIPART F and Phase from partial structure DP, SIGDP Anomalous difference and Sigma I, SIGI I and Sigma So OK, fair enough IP, SIGIP was a typo (happens all the time in CCP4 documentation!), so problem solved, for my third go let's try: echo 'labi I=IMEAN SIGI=SIGIMEAN FREE=FreeR_flag\noutp shelx' | mtz2various hklin 16294 hklout 16294.hkl Yes it worked!!! or did it??? I see that the output file has HKLF 3 in the header, and when I read the log file I see: NB Amplitude Output , flagged with HKLF 3 ***. No good, Shel-X won't take free R flags with HKLF 3 - and I thought mtz2various.doc said that if I/SIGI are assigned then I/SIGI are output? (which would make a lot of sense!). Then I read in mtz2various.doc: FSQUARED If this flag is set, the program expects F and SIGF and will output I and SIGI: I = F*F, SIGI = 2*SIGF*F + SIGF*SIGF. These intensities are not necessarily the same as the measured intensities (pre-TRUNCATE); it is better to use the measured values if you have them. I completely agree with the last sentence! - but how to make it read I's and write I's? So in desperation to get something working, my fourth go is with: echo 'labi FP=F SIGFP=SIGF FREE=FreeR_flag\nfsqu\noutp shelx' | mtz2various hklin 16294 hklout 16294.hkl This works: a file in HKLF 4 format containing apparently I SIGI is produced, but again Shel-X complains: -48 0 27 37.49 19.74 -1 ?? ** REFLECTION 30 HAS WRONG FORMAT ** Aaaargh - back to step 1! So next I seek advice from a Shel-X user who says they've never seen the FREE text at the end of the test reflection lines, e.g.: -48 0 27 114.07 151.73 -1 FREE and I say something along the lines of don't be silly there's no way the program would write it out like that if Shel-X didn't accept it - and just to prove to them how dumb their suggestion is I do a global edit of the file, and of course it turns out they're quite right, and technically it works - except of course that I is
Re: [ccp4bb] MTZ to Shel-X?
Yes I have always been annoyed by that. I tend to use sftools to trim and then rename the columns I want followed by mtz2various with something along the lines of labin FP=F SIGFP=SIGF FREE=FreeR and then finally edit the resulting text file to remove the header and all the FREE's as you did. It's always annoyed me that you have to do these extra steps. I have a conspiracy theory along the lines that no matter how user friendly GUI's/crystallography programs may become, veteran crystallographers will always make sure there are a couple little twists just to show who's boss! Simon -Original Message- From: CCP4 bulletin board [mailto:[EMAIL PROTECTED] On Behalf Of Ian Tickle Sent: 01 March 2007 18:18 To: CCP4BB@JISCMAIL.AC.UK Subject: [ccp4bb] MTZ to Shel-X? All, I thought this would be a simple task, but for the life of me I can't see how to do it! All I want to do is convert an MTZ file to Shel-X format for refinement. I thought it would take me 2 secs, but it's taken me at least 5 attempts, and it's still not right! First I tried mtz2various with this command line: echo 'labi FP=F SIGFP=SIGF FREE=FreeR_flag\noutp shelx' | mtz2various hklin 16294 hklout 16294.hkl with 'HKLF 3' in my .ins file. This works fine, but then shelxl complains with: -48 0 27 37.49 19.74 -1 ?? ** REFLECTION 30 HAS WRONG FORMAT ** OK fair enough I should have read the Shel-X manual first, because it appears that 'HKLF 3' doesn't accept free R flags, it has to be 'HKLF 4' (i.e. I's not F's). So when all else fails read the mtz2various documentation (at least read it more thoroughly): * If only FP SIGFP or IP SIGIP are assigned on LABIN, then h k l FP SIGFP or h k l IP SIGIP is output. Great, precisely what I want! So for my second try let's go with: echo 'labi IP=IMEAN SIGIP=SIGIMEAN FREE=FreeR_flag\noutp shelx' | mtz2various hklin 16294 hklout 16294.hkl but no: Data line--- labi IP=IMEAN SIGIP=SIGIMEAN FREE=FreeR_flag MtzParseLabin: neither label recognised: IP IMEAN MTZ2VARIOUS: Error in label assignments in LABIN Yet the output also says: LABIN program label=file label ... Valid program labels are: FP, SIGFP and optionally FPH, SIGFPH, IP, SIGIP, FC, PHIC, W, FPART, PHIPART, PHIB, FOM, HLA HLB HLC HLD FREE My first thought was I've got the IMEAN/SIGIMEAN label wrong, but no, mtzdump says: Col SortMinMaxNum % Mean Mean Resolution Type Column num order Missing complete abs. LowHigh label 1 ASC-49 43 0 100.00 -8.7 18.6 48.75 1.33 H H 2 NONE 0 32 0 100.00 12.0 12.0 48.75 1.33 H K 3 NONE 0 41 0 100.00 15.7 15.7 48.75 1.33 H L 4 NONE4.2 854.8 314 99.0182.2682.26 34.92 1.33 F F 5 NONE0.7 241.2 314 99.01 3.38 3.38 34.92 1.33 Q SIGF 6 NONE -58.6 8600.6 314 99.01 146.23 146.25 34.92 1.33 J IMEAN 7 NONE0.5 5370.6 314 99.01 5.88 5.88 34.92 1.33 Q SIGIMEAN 8 NONE0.019.0 0 100.00 9.48 9.48 48.75 1.33 I FreeR_flag Then I look further down in mtz2various.doc: Input labels accepted are: H, K, L Indices FP, SIGFP F and Sigma for native FPH, SIGFPH F and Sigma for derivative FC, PHIC F and Phase from model FPART, PHIPART F and Phase from partial structure DP, SIGDP Anomalous difference and Sigma I, SIGI I and Sigma So OK, fair enough IP, SIGIP was a typo (happens all the time in CCP4 documentation!), so problem solved, for my third go let's try: echo 'labi I=IMEAN SIGI=SIGIMEAN FREE=FreeR_flag\noutp shelx' | mtz2various hklin 16294 hklout 16294.hkl Yes it worked!!! or did it??? I see that the output file has HKLF 3 in the header, and when I read the log file I see: NB Amplitude Output , flagged with HKLF 3 ***. No good, Shel-X won't take free R flags with HKLF 3 - and I thought mtz2various.doc said that if I/SIGI are assigned then I/SIGI are output? (which would make a lot of sense!). Then I read in mtz2various.doc: FSQUARED If this flag is set, the program expects F and SIGF and will output I and SIGI: I = F*F, SIGI = 2*SIGF*F + SIGF*SIGF. These intensities are not necessarily the same as the measured intensities (pre-TRUNCATE); it is better to use the measured values if you have them. I completely agree with the last sentence! - but how to make it read I's and write I's? So in desperation to get something working, my fourth go is with: echo 'labi FP=F SIGFP=SIGF FREE=FreeR_flag\nfsqu\noutp shelx' | mtz2various hklin 16294 hklout 16294.hkl This works: a file in HKLF 4 format containing apparently I SIGI is produced, but again Shel-X complains: -48 0 27 37.49 19.74 -1 ?? ** REFLECTION 30 HAS WRONG FORMAT ** Aaaargh - back to step 1! So next I seek advice from a
