Dear Keitaro, I have come to the exact same conclusions, and the next version of XDS will revert to the old refinement behavior.
As a preview, I've uploaded a fixed Linux binary (not the whole package, just xds_par) to ftp://turn5.biologie.uni-konstanz.de/pub/xds/xds_par.refi_cell_position_together.bz2 . Thanks - also to Oliver for reporting the problem! Kay On Fri, 26 Jan 2018 16:56:30 +0900, Keitaro Yamashita <keitaro.yamash...@riken.jp> wrote: >Dear Kay, > >I also tested this using a publicly available data of thaumatin >https://zenodo.org/record/10271 >(resolution ~1.4 Å, wavelength= 0.97625 Å). >The camera distance from header is 265.27 mm. I tested this original >distance and some shifts (+1, +2, +4, ..., +32 mm) with different >versions of XDS. > >As a result, IDXREF of the old version (170615) successfully refined >camera distance to ~265.0 mm if shift was less than or equal to +16 >mm. The statistics in CORRECT.LP was as good as that without shift. > >If the camera distance was fixed in IDXREF, statistics was >compromised, but the refinement in CORRECT worked well and recycling >of GXPARM.XDS improved the data quality. > >However, the new version (170720 or later) changed camera distance >very little. Even with +1 mm shift the statistics was deteriorated, >and CORRECT (GXPARM.XDS) also didn't change the distance a lot, which >means the recycling of optimized parameters do not help. The new >version seems to stick to the given camera distance too much as >pointed out by many people. > >I agree that camera distance in the header should be accurate. >However, I also wish to revert to the former behavior of XDS that >expires very soon. > >Actually in previous versions (170615 or former) fixing POSITION in >REFINE(IDXREF) was recommended, but the recycling could give an >accurate distance. The current version seems to virtually have no >option to refine the distance. > >Best regards, >Keitaro > >On 24 January 2018 at 05:16, Kay Diederichs ><kay.diederi...@uni-konstanz.de> wrote: >> Dear Oliver, >> >> sorry for the trouble! >> A default should be correct in the majority of situations, but it's >> impossible to have it work for _all_ situations. The XDS default for >> REFINE(IDXREF) was changed (i.e. POSITION was removed) to improve the >> indexing for weak and lousy data, _and_ because the distance values from the >> header are nowadays almost always accurate. For data with very high >> resolution such as yours, and significantly wrong distance, this means that >> at high resolution in particular this default will lead to worse data. >> generate_XDS.INP (in XDSwiki) and (I think) autoPROC and xia2 explicitly >> include POSITION in the REFINE(IDXREF), i.e. override the default. Where did >> you get XDS.INP from? >> Some comments: >> a) this is why I recommend, for data sets that may be important, to always >> do one cycle of optimization, i.e. after CORRECT, "mv GXPARM.XDS XPARM.XDS", >> change XDS.INP to have the correct high-resolution cutoff (cutting after the >> last shell which still has a "*" in the CC1/2 column), adjust the >> FRIEDEL'S_LAW= setting, and run JOB=INTEGRATE CORRECT. In your case this >> would make the refined distance available to INTEGRATE, and should result in >> very good data. >> b) at which beamline did you collect the data? Such a high difference >> between refined distance and distance from header is unusual, and should be >> reported to (and fixed by) the beamline staff. >> c) for this beamline at least, one should use REFINE(IDXREF)=CELL BEAM >> ORIENTATION AXIS POSITION . I understand that you tried this and it didn't >> work? Maybe there is something else wrong then, e.g. another line later in >> XDS.INP resetting REFINE(IDXREF) to something else. >> >> If you cannot find the reason why including POSITiON into REFINE(IDXREF) >> does not help, pls send me enough data to reproduce the problem. >> >> best wishes, >> >> Kay >> >> On Tue, 23 Jan 2018 14:31:09 +0000, "Weiergräber, Oliver H." >> <o.h.weiergrae...@fz-juelich.de> wrote: >> >>>Dear all, >>> >>>After upgrading XDS from build date 20170601 to 20171218, I am experiencing >>>severe degradation of apparent data quality reported by CORRECT for certain >>>data sets. Following first indications of issues with a slightly problematic >>>candidate, I went back to a previously very well-behaved test case with >>>diffraction extending beyond 1.1 A. Using the same input with both program >>>versions, statistics are not too different out to approx. 1.6 A, but become >>>more and more divergent in outer shells. These are the values for the >>>highest-resolution shell (1.10--1.16 A): >>> >>>20170601: I/sigI = 1.80; Rmeas = 59.9 %; CC(1/2) = 82.0 % >>>20171218: I/sigI = 0.14; Rmeas = 506.4 %; CC(1/2) = 5.8 % >>> >>>The refined cell constants are unreasonably different as well. Obviously, >>>something is going awfully wrong here, presumably related to errors in >>>geometry parameters (which are expected to become increasingly detrimental >>>with decreasing dmin). As it turns out, geometry refinement behaves >>>differently in the latest version of XDS: most notably, IDXREF no longer >>>refines the detector distance by default. This is significant in this case, >>>as in version 20170601 the distance would shift by as much as 1.3 mm, >>>resulting in successful integration and scaling. Unfortunately, re-adding >>>POSITION refinement into REFINE(IDXREF) does not help, and even having it in >>>all refinement scopes (IDXREF, INTEGRATE, CORRECT) only yields a limited >>>improvement of CORRECT statistics. >>>It is worth noting that examination of a dataset from an unrelated crystal >>>(dmin = 1.4 A) did not reveal such enormous differences -- in this case, >>>however, the shift in detector distance applied in the old version of XDS >>>was quite small (0.2 mm), which, together with the lower resolution, >>>explains the absence of large discrepancies. >>> >>>To sum up, I suspect that, due to recent changes to the XDS code, the >>>refinement of geometry parameters is now overly restrained, resulting in >>>drastic problems in cases where the detector distance is not as precise as >>>desirable (and even more so at very high resolution). For such datasets, the >>>new version appears to be essentially unusable (at least within the >>>parameter space I have tested), and even in modest cases may often be >>>inferior to the previous one. Is there an option to revert to the former >>>behaviour? >>> >>>Best regards >>>Oliver >>> >>>================================================ >>> PD Dr. Oliver H. Weiergr�ber >>> Institute of Complex Systems >>> ICS-6: Structural Biochemistry >>> Tel.: +49 2461 61-2028 >>> Fax: +49 2461 61-9540 >>>================================================ >>> >>> >>> >>> >>>------------------------------------------------------------------------------------------------ >>>------------------------------------------------------------------------------------------------ >>>Forschungszentrum Juelich GmbH >>>52425 Juelich >>>Sitz der Gesellschaft: Juelich >>>Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher >>>Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), >>>Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, >>>Prof. Dr. Sebastian M. Schmidt >>>------------------------------------------------------------------------------------------------ >>>------------------------------------------------------------------------------------------------
