Re: [ccp4bb] intensity statistics and twinning

[Tim Gruene]

Dear Hauke,

others pointed out that there is little suspicious about your data, and you 
can probably continue refining in P21. You also have a reasonably large unit 
cell, so stick to the defaults regarding Rfree selection.

4.5A resolution is just quite hard to work with, but otherwise I don't think 
there is anything problematic about your data.

Best regards,
Tim

On Monday, October 31, 2016 09:44:35 AM Hillen, Hauke wrote:
> Dear Tim,
> 
> In this case, I was using my anomalous dataset truncated at 5A resolution
> for refinement and used Phenix to generate a Free R array:
> 
> Number of work/free reflections by resolution:
>  work  free  %free
>   bin  1: 49.6881 - 10.7634 [4605/4610]  4403   202   4.4%
>   bin  2: 10.7634 -  8.5575 [4565/4568]  4365   200   4.4%
>   bin  3:  8.5575 -  7.4799 [4588/4591]  4389   199   4.3%
>   bin  4:  7.4799 -  6.7979 [4592/4597]  4394   198   4.3%
>   bin  5:  6.7979 -  6.3117 [4616/4619]  4415   201   4.4%
>   bin  6:  6.3117 -  5.9402 [4582/4587]  4386   196   4.3%
>   bin  7:  5.9402 -  5.6431 [4589/4594]  4388   201   4.4%
>   bin  8:  5.6431 -  5.3978 [4567/4570]  4367   200   4.4%
>   bin  9:  5.3978 -  5.1902 [4572/4577]  4372   200   4.4%
>   bin 10:  5.1902 -  5.0113 [4621/4627]  4420   201   4.3%
> overall 43899  1998   4.4%
> 
> I assume that if there is really twinning, one should choose rather more
> reflections than less for the free R array to have around 5% for each twin
> fraction?
> 
> Kind regards,
> Hauke
> 
> 
> ---
> Hauke Hillen
> Department of Molecular Biology
> Prof. Dr. Patrick Cramer
> Max-Planck-Institute for Biophysical Chemistry
> Am Fassberg 11
> D-37077 Göttingen
> Germany
> 
> E-Mail: hauke.hil...@mpibpc.mpg.de
> Phone: +49-551-201-2864
> 
> 
> 
> 
> 
> On Oct 28, 2016, at 4:03 PM, Tim Gruene
> > wrote:
> 
> Dear Hauke,
> 
> when you use twin refinement, your R-factor is bound to drop whether it is
> twinned or not. In your case Rfree rises, which is actually hinting at your
> data not being twinned. How many reflections do you have in the Rfree set?
> 
> Best,
> Tim
> 
> On Friday, October 28, 2016 1:59:29 PM CEST Hillen, Hauke wrote:
> Dear Kay,
> 
> Thank you for your answer!
> I have meanwhile processed my anomalous data also also as P1 and solved the
> structure with MR-SAD in P1, which works flawlessly and gives the same
> solution. Compared to P21, I now have 4 instead of 2 molecules in the ASU,
> which makes sense. I then tried refinement in P1 with or without applying a
> twin law in phenix.refine. The result seems very clear. Without twin law,
> R/Rfree after one round of rigid body refinement drop to 34/38. With twin
> refinement, the same refinement results in R/Rfree of 29/42. My current
> suspicion is therefore that my data are truly P1, with an NCS axis parallel
> to a crystallographic axis (suggested by Xtriage). Could this
> pseudosymmetry lead XDS to falsely identify my data as P2 instead of P1?
> 
> Best wishes,
> Hauke
> 
> 
> ---
> Hauke Hillen
> Department of Molecular Biology
> Prof. Dr. Patrick Cramer
> Max-Planck-Institute for Biophysical Chemistry
> Am Fassberg 11
> D-37077 Göttingen
> Germany
> 
> E-Mail:
> hauke.hil...@mpibpc.mpg.de hil...@mpibpc.mpg.de> Phone: +49-551-201-2864
> 
> 
> 
> 
> 
> On Oct 28, 2016, at 3:51 PM, Kay Diederichs
>  o:kay.diederi...@uni-konstanz.de>> wrote:
> 
> Dear Hauke,
> 
> the conversion of intensities to amplitudes is an area with unsatisfactory
> solutions, and the approaches of (and the assumptions made by) the
> different programs differ. Low-resolution data are often anisotropic, which
> adds difficulties. You could try CCP4 truncate/ctruncate and
> GlobalPhasing's staraniso. The latter in particular should treat
> anisotropic diffraction more properly. Probably you get yet other answers
> from xtriage when fed with those amplitudes. Possibly this will add to the
> confusion, but I'm not sure it is worth worrying about this too much.
> 
> Having said that, it seems to me like the primary and most unspoiled data
> are the intensities. I don't see anything suspicious in the
> xtriage-on-intensities output that you posted.
> 
> good luck with your difficult project!
> 
> Kay
> 
> 
> --
> --
> Paul Scherrer Institute
> Dr. Tim Gruene
> - persoenlich -
> OFLC/102
> CH-5232 Villigen PSI
> phone: +41 (0)56 310 5297
> 
> GPG Key ID = A46BEE1A
-- 
--
Paul Scherrer Institut
Dr. Tim Gruene
- persoenlich -
Principal Investigator
Biology and Chemistry
OFLC/102
CH-5232 Villigen PSI

Phone: +41 (0)56 310 5297

GPG Key ID = A46BEE1A



signature.asc
Description: This is a digitally signed message part.

Re: [ccp4bb] intensity statistics and twinning

[Hillen, Hauke]

Dear Tim,

In this case, I was using my anomalous dataset truncated at 5A resolution for 
refinement and used Phenix to generate a Free R array:

Number of work/free reflections by resolution:
 work  free  %free
  bin  1: 49.6881 - 10.7634 [4605/4610]  4403   202   4.4%
  bin  2: 10.7634 -  8.5575 [4565/4568]  4365   200   4.4%
  bin  3:  8.5575 -  7.4799 [4588/4591]  4389   199   4.3%
  bin  4:  7.4799 -  6.7979 [4592/4597]  4394   198   4.3%
  bin  5:  6.7979 -  6.3117 [4616/4619]  4415   201   4.4%
  bin  6:  6.3117 -  5.9402 [4582/4587]  4386   196   4.3%
  bin  7:  5.9402 -  5.6431 [4589/4594]  4388   201   4.4%
  bin  8:  5.6431 -  5.3978 [4567/4570]  4367   200   4.4%
  bin  9:  5.3978 -  5.1902 [4572/4577]  4372   200   4.4%
  bin 10:  5.1902 -  5.0113 [4621/4627]  4420   201   4.3%
overall 43899  1998   4.4%

I assume that if there is really twinning, one should choose rather more 
reflections than less for the free R array to have around 5% for each twin 
fraction?

Kind regards,
Hauke


---
Hauke Hillen
Department of Molecular Biology
Prof. Dr. Patrick Cramer
Max-Planck-Institute for Biophysical Chemistry
Am Fassberg 11
D-37077 Göttingen
Germany

E-Mail: hauke.hil...@mpibpc.mpg.de
Phone: +49-551-201-2864





On Oct 28, 2016, at 4:03 PM, Tim Gruene 
> wrote:

Dear Hauke,

when you use twin refinement, your R-factor is bound to drop whether it is
twinned or not. In your case Rfree rises, which is actually hinting at your
data not being twinned. How many reflections do you have in the Rfree set?

Best,
Tim

On Friday, October 28, 2016 1:59:29 PM CEST Hillen, Hauke wrote:
Dear Kay,

Thank you for your answer!
I have meanwhile processed my anomalous data also also as P1 and solved the
structure with MR-SAD in P1, which works flawlessly and gives the same
solution. Compared to P21, I now have 4 instead of 2 molecules in the ASU,
which makes sense. I then tried refinement in P1 with or without applying a
twin law in phenix.refine. The result seems very clear. Without twin law,
R/Rfree after one round of rigid body refinement drop to 34/38. With twin
refinement, the same refinement results in R/Rfree of 29/42. My current
suspicion is therefore that my data are truly P1, with an NCS axis parallel
to a crystallographic axis (suggested by Xtriage). Could this
pseudosymmetry lead XDS to falsely identify my data as P2 instead of P1?

Best wishes,
Hauke


---
Hauke Hillen
Department of Molecular Biology
Prof. Dr. Patrick Cramer
Max-Planck-Institute for Biophysical Chemistry
Am Fassberg 11
D-37077 Göttingen
Germany

E-Mail: 
hauke.hil...@mpibpc.mpg.de
Phone: +49-551-201-2864





On Oct 28, 2016, at 3:51 PM, Kay Diederichs
>
wrote:

Dear Hauke,

the conversion of intensities to amplitudes is an area with unsatisfactory
solutions, and the approaches of (and the assumptions made by) the
different programs differ. Low-resolution data are often anisotropic, which
adds difficulties. You could try CCP4 truncate/ctruncate and
GlobalPhasing's staraniso. The latter in particular should treat
anisotropic diffraction more properly. Probably you get yet other answers
from xtriage when fed with those amplitudes. Possibly this will add to the
confusion, but I'm not sure it is worth worrying about this too much.

Having said that, it seems to me like the primary and most unspoiled data
are the intensities. I don't see anything suspicious in the
xtriage-on-intensities output that you posted.

good luck with your difficult project!

Kay


--
--
Paul Scherrer Institute
Dr. Tim Gruene
- persoenlich -
OFLC/102
CH-5232 Villigen PSI
phone: +41 (0)56 310 5297

GPG Key ID = A46BEE1A

Re: [ccp4bb] intensity statistics and twinning

[Kay Diederichs]

Hi Hauke,

so your refinement results in P1 seem to say it's untwinned. I don't see (but I 
may be overlooking something?) the evidence, from what you write, that it could 
not be P21 or P2, because any P2 or P21 structure can be refined in P1 (but of 
course not the other way round). In other words, I'd expect that two molecules 
out of the four of your P1 solution can be refined in P2 or P21 to the same (or 
rather better) Rwork/Rfree. That would prove P2 or P21.

If there is really no 2-fold NCS solution in P2/P21 then it is P1 with 4-fold 
NCS.

Concerning your last question whether XDS or pointless could have mistaken a 
two-fold NCS along a crystallographic axis as a crystallographic symmetry 
element, I'd need to see the pointless table after "Analysing rotational 
symmetry in lattice group ...", and the relevant table in CORRECT.LP where it 
tests some spacegroups.

best,

Kay

Re: [ccp4bb] intensity statistics and twinning

[Tim Gruene]

Dear Hauke,

when you use twin refinement, your R-factor is bound to drop whether it is 
twinned or not. In your case Rfree rises, which is actually hinting at your 
data not being twinned. How many reflections do you have in the Rfree set? 

Best,
Tim

On Friday, October 28, 2016 1:59:29 PM CEST Hillen, Hauke wrote:
> Dear Kay,
> 
> Thank you for your answer!
> I have meanwhile processed my anomalous data also also as P1 and solved the
> structure with MR-SAD in P1, which works flawlessly and gives the same
> solution. Compared to P21, I now have 4 instead of 2 molecules in the ASU,
> which makes sense. I then tried refinement in P1 with or without applying a
> twin law in phenix.refine. The result seems very clear. Without twin law,
> R/Rfree after one round of rigid body refinement drop to 34/38. With twin
> refinement, the same refinement results in R/Rfree of 29/42. My current
> suspicion is therefore that my data are truly P1, with an NCS axis parallel
> to a crystallographic axis (suggested by Xtriage). Could this
> pseudosymmetry lead XDS to falsely identify my data as P2 instead of P1?
 
> Best wishes,
> Hauke
> 
> 
> ---
> Hauke Hillen
> Department of Molecular Biology
> Prof. Dr. Patrick Cramer
> Max-Planck-Institute for Biophysical Chemistry
> Am Fassberg 11
> D-37077 Göttingen
> Germany
> 
> E-Mail: hauke.hil...@mpibpc.mpg.de
> Phone: +49-551-201-2864
> 
> 
> 
> 
> 
> On Oct 28, 2016, at 3:51 PM, Kay Diederichs
> >
> wrote:
 
> Dear Hauke,
> 
> the conversion of intensities to amplitudes is an area with unsatisfactory
> solutions, and the approaches of (and the assumptions made by) the
> different programs differ. Low-resolution data are often anisotropic, which
> adds difficulties. You could try CCP4 truncate/ctruncate and
> GlobalPhasing's staraniso. The latter in particular should treat
> anisotropic diffraction more properly. Probably you get yet other answers
> from xtriage when fed with those amplitudes. Possibly this will add to the
> confusion, but I'm not sure it is worth worrying about this too much.
 
> Having said that, it seems to me like the primary and most unspoiled data
> are the intensities. I don't see anything suspicious in the
> xtriage-on-intensities output that you posted.
 
> good luck with your difficult project!
> 
> Kay
> 

-- 
--
Paul Scherrer Institute
Dr. Tim Gruene
- persoenlich -
OFLC/102
CH-5232 Villigen PSI
phone: +41 (0)56 310 5297

GPG Key ID = A46BEE1A


signature.asc
Description: This is a digitally signed message part.

Re: [ccp4bb] intensity statistics and twinning

[Hillen, Hauke]

Dear Kay,

Thank you for your answer!
I have meanwhile processed my anomalous data also also as P1 and solved the 
structure with MR-SAD in P1, which works flawlessly and gives the same 
solution. Compared to P21, I now have 4 instead of 2 molecules in the ASU, 
which makes sense. I then tried refinement in P1 with or without applying a 
twin law in phenix.refine. The result seems very clear. Without twin law, 
R/Rfree after one round of rigid body refinement drop to 34/38. With twin 
refinement, the same refinement results in R/Rfree of 29/42. My current 
suspicion is therefore that my data are truly P1, with an NCS axis parallel to 
a crystallographic axis (suggested by Xtriage). Could this pseudosymmetry lead 
XDS to falsely identify my data as P2 instead of P1?

Best wishes,
Hauke


---
Hauke Hillen
Department of Molecular Biology
Prof. Dr. Patrick Cramer
Max-Planck-Institute for Biophysical Chemistry
Am Fassberg 11
D-37077 Göttingen
Germany

E-Mail: hauke.hil...@mpibpc.mpg.de
Phone: +49-551-201-2864





On Oct 28, 2016, at 3:51 PM, Kay Diederichs 
> wrote:

Dear Hauke,

the conversion of intensities to amplitudes is an area with unsatisfactory 
solutions, and the approaches of (and the assumptions made by) the different 
programs differ. Low-resolution data are often anisotropic, which adds 
difficulties. You could try CCP4 truncate/ctruncate and GlobalPhasing's 
staraniso. The latter in particular should treat anisotropic diffraction more 
properly. Probably you get yet other answers from xtriage when fed with those 
amplitudes. Possibly this will add to the confusion, but I'm not sure it is 
worth worrying about this too much.

Having said that, it seems to me like the primary and most unspoiled data are 
the intensities. I don't see anything suspicious in the xtriage-on-intensities 
output that you posted.

good luck with your difficult project!

Kay

Re: [ccp4bb] intensity statistics and twinning

[Kay Diederichs]

Dear Hauke,

the conversion of intensities to amplitudes is an area with unsatisfactory 
solutions, and the approaches of (and the assumptions made by) the different 
programs differ. Low-resolution data are often anisotropic, which adds 
difficulties. You could try CCP4 truncate/ctruncate and GlobalPhasing's 
staraniso. The latter in particular should treat anisotropic diffraction more 
properly. Probably you get yet other answers from xtriage when fed with those 
amplitudes. Possibly this will add to the confusion, but I'm not sure it is 
worth worrying about this too much. 
 
Having said that, it seems to me like the primary and most unspoiled data are 
the intensities. I don't see anything suspicious in the xtriage-on-intensities 
output that you posted. 

good luck with your difficult project!

Kay

Re: [ccp4bb] intensity statistics and twinning

[Keller, Jacob]

It would be helpful to know what your current R values are in modelling, and 
also the merging statistics. It looks like you might have a twinned p1 crystal.

JPK

From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Hillen, 
Hauke
Sent: Thursday, October 27, 2016 3:11 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] intensity statistics and twinning

Dear ccp4 community,

I am currently working on some low resolution datasets (around 4.5A). The space 
group seems to be P21, as suggested by XDS and pointless. I have collected many 
datasets of these crystals, both native as well as SeMet-labeled. Using MR-SAD 
I have been able to obtain a clearly interpretable electron density map for all 
features I expect and heavy atom sites that make sense for both the model used 
in MR and the yet unmodeled components. So far, so good.

While routinely analyzing my datasets with Phenix Xtriage, I have noticed that 
the intensity statistics for all of these datasets look unusual. In fact, 
Xtriage complains about this with the message: „The intensity statistics look 
unusual, but twinning is not indicated or possible in the given space group“ 
when processed in P21.
The occurence of this message depends somewhat on the typ of input file I use 
for the same dataset as well as the input parameters (high resolution cut-off). 
If I use XDSCONV to convert the intensities to amplitudes for phenix, this 
message appears. If I use the output of XSCALE directly as intensities, this 
message does not appear, yet the actual statistics are somewhat similar. I have 
attached the log file output for four scenarios at the end of this message (P21 
intensities, P21 amplitudes, P1 intensities, P1 amplitudes).
These results got me questioning whether the true space group is really P21, or 
whether it could be that it is P1 with some twinning issue. Since the Xtriage 
output regarding the „normality“ of the intensity statistics varies upon the 
input format, I assume that this case may be somewhat borderline. Since I have 
very little experience both with low-resolution crystals as well as with 
twinning, I am a bit unsure how to proceed with this data.
How can I distinguish between a partially twinned P1 crystal and an untwinned 
P21 crystal? It is my impression from previous discussions here that 
distinguishing twinned from untwinned data simply by comparing refinement 
results with and without twin laws is not always conclusive, as the R-factors 
are not directly comparable. If the crystal is truly P21, could these issues 
arise from intensity to amplitude conversion problems? (Xtriage also suggests 
this as a possibility) If so, can these be overcome? Or could the deviation 
from ideal intensities simply originate from the low quality (= resolution) of 
the data and are within the range of tolerance for such a dataset? Could this 
be some type of pseudosymmetry issue? And finally, what

I would be very grateful for any advice on how to proceed with these data!


Kind regards,
Hauke



Processed as P21, intensity input:

=== Diagnostic tests for twinning and pseudosymmetry ==

Using data between 10.00 to 3.50 Angstrom.

--Patterson analyses--

Largest Patterson peak with length larger than 15 Angstrom:
Frac. coord.  :0.1640.000   -0.021
Distance to origin:   17.720
Height relative to origin :3.072 %
p_value(height)   :1.000e+00

Explanation
The p-value, the probability that a peak of the specified height or larger
is found in a Patterson function of a macromolecule that does not have any
translational pseudo-symmetry, is equal to 1.000e+00.  p_values smaller than
0.05 might indicate weak translational pseudo symmetry, or the self vector of
a large anomalous scatterer such as Hg, whereas values smaller than 1e-3 are
a very strong indication for the presence of translational pseudo symmetry.


 --Wilson ratio and moments--

Acentric reflections:


  /^2:1.935   (untwinned: 2.000; perfect twin 1.500)
  ^2/:0.805   (untwinned: 0.785; perfect twin 0.885)
  <|E^2 - 1|>:0.696   (untwinned: 0.736; perfect twin 0.541)

Centric reflections:


  /^2:2.431   (untwinned: 3.000; perfect twin 2.000)
  ^2/:0.733   (untwinned: 0.637; perfect twin 0.785)
  <|E^2 - 1|>:0.812   (untwinned: 0.968; perfect twin 0.736)


  --NZ test for twinning and TNCS--


The NZ test is diagnostic for both twinning and translational NCS.  Note
however that if both are present, the effects may cancel each other out,
therefore the results of the Patterson analysis and L-test also need to be
considered.


 Maximum deviation acentric  :  0.028
 Maximum deviation centric   :  0.103

 _acentric  : -0.009
 _centric   : -0.061


 --L test for acentric data--

Using difference