Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Pearce, N.M. (Nick)]

Oh, sorry, I'm with you — thanks for clarifying. I thought you were suggesting 
some other effect (that I was unaware of) than standard systematic absences.

I don’t think this is the problem in our case, but you’re right the spacegroup 
issue is an interesting one to look out for if any reindexing occurs.

Thanks,
Nick

On 6 Apr 2018, at 02:02, James Holton 
mailto:jmhol...@slac.stanford.edu>> wrote:


I say "putative" because I don't know what your space group is.

In P212121 the reflection h,k,l = 0,0,1 is absent, but in P222 it is not 
absent.  So, if your unit cell is a=30 b=40 c=60 the lowest-angle hkl you will 
get is at 60 A resolution (0,0,1) in P222, but the lowest-angle reflection you 
will get out of P212121 will be (0,1,1), at 33.3 A resolution.  This is because 
0,1,0 is also absent.  So, if you ever specify P212121 in your pipeline the 
0,0,1 reflection will be lost.  Same thing happens with most any 
screw-vs-rotation axis assignment.

You loose other reflections to absences too, of course, but the lowest-order 
ones have an annoying habit of defining the "resolution range", and this can 
sometimes get set at one point in the pipeline and applied to subsequent 
operations, even if you change the space group back.  This could also be 
happening to you?

It is also possible to a subtle change in unit cell can move your lowest-order 
(and also the highest order) reflections across the defined "resolution range" 
boundaries.  Sometimes even round-off error can be enough.

So, if low-resolution is important it is always a good idea to replace the 
low-angle resolution limit with  A.  Just be sure your beamstop was 
properly masked off.

-James Holton
MAD Scientist


On 4/5/2018 10:55 AM, Pearce, N.M. (Nick) wrote:
Could you expand a bit on what you mean by a “putative” systematic absence? 
(e.g. why only the lowest order hkl?)



On 5 Apr 2018, at 19:39, James Holton 
mailto:jmhol...@slac.stanford.edu>> wrote:


You need to be careful with the exact space group at the particular stage in 
your pipeline here.  Often the lowest-order hkl is a putative systematic 
absence, so if you uniqueify in P222 you will get it, but if you uniqueify in 
P212121, then you won't.  That sort of thing.  Note that it doesn't matter what 
the "true" space group is, it only matters what is in the mtz header when you 
run uniqueify.

Could that be what is going on?

-James Holton

MAD Scisntist

On 4/5/2018 3:52 AM, Frank von Delft wrote:

Hello - can anybody shed light on this mystery:

We need (for PanDDA analysis) a lot of datasets each to have the complete set 
of low resolution indices, whether measured or not.  (Refmac adds the estimates 
as DFc, which is crucial when comparing maps.)

In ccp4, there are two obvious ways to get these indices complete:

  *   uniqueify
  *   CAD using the keyword "RESOLUTION FILE 1 999 "  (999 is the low 
resolution limit).

Mystifyingly, in ~1% of datasets, one or the other route misses one or two 
indices.  Our work-around is to go belt-and-braces and run both for each 
dataset.


It does however remain a bug.  Does anybody have any idea what's happening?  We 
can send example datasets to any volunteers who want to fiddle with it.

phx

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[graeme.win...@diamond.ac.uk]

Hi Frank

I think this is a feature of (c)truncate - to avoid actually absent reflections 
messing up the French & Wilson correction for present-but-weak-or-negative 
reflections they are chucked out.

If you put data in e.g. P212121 into AIMLESS absent reflections will be written 
out. They only vanish when you make F’s.

Cheers Graeme

On 6 Apr 2018, at 08:11, Frank von Delft 
mailto:frank.vonde...@sgc.ox.ac.uk>> wrote:


Thanks James, that's a very useful steer - this is definitely an easy thing to 
get mixed up with, we can go scratch around there.

I now vaguely recall an ancient BB thread, where people were asking why the 
systematic absences get scrubbed out of the mtz files at all.  I must agree 
that I do not understand the rationale either:  the symmetry should be simply a 
label attached to the reflections, not something that wipes out potential 
observations.

(I'd be very happy to be wrong about this.)

Frank





On 06/04/2018 01:01, James Holton wrote:

I say "putative" because I don't know what your space group is.

In P212121 the reflection h,k,l = 0,0,1 is absent, but in P222 it is not 
absent.  So, if your unit cell is a=30 b=40 c=60 the lowest-angle hkl you will 
get is at 60 A resolution (0,0,1) in P222, but the lowest-angle reflection you 
will get out of P212121 will be (0,1,1), at 33.3 A resolution.  This is because 
0,1,0 is also absent.  So, if you ever specify P212121 in your pipeline the 
0,0,1 reflection will be lost.  Same thing happens with most any 
screw-vs-rotation axis assignment.

You loose other reflections to absences too, of course, but the lowest-order 
ones have an annoying habit of defining the "resolution range", and this can 
sometimes get set at one point in the pipeline and applied to subsequent 
operations, even if you change the space group back.  This could also be 
happening to you?

It is also possible to a subtle change in unit cell can move your lowest-order 
(and also the highest order) reflections across the defined "resolution range" 
boundaries.  Sometimes even round-off error can be enough.

So, if low-resolution is important it is always a good idea to replace the 
low-angle resolution limit with  A.  Just be sure your beamstop was 
properly masked off.

-James Holton
MAD Scientist


On 4/5/2018 10:55 AM, Pearce, N.M. (Nick) wrote:
Could you expand a bit on what you mean by a “putative” systematic absence? 
(e.g. why only the lowest order hkl?)



On 5 Apr 2018, at 19:39, James Holton 
mailto:jmhol...@slac.stanford.edu>> wrote:


You need to be careful with the exact space group at the particular stage in 
your pipeline here.  Often the lowest-order hkl is a putative systematic 
absence, so if you uniqueify in P222 you will get it, but if you uniqueify in 
P212121, then you won't.  That sort of thing.  Note that it doesn't matter what 
the "true" space group is, it only matters what is in the mtz header when you 
run uniqueify.

Could that be what is going on?

-James Holton

MAD Scisntist

On 4/5/2018 3:52 AM, Frank von Delft wrote:

Hello - can anybody shed light on this mystery:

We need (for PanDDA analysis) a lot of datasets each to have the complete set 
of low resolution indices, whether measured or not.  (Refmac adds the estimates 
as DFc, which is crucial when comparing maps.)

In ccp4, there are two obvious ways to get these indices complete:

  *   uniqueify
  *   CAD using the keyword "RESOLUTION FILE 1 999 "  (999 is the low 
resolution limit).

Mystifyingly, in ~1% of datasets, one or the other route misses one or two 
indices.  Our work-around is to go belt-and-braces and run both for each 
dataset.


It does however remain a bug.  Does anybody have any idea what's happening?  We 
can send example datasets to any volunteers who want to fiddle with it.

phx







-- 
This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Frank von Delft]

Thanks James, that's a very useful steer - this is definitely an easy 
thing to get mixed up with, we can go scratch around there.


I now vaguely recall an ancient BB thread, where people were asking why 
the systematic absences get scrubbed out of the mtz files at all.  I 
must agree that I do not understand the rationale either:  the symmetry 
should be simply a label attached to the reflections, not something that 
wipes out potential observations.


(I'd be very happy to be wrong about this.)

Frank





On 06/04/2018 01:01, James Holton wrote:


I say "putative" because I don't know what your space group is.

In P212121 the reflection h,k,l = 0,0,1 is absent, but in P222 it is 
not absent.  So, if your unit cell is a=30 b=40 c=60 the lowest-angle 
hkl you will get is at 60 A resolution (0,0,1) in P222, but the 
lowest-angle reflection you will get out of P212121 will be (0,1,1), 
at 33.3 A resolution.  This is because 0,1,0 is also absent.  So, if 
you ever specify P212121 in your pipeline the 0,0,1 reflection will be 
lost.  Same thing happens with most any screw-vs-rotation axis 
assignment.


You loose other reflections to absences too, of course, but the 
lowest-order ones have an annoying habit of defining the "resolution 
range", and this can sometimes get set at one point in the pipeline 
and applied to subsequent operations, even if you change the space 
group back.  This could also be happening to you?


It is also possible to a subtle change in unit cell can move your 
lowest-order (and also the highest order) reflections across the 
defined "resolution range" boundaries.  Sometimes even round-off error 
can be enough.


So, if low-resolution is important it is always a good idea to replace 
the low-angle resolution limit with  A.  Just be sure your 
beamstop was properly masked off.


-James Holton
MAD Scientist


On 4/5/2018 10:55 AM, Pearce, N.M. (Nick) wrote:
Could you expand a bit on what you mean by a “putative” systematic 
absence? (e.g. why only the lowest order hkl?)




On 5 Apr 2018, at 19:39, James Holton > wrote:


You need to be careful with the exact space group at the particular 
stage in your pipeline here.  Often the lowest-order hkl is a 
putative systematic absence, so if you uniqueify in P222 you will 
get it, but if you uniqueify in P212121, then you won't.  That sort 
of thing.  Note that it doesn't matter what the "true" space group 
is, it only matters what is in the mtz header when you run uniqueify.


Could that be what is going on?

-James Holton

MAD Scisntist


On 4/5/2018 3:52 AM, Frank von Delft wrote:


Hello - can anybody shed light on this mystery:

We need (for PanDDA analysis) a lot of datasets each to have the 
complete set of low resolution indices, whether measured or not. 
(Refmac adds the estimates as DFc, which is crucial when comparing 
maps.)


In ccp4, there are two obvious ways to get these indices complete:

  * uniqueify
  * CAD using the keyword "RESOLUTION FILE 1 999 "  (999
is the low resolution limit).

Mystifyingly, in ~1% of datasets, one or the other route misses one 
or two indices.  Our work-around is to go belt-and-braces and run 
both for each dataset.



It does however remain a bug.  Does anybody have any idea what's 
happening?  We can send example datasets to any volunteers who want 
to fiddle with it.


phx

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[James Holton]

I say "putative" because I don't know what your space group is.

In P212121 the reflection h,k,l = 0,0,1 is absent, but in P222 it is not 
absent.  So, if your unit cell is a=30 b=40 c=60 the lowest-angle hkl 
you will get is at 60 A resolution (0,0,1) in P222, but the lowest-angle 
reflection you will get out of P212121 will be (0,1,1), at 33.3 A 
resolution.  This is because 0,1,0 is also absent.  So, if you ever 
specify P212121 in your pipeline the 0,0,1 reflection will be lost.  
Same thing happens with most any screw-vs-rotation axis assignment.


You loose other reflections to absences too, of course, but the 
lowest-order ones have an annoying habit of defining the "resolution 
range", and this can sometimes get set at one point in the pipeline and 
applied to subsequent operations, even if you change the space group 
back.  This could also be happening to you?


It is also possible to a subtle change in unit cell can move your 
lowest-order (and also the highest order) reflections across the defined 
"resolution range" boundaries.  Sometimes even round-off error can be 
enough.


So, if low-resolution is important it is always a good idea to replace 
the low-angle resolution limit with  A.  Just be sure your beamstop 
was properly masked off.


-James Holton
MAD Scientist


On 4/5/2018 10:55 AM, Pearce, N.M. (Nick) wrote:
Could you expand a bit on what you mean by a “putative” systematic 
absence? (e.g. why only the lowest order hkl?)




On 5 Apr 2018, at 19:39, James Holton > wrote:


You need to be careful with the exact space group at the particular 
stage in your pipeline here. Often the lowest-order hkl is a putative 
systematic absence, so if you uniqueify in P222 you will get it, but 
if you uniqueify in P212121, then you won't.  That sort of thing.  
Note that it doesn't matter what the "true" space group is, it only 
matters what is in the mtz header when you run uniqueify.


Could that be what is going on?

-James Holton

MAD Scisntist


On 4/5/2018 3:52 AM, Frank von Delft wrote:


Hello - can anybody shed light on this mystery:

We need (for PanDDA analysis) a lot of datasets each to have the 
complete set of low resolution indices, whether measured or not. 
(Refmac adds the estimates as DFc, which is crucial when comparing 
maps.)


In ccp4, there are two obvious ways to get these indices complete:

  * uniqueify
  * CAD using the keyword "RESOLUTION FILE 1 999 "  (999 is
the low resolution limit).

Mystifyingly, in ~1% of datasets, one or the other route misses one 
or two indices.  Our work-around is to go belt-and-braces and run 
both for each dataset.



It does however remain a bug.  Does anybody have any idea what's 
happening?  We can send example datasets to any volunteers who want 
to fiddle with it.


phx

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Pavel Afonine]

Just in case you find it helpful, you can get 100% complete set of
reflections (Fcalc) in specified resolution range using

phenix.fmodel model.pdb high_res=2.5 low_res=15

or if you leave out low_res it will go all the way up to theoretical limit
of low resolution.

If you have/use cctbx then I can show to do this at lower (Python) level.

Pavel

On Thu, Apr 5, 2018 at 3:52 AM, Frank von Delft  wrote:

> Hello - can anybody shed light on this mystery:
>
> We need (for PanDDA analysis) a lot of datasets each to have the complete
> set of low resolution indices, whether measured or not.  (Refmac adds the
> estimates as DFc, which is crucial when comparing maps.)
>
> In ccp4, there are two obvious ways to get these indices complete:
>
>- uniqueify
>- CAD using the keyword "RESOLUTION FILE 1 999 "  (999 is the
>low resolution limit).
>
> Mystifyingly, in ~1% of datasets, one or the other route misses one or two
> indices.  Our work-around is to go belt-and-braces and run both for each
> dataset.
>
>
> It does however remain a bug.  Does anybody have any idea what's
> happening?  We can send example datasets to any volunteers who want to
> fiddle with it.
>
> phx
>
>
>

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Pearce, N.M. (Nick)]

Could you expand a bit on what you mean by a “putative” systematic absence? 
(e.g. why only the lowest order hkl?)



On 5 Apr 2018, at 19:39, James Holton 
mailto:jmhol...@slac.stanford.edu>> wrote:


You need to be careful with the exact space group at the particular stage in 
your pipeline here.  Often the lowest-order hkl is a putative systematic 
absence, so if you uniqueify in P222 you will get it, but if you uniqueify in 
P212121, then you won't.  That sort of thing.  Note that it doesn't matter what 
the "true" space group is, it only matters what is in the mtz header when you 
run uniqueify.

Could that be what is going on?

-James Holton

MAD Scisntist

On 4/5/2018 3:52 AM, Frank von Delft wrote:

Hello - can anybody shed light on this mystery:

We need (for PanDDA analysis) a lot of datasets each to have the complete set 
of low resolution indices, whether measured or not.  (Refmac adds the estimates 
as DFc, which is crucial when comparing maps.)

In ccp4, there are two obvious ways to get these indices complete:

  *   uniqueify
  *   CAD using the keyword "RESOLUTION FILE 1 999 "  (999 is the low 
resolution limit).

Mystifyingly, in ~1% of datasets, one or the other route misses one or two 
indices.  Our work-around is to go belt-and-braces and run both for each 
dataset.


It does however remain a bug.  Does anybody have any idea what's happening?  We 
can send example datasets to any volunteers who want to fiddle with it.

phx

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[James Holton]

You need to be careful with the exact space group at the particular 
stage in your pipeline here.  Often the lowest-order hkl is a putative 
systematic absence, so if you uniqueify in P222 you will get it, but if 
you uniqueify in P212121, then you won't. That sort of thing.  Note that 
it doesn't matter what the "true" space group is, it only matters what 
is in the mtz header when you run uniqueify.


Could that be what is going on?

-James Holton

MAD Scisntist


On 4/5/2018 3:52 AM, Frank von Delft wrote:


Hello - can anybody shed light on this mystery:

We need (for PanDDA analysis) a lot of datasets each to have the 
complete set of low resolution indices, whether measured or not.  
(Refmac adds the estimates as DFc, which is crucial when comparing maps.)


In ccp4, there are two obvious ways to get these indices complete:

  * uniqueify
  * CAD using the keyword "RESOLUTION FILE 1 999 "  (999 is
the low resolution limit).

Mystifyingly, in ~1% of datasets, one or the other route misses one or 
two indices.  Our work-around is to go belt-and-braces and run both 
for each dataset.



It does however remain a bug.  Does anybody have any idea what's 
happening?  We can send example datasets to any volunteers who want to 
fiddle with it.


phx

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Kay Diederichs]

Hi Frank,

could you please be more specific, and give examples - the more the better? 
That would help enormously in debugging. Information required is cell, 
spacegroup, and which reflection(s) (as h,k,l triples) is/are missing. 

best,

Kay


On Thu, 5 Apr 2018 11:52:37 +0100, Frank von Delft 
 wrote:

>Hello - can anybody shed light on this mystery:
>
>We need (for PanDDA analysis) a lot of datasets each to have the 
>complete set of low resolution indices, whether measured or not. (Refmac 
>adds the estimates as DFc, which is crucial when comparing maps.)
>
>In ccp4, there are two obvious ways to get these indices complete:
>
>  * uniqueify
>  * CAD using the keyword "RESOLUTION FILE 1 999 " (999 is the
>low resolution limit).
>
>Mystifyingly, in ~1% of datasets, one or the other route misses one or 
>two indices.  Our work-around is to go belt-and-braces and run both for 
>each dataset.
>
>
>It does however remain a bug.  Does anybody have any idea what's 
>happening?  We can send example datasets to any volunteers who want to 
>fiddle with it.
>
>phx
>
>
>

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Ian Clifton]

I  wrote:

> Frank von Delft  writes:
>
>> No, the point is: uniqueify manages to not always do this. 
>
> The “uniqueify” script depends on the “unique” program, which seems to
> contain a built‐in low‐resolution limit of 50Å. Could it be this isn’t
> always low enough?

No, scrub that—the low resolution limit of the file that comes out
depends on the cell that you put in (that’ll teach me to use round
numbers for my test cell).
-- 
Ian ◎

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Ian Clifton]

Frank von Delft  writes:

> No, the point is: uniqueify manages to not always do this. 

The “uniqueify” script depends on the “unique” program, which seems to
contain a built‐in low‐resolution limit of 50Å. Could it be this isn’t
always low enough?
-- 
Ian Clifton ⚗ ℡: +44 1865 275677
Chemistry Research Laboratory ℻: +44 1865 285002
Oxford University 📧: ian.clif...@chem.ox.ac.uk
Mansfield Road   Oxford OX1 3TA   UK

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Frank von Delft]

No, the point is:  uniqueify manages to not /alw//ays/ do this.

I suppose I'm really asking:  who wants an example file to debug it?  
Because we have failed utterly.


Frank


On 05/04/2018 12:04, Eleanor Dodson wrote:

You need to be a bit more specific! - unbiqueify is meant to do this..
I didnt know CAD would generate indices not in the input file, unless 
you asked to generate a full set of FreeR terms, when the job I 
thought ran uniqueify?

Eleanor

On 5 April 2018 at 11:52, Frank von Delft > wrote:


Hello - can anybody shed light on this mystery:

We need (for PanDDA analysis) a lot of datasets each to have the
complete set of low resolution indices, whether measured or not. 
(Refmac adds the estimates as DFc, which is crucial when comparing
maps.)

In ccp4, there are two obvious ways to get these indices complete:

  * uniqueify
  * CAD using the keyword "RESOLUTION FILE 1 999 "  (999
is the low resolution limit).

Mystifyingly, in ~1% of datasets, one or the other route misses
one or two indices.  Our work-around is to go belt-and-braces and
run both for each dataset.


It does however remain a bug.  Does anybody have any idea what's
happening?  We can send example datasets to any volunteers who
want to fiddle with it.

phx

Re: [ccp4bb] (arcane) How to generate complete set of indices at low res

[Eleanor Dodson]

You need to be a bit more specific! - unbiqueify is meant to do this..
I didnt know CAD would generate indices not in the input file, unless you
asked to generate a full set of FreeR terms, when the job I thought ran
uniqueify?
Eleanor

On 5 April 2018 at 11:52, Frank von Delft 
wrote:

> Hello - can anybody shed light on this mystery:
>
> We need (for PanDDA analysis) a lot of datasets each to have the complete
> set of low resolution indices, whether measured or not.  (Refmac adds the
> estimates as DFc, which is crucial when comparing maps.)
>
> In ccp4, there are two obvious ways to get these indices complete:
>
>- uniqueify
>- CAD using the keyword "RESOLUTION FILE 1 999 "  (999 is the
>low resolution limit).
>
> Mystifyingly, in ~1% of datasets, one or the other route misses one or two
> indices.  Our work-around is to go belt-and-braces and run both for each
> dataset.
>
>
> It does however remain a bug.  Does anybody have any idea what's
> happening?  We can send example datasets to any volunteers who want to
> fiddle with it.
>
> phx
>
>
>