[ccp4bb] AW: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Herman . Schreuder]

Dear Jacob,

The big advantage of microscopes (whether using electrons or X-rays) is of 
course that you 1) do not need crystals and 2) get phase information. However, 
aligning an extremely large number of single molecule images is non-trivial and 
this is the reason it is still very hard, if not impossible, to use this 
technique for molecules with a mw of less than 100 kDa. 

Also, the X-ray image of a single molecule would be extremely weak and I am not 
sure current technology would be able to record such an image.

Crystals have billions of molecules, almost perfectly aligned which produce 
very good electron density maps. It is just that many proteins are very 
difficult if not impossible to crystallize that makes cryoEM so popular.

Best,
Herman 


 

-Ursprüngliche Nachricht-
Von: Keller, Jacob [mailto:kell...@janelia.hhmi.org] 
Gesendet: Freitag, 10. November 2017 16:55
An: Schreuder, Herman /DE; CCP4BB@JISCMAIL.AC.UK
Betreff: [EXTERNAL] RE: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging 
Conundrum

"Quality of image" has a lot of parameters, including resolution, noise, 
systematic errors, etc. I am not aware of a global "quality of image" metric.

One other consideration, not related to your comment: imagine if we had an 
x-ray lens through which we could take confocal images of a protein molecule or 
crystal, output as a voxel array. Would we really still prefer to measure 
diffraction patterns rather than the equivalent real space image, even assuming 
we had some perfect way to solve the phase problem? Or conversely, should we 
try to do fluorescence imaging in diffraction mode, due to its purported 
information efficiency?

JPK

-Original Message-
From: herman.schreu...@sanofi.com [mailto:herman.schreu...@sanofi.com]
Sent: Friday, November 10, 2017 10:22 AM
To: Keller, Jacob <kell...@janelia.hhmi.org>; CCP4BB@JISCMAIL.AC.UK
Subject: AW: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

At the bottom line, it is the quality of the image, not only the amount of 
pixels that counts. Adding more megapixels to a digital camera with a poor lens 
(as some manufacturers did), did not result in any sharper or better images.
Herman


-Ursprüngliche Nachricht-
Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von Keller, 
Jacob
Gesendet: Freitag, 10. November 2017 15:48
An: CCP4BB@JISCMAIL.AC.UK
Betreff: [EXTERNAL] Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging 
Conundrum

It seems, then, to be generally agreed that the conversion between voxels and 
Fourier terms was valid, each containing the same amount of information, but 
the problem was in the representation, and there was just trickery of the eye. 
I was thinking and hoping this would be so, since it allows a pretty direct 
comparison of crystal data to microscopic imaging data. I guess a litmus test 
would be to decide whether a voxel version of the electron density map would 
work equivalently well in crystallographic software, which I suspect it would. 
If so, then the same techniques--so effective in extracting information for the 
relatively information-poor crystal structures--could be used on fluorescence 
imaging data, which come in voxels.

Regarding information-wealth, in Dale's example, the whole hkl set was 4.1 MB. 
One frame in a garden-variety XYZT fluorescence image, however, contains about 
2000 x 2000 x 100 voxels at 16-bit, i.e., 400 million bits or 50 MB. In some 
data sets, these frames come at 10 Hz or more. I suspect that the I/sigma is 
also much better in the latter. So, with these data, and keeping a 
data:parameters ratio of ~4, one could model about 100 million parameters. This 
type of modelling, or any type of modelling for that matter, remains almost 
completely absent in the imaging world, perhaps because the data size is 
currently so unwieldy, perhaps also because sometimes people get nervous about 
model biases, perhaps also because people are still improving the imaging 
techniques. But just imagine what could be done with some crystallography-style 
modelling!

Jacob Keller



-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Tristan 
Croll
Sent: Friday, November 10, 2017 8:36 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

Or a nice familiar 2D example: the Ramachandran plot with 7.5 degree binning, 
as a grid (left) or with bicubic smoothing (right). Different visualisations of 
the same data, but the right-hand image uses it better.

On 2017-11-10 08:24, herman.schreu...@sanofi.com wrote:
> In line with Dale's suggestions, I would suggest that you reformat 
> your voxel map into the format of an electron density map and look at 
> it with coot. I am sure it will look much better and much more like 
> the electron density we are used to look at. Alternatively, you could 
> 

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Dale Tronrud]

On 11/12/2017 6:48 AM, Kay Diederichs wrote:
> On Fri, 10 Nov 2017 14:04:26 -0800, Dale Tronrud  
> wrote:
> ...
>>
>>   My belief is that the fact that our spot intensities represent the
>> amplitude (squared) of a series of Sin waves is the result of the hard
>> work of people like Bob who give us monochromatic illumination.
>> "Monochromatic" simply means it is a pure Sin wave.  If Bob could get
>> that shiny new ring of his to produce an electromagnetic square wave his
>> users would still get diffraction patterns with spots but they would
>> have to come up with programs that would perform Fourier summations of
>> square waves to calculate electron density.  Our instrument is an analog
>> computer for calculating the Sin wave Fourier transform of the electron
>> density of our crystal because we designed it to do exactly that.
>>
>> Dale Tronrud
>>
> ...
> 
> Hi Dale,
> 
> Well, perhaps I understand you wrongly, but I'd say if Bob would succeed in 
> making his synchrotron produce "square" instead of sine waves then we would 
> not have to change our programs too much, because a "square wave" can be 
> viewed as (or decomposed into) superpositions of a sine wave of a given 
> frequency/energy with its higher harmonics, at known amplitude ratios.
> This would be similar in some way to a Laue experiment, but not using a 
> continuum of energies, only discrete ones. The higher harmonics would just 
> change the intensities a bit (e.g. the 1,2,3 reflection would get some 
> additional intensity from the 2,4,6 and 3,6,9 reflection), and that change 
> could to a large extent be accounted for computationally, like we currently 
> do in de-twinning with low alpha. 
> That would probably be done in data processing, and might not affect the 
> downstream steps like map calculation.

   What you are describing (which is absolutely correct) sounds like a
lot more programming work than writing a square-wave Fourier transform
program.

   All I'm doing is trying to answer the very intriguing question that
beginners ask, but us old-timers tend to forget - Why are the intensity
of the Bragg spots the square of the amplitude of SIN waves?  The answer
I'm proposing is that the illumination source is a Sin wave so the
diffraction spots are in reference to Sin waves.  If Bob could give us
square waves the spot intensity would be proportional to the square of
the square wave Fourier transform of the density.  If ALS could give us
triangular waves their spots would tell us about the triangular wave
Fourier transform.

   While you want to continue to live in the Sin-wave world despite
having square waves in your experiment, I could be perverse and do the
same from my world.  Your Sin waves can be expressed as a sum of the
harmonics of my square waves and I could say that the intensity of what
you call the 1,2,3 reflection contains information from what I would
call the 1,2,3 and 2,4,6 and 3,6,9 (and so on) reflections.  The
mathematics is general and not specific to Sin waves.  It just happens
that it is easier for Bob to provide us with Sin wave illumination and
so our analysis uses Sin waves.

   This is quite abstract, but in the free electron laser world the
pulses are getting so short that they can't make the plane-wave
approximation and have to analyze their images in terms of the wave
packet, with its inherent bandwidth and coherence between the individual
frequencies within the packet.  See, my Sin-wave bias is showing -
"bandwidth" and "frequencies" both come from an insistence on reducing
all problems to Sin waves.  Maybe the free electron people would do
better by following Ethan and thinking about wavelets...

Dale Tronrud

> 
> best,
> 
> Kay
> 

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Kay Diederichs]

On Fri, 10 Nov 2017 14:04:26 -0800, Dale Tronrud  wrote:
...
>
>   My belief is that the fact that our spot intensities represent the
>amplitude (squared) of a series of Sin waves is the result of the hard
>work of people like Bob who give us monochromatic illumination.
>"Monochromatic" simply means it is a pure Sin wave.  If Bob could get
>that shiny new ring of his to produce an electromagnetic square wave his
>users would still get diffraction patterns with spots but they would
>have to come up with programs that would perform Fourier summations of
>square waves to calculate electron density.  Our instrument is an analog
>computer for calculating the Sin wave Fourier transform of the electron
>density of our crystal because we designed it to do exactly that.
>
>Dale Tronrud
>
...

Hi Dale,

Well, perhaps I understand you wrongly, but I'd say if Bob would succeed in 
making his synchrotron produce "square" instead of sine waves then we would not 
have to change our programs too much, because a "square wave" can be viewed as 
(or decomposed into) superpositions of a sine wave of a given frequency/energy 
with its higher harmonics, at known amplitude ratios.
This would be similar in some way to a Laue experiment, but not using a 
continuum of energies, only discrete ones. The higher harmonics would just 
change the intensities a bit (e.g. the 1,2,3 reflection would get some 
additional intensity from the 2,4,6 and 3,6,9 reflection), and that change 
could to a large extent be accounted for computationally, like we currently do 
in de-twinning with low alpha. 
That would probably be done in data processing, and might not affect the 
downstream steps like map calculation.

best,

Kay

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Dale Tronrud]

On 11/10/2017 1:38 PM, Robert Sweet wrote:
> This has been a fascinating thread. Thanks.
> 
> I will dip my oar in the water.  Here are a couple of snippets.
> 
>> Jacob: It was good of proto-crystallographers to invent diffraction as
>> a way to apply Fourier Series.
> 
> and
> 
>> Ethan: So here's the brain-teaser: Why does Nature use Fourier
>> transforms rather than Wavelet transforms? Or does she?
> 
> Probably Jacob was joking, but I believe we should say that physicists
> (and Ms. Nature) employ the Fourier transform/synthesis because this
> models pretty precisely the way that we believe light rays/waves of all
> energies interfere with one another.
> 
> Warm regards, Bob

   My belief is that the fact that our spot intensities represent the
amplitude (squared) of a series of Sin waves is the result of the hard
work of people like Bob who give us monochromatic illumination.
"Monochromatic" simply means it is a pure Sin wave.  If Bob could get
that shiny new ring of his to produce an electromagnetic square wave his
users would still get diffraction patterns with spots but they would
have to come up with programs that would perform Fourier summations of
square waves to calculate electron density.  Our instrument is an analog
computer for calculating the Sin wave Fourier transform of the electron
density of our crystal because we designed it to do exactly that.

Dale Tronrud


> 
> 
> On Fri, 10 Nov 2017, Keller, Jacob wrote:
> 
 My understanding is that EM people will routinely switch to
 diffraction mode when they want accurate measurements.  You lose the
 phase information but, since EM lenses tend to have imperfections,
 you get better measurements of the intensities.
>>
>> Only to my knowledge in the case of crystalline samples like 2D crystals.
>>
 Of course the loss of phases is a serious problem when you don't
 have a model of the object as precise as our atomic models.
>>
>> From where does this precision arise, I wonder? I guess priors for
>> atom-based models are pretty invariant. On the other hand, who says
>> that such priors, albeit of many more varieties, don't exist for
>> larger biological samples, such as zebrafish brains and drosophila
>> embryos/larvae? Anyway, right now, the state of the art of modelling
>> in these fluorescence data sets is hand-drawing circles around things
>> that look interesting, hoping the sample does not shift too much, or
>> perhaps using some tracking. But it could be so much better!
>>
>> JPK
>>
>>
> 

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Robert Sweet]

This has been a fascinating thread. Thanks.

I will dip my oar in the water.  Here are a couple of snippets.

Jacob: It was good of proto-crystallographers to invent diffraction as a 
way to apply Fourier Series.


and

Ethan: So here's the brain-teaser: Why does Nature use Fourier 
transforms rather than Wavelet transforms? Or does she?


Probably Jacob was joking, but I believe we should say that physicists 
(and Ms. Nature) employ the Fourier transform/synthesis because this 
models pretty precisely the way that we believe light rays/waves of all 
energies interfere with one another.


Warm regards, Bob


On Fri, 10 Nov 2017, Keller, Jacob wrote:


My understanding is that EM people will routinely switch to diffraction mode 
when they want accurate measurements.  You lose the phase information but, 
since EM lenses tend to have imperfections, you get better measurements of the 
intensities.


Only to my knowledge in the case of crystalline samples like 2D crystals.


Of course the loss of phases is a serious problem when you don't have a model 
of the object as precise as our atomic models.


From where does this precision arise, I wonder? I guess priors for atom-based 
models are pretty invariant. On the other hand, who says that such priors, 
albeit of many more varieties, don't exist for larger biological samples, such 
as zebrafish brains and drosophila embryos/larvae? Anyway, right now, the state 
of the art of modelling in these fluorescence data sets is hand-drawing circles 
around things that look interesting, hoping the sample does not shift too much, 
or perhaps using some tracking. But it could be so much better!

JPK

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Keller, Jacob]

>>My understanding is that EM people will routinely switch to diffraction mode 
>>when they want accurate measurements.  You lose the phase information but, 
>>since EM lenses tend to have imperfections, you get better measurements of 
>>the intensities.

Only to my knowledge in the case of crystalline samples like 2D crystals.

>>Of course the loss of phases is a serious problem when you don't have a model 
>>of the object as precise as our atomic models.

From where does this precision arise, I wonder? I guess priors for atom-based 
models are pretty invariant. On the other hand, who says that such priors, 
albeit of many more varieties, don't exist for larger biological samples, such 
as zebrafish brains and drosophila embryos/larvae? Anyway, right now, the state 
of the art of modelling in these fluorescence data sets is hand-drawing circles 
around things that look interesting, hoping the sample does not shift too much, 
or perhaps using some tracking. But it could be so much better!

JPK

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Dale Tronrud]

On 11/10/2017 7:55 AM, Keller, Jacob wrote:
> "Quality of image" has a lot of parameters, including resolution, noise, 
> systematic errors, etc. I am not aware of a global "quality of image" metric.
> 
> One other consideration, not related to your comment: imagine if we had an 
> x-ray lens through which we could take confocal images of a protein molecule 
> or crystal, output as a voxel array. Would we really still prefer to measure 
> diffraction patterns rather than the equivalent real space image, even 
> assuming we had some perfect way to solve the phase problem? Or conversely, 
> should we try to do fluorescence imaging in diffraction mode, due to its 
> purported information efficiency?

   It depends on the quality of your lens.  My understanding is that EM
people will routinely switch to diffraction mode when they want accurate
measurements.  You lose the phase information but, since EM lenses tend
to have imperfections, you get better measurements of the intensities.
Of course the loss of phases is a serious problem when you don't have a
model of the object as precise as our atomic models.

   The lens in a microscope tends to be of very high quality and you
don't have precise models of the object to calculate phases so there is
no advantage of going to "diffraction mode"..

Dale Tronrud

> 
> JPK
> 
> -Original Message-
> From: herman.schreu...@sanofi.com [mailto:herman.schreu...@sanofi.com] 
> Sent: Friday, November 10, 2017 10:22 AM
> To: Keller, Jacob <kell...@janelia.hhmi.org>; CCP4BB@JISCMAIL.AC.UK
> Subject: AW: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum
> 
> At the bottom line, it is the quality of the image, not only the amount of 
> pixels that counts. Adding more megapixels to a digital camera with a poor 
> lens (as some manufacturers did), did not result in any sharper or better 
> images.
> Herman
> 
> 
> -Ursprüngliche Nachricht-
> Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von 
> Keller, Jacob
> Gesendet: Freitag, 10. November 2017 15:48
> An: CCP4BB@JISCMAIL.AC.UK
> Betreff: [EXTERNAL] Re: [ccp4bb] AW: Re: [ccp4bb] Basic 
> Crystallography/Imaging Conundrum
> 
> It seems, then, to be generally agreed that the conversion between voxels and 
> Fourier terms was valid, each containing the same amount of information, but 
> the problem was in the representation, and there was just trickery of the 
> eye. I was thinking and hoping this would be so, since it allows a pretty 
> direct comparison of crystal data to microscopic imaging data. I guess a 
> litmus test would be to decide whether a voxel version of the electron 
> density map would work equivalently well in crystallographic software, which 
> I suspect it would. If so, then the same techniques--so effective in 
> extracting information for the relatively information-poor crystal 
> structures--could be used on fluorescence imaging data, which come in voxels.
> 
> Regarding information-wealth, in Dale's example, the whole hkl set was 4.1 
> MB. One frame in a garden-variety XYZT fluorescence image, however, contains 
> about 2000 x 2000 x 100 voxels at 16-bit, i.e., 400 million bits or 50 MB. In 
> some data sets, these frames come at 10 Hz or more. I suspect that the 
> I/sigma is also much better in the latter. So, with these data, and keeping a 
> data:parameters ratio of ~4, one could model about 100 million parameters. 
> This type of modelling, or any type of modelling for that matter, remains 
> almost completely absent in the imaging world, perhaps because the data size 
> is currently so unwieldy, perhaps also because sometimes people get nervous 
> about model biases, perhaps also because people are still improving the 
> imaging techniques. But just imagine what could be done with some 
> crystallography-style modelling!
> 
> Jacob Keller
> 
> 
> 
> -----Original Message-----
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Tristan 
> Croll
> Sent: Friday, November 10, 2017 8:36 AM
> To: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum
> 
> Or a nice familiar 2D example: the Ramachandran plot with 7.5 degree binning, 
> as a grid (left) or with bicubic smoothing (right). Different visualisations 
> of the same data, but the right-hand image uses it better.
> 
> On 2017-11-10 08:24, herman.schreu...@sanofi.com wrote:
>> In line with Dale's suggestions, I would suggest that you reformat 
>> your voxel map into the format of an electron density map and look at 
>> it with coot. I am sure it will look much better and much more like 
>> the electron density we are used to look at. Alternatively, you c

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Dale Tronrud]

   A second observation of the same experimental quantity does not
double the amount of "information".  We know from the many discussions
on this forum that the improvement of multiplicity is diminishing with
repetition.

   Measuring "information content" is very hard.  You can't just count
the bytes and say that measures the information content.  My example of
an oversampled map proves the point - The file is much bigger but can be
calculated exactly from the same, relatively small, number of
reflections.  The ultimate extreme is a map calculated from just the
F000 term.  One number can produces a map with gigibytes of data - It
just happens that all the numbers are equal.

   While our Bragg spots are pretty much independent measurements, after
merging, Herman is right about microscopes.  The physical nature of the
instrument introduces relationships between the values of the voxels so
the information content is smaller, perhaps by a lot, than the number of
bytes in the image.  You have to have a deep understanding of the lens
system to work out what is going on.  And a second image of the same
instrument of the same object measured a mSec later will be very highly
correlated with the first and add very little new "information" to the
experiment.

   BTW while we write maps as a set of numbers arranged in the 3D array,
it is not equivalent to an image.  The pixels, or voxels in 3D, indicate
the average value of that region while our map files contain the value
of the density at a particular point.  Our numbers are very distinct,
while pixels can be quite confusing.  In many detectors the area
averaged over is somewhat larger than the spacing of the pixels giving
the illusion of greater detail w/o actually providing more information.
This occurs in our CCD detectors where the X-ray photons are converted
to a lower frequency light by some sort of phosphor and in a microscope
by a poor lens (also as mentioned by Herman).

   Measuring information content is hard, which is why it is usually not
considered a rigorous quantity.  The classic example is the value of
ratio of the circumference of a circle to its diameter.  This number has
an infinite number of digits which could be considered an infinite
amount of information.  I can simply type "Pi", however, and accurately
express that infinity of information.  Just how much information is present?

Dale Tronrud

On 11/10/2017 6:47 AM, Keller, Jacob wrote:
> It seems, then, to be generally agreed that the conversion between voxels and 
> Fourier terms was valid, each containing the same amount of information, but 
> the problem was in the representation, and there was just trickery of the 
> eye. I was thinking and hoping this would be so, since it allows a pretty 
> direct comparison of crystal data to microscopic imaging data. I guess a 
> litmus test would be to decide whether a voxel version of the electron 
> density map would work equivalently well in crystallographic software, which 
> I suspect it would. If so, then the same techniques--so effective in 
> extracting information for the relatively information-poor crystal 
> structures--could be used on fluorescence imaging data, which come in voxels.
> 
> Regarding information-wealth, in Dale's example, the whole hkl set was 4.1 
> MB. One frame in a garden-variety XYZT fluorescence image, however, contains 
> about 2000 x 2000 x 100 voxels at 16-bit, i.e., 400 million bits or 50 MB. In 
> some data sets, these frames come at 10 Hz or more. I suspect that the 
> I/sigma is also much better in the latter. So, with these data, and keeping a 
> data:parameters ratio of ~4, one could model about 100 million parameters. 
> This type of modelling, or any type of modelling for that matter, remains 
> almost completely absent in the imaging world, perhaps because the data size 
> is currently so unwieldy, perhaps also because sometimes people get nervous 
> about model biases, perhaps also because people are still improving the 
> imaging techniques. But just imagine what could be done with some 
> crystallography-style modelling!
> 
> Jacob Keller
> 
> 
> 
> -Original Message-
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Tristan 
> Croll
> Sent: Friday, November 10, 2017 8:36 AM
> To: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum
> 
> Or a nice familiar 2D example: the Ramachandran plot with 7.5 degree binning, 
> as a grid (left) or with bicubic smoothing (right). Different visualisations 
> of the same data, but the right-hand image uses it better.
> 
> On 2017-11-10 08:24, herman.schreu...@sanofi.com wrote:
>> In line with Dale's suggestions, I would suggest that you reformat 
>> your voxel map into the format of an electron density map and l

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Keller, Jacob]

"Quality of image" has a lot of parameters, including resolution, noise, 
systematic errors, etc. I am not aware of a global "quality of image" metric.

One other consideration, not related to your comment: imagine if we had an 
x-ray lens through which we could take confocal images of a protein molecule or 
crystal, output as a voxel array. Would we really still prefer to measure 
diffraction patterns rather than the equivalent real space image, even assuming 
we had some perfect way to solve the phase problem? Or conversely, should we 
try to do fluorescence imaging in diffraction mode, due to its purported 
information efficiency?

JPK

-Original Message-
From: herman.schreu...@sanofi.com [mailto:herman.schreu...@sanofi.com] 
Sent: Friday, November 10, 2017 10:22 AM
To: Keller, Jacob <kell...@janelia.hhmi.org>; CCP4BB@JISCMAIL.AC.UK
Subject: AW: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

At the bottom line, it is the quality of the image, not only the amount of 
pixels that counts. Adding more megapixels to a digital camera with a poor lens 
(as some manufacturers did), did not result in any sharper or better images.
Herman


-Ursprüngliche Nachricht-
Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von Keller, 
Jacob
Gesendet: Freitag, 10. November 2017 15:48
An: CCP4BB@JISCMAIL.AC.UK
Betreff: [EXTERNAL] Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging 
Conundrum

It seems, then, to be generally agreed that the conversion between voxels and 
Fourier terms was valid, each containing the same amount of information, but 
the problem was in the representation, and there was just trickery of the eye. 
I was thinking and hoping this would be so, since it allows a pretty direct 
comparison of crystal data to microscopic imaging data. I guess a litmus test 
would be to decide whether a voxel version of the electron density map would 
work equivalently well in crystallographic software, which I suspect it would. 
If so, then the same techniques--so effective in extracting information for the 
relatively information-poor crystal structures--could be used on fluorescence 
imaging data, which come in voxels.

Regarding information-wealth, in Dale's example, the whole hkl set was 4.1 MB. 
One frame in a garden-variety XYZT fluorescence image, however, contains about 
2000 x 2000 x 100 voxels at 16-bit, i.e., 400 million bits or 50 MB. In some 
data sets, these frames come at 10 Hz or more. I suspect that the I/sigma is 
also much better in the latter. So, with these data, and keeping a 
data:parameters ratio of ~4, one could model about 100 million parameters. This 
type of modelling, or any type of modelling for that matter, remains almost 
completely absent in the imaging world, perhaps because the data size is 
currently so unwieldy, perhaps also because sometimes people get nervous about 
model biases, perhaps also because people are still improving the imaging 
techniques. But just imagine what could be done with some crystallography-style 
modelling!

Jacob Keller



-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Tristan 
Croll
Sent: Friday, November 10, 2017 8:36 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

Or a nice familiar 2D example: the Ramachandran plot with 7.5 degree binning, 
as a grid (left) or with bicubic smoothing (right). Different visualisations of 
the same data, but the right-hand image uses it better.

On 2017-11-10 08:24, herman.schreu...@sanofi.com wrote:
> In line with Dale's suggestions, I would suggest that you reformat 
> your voxel map into the format of an electron density map and look at 
> it with coot. I am sure it will look much better and much more like 
> the electron density we are used to look at. Alternatively, you could 
> display an bona fide electron density map as voxel blocks and I am 
> sure it will look similar to the voxel map you showed in your first 
> email.
> 
> Best,
> Herman
> 
> -Ursprüngliche Nachricht-
> Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von 
> Dale Tronrud
> Gesendet: Freitag, 10. November 2017 08:08
> An: CCP4BB@JISCMAIL.AC.UK
> Betreff: [EXTERNAL] Re: [ccp4bb] Basic Crystallography/Imaging 
> Conundrum
> 
>Ethan and I apparently agree that anomalous scattering is "normal"
> and Friedel's Law is just an approximation.  I'll presume that your 
> "unique" is assuming otherwise and your 62,500 reflections only 
> include half of reciprocal space.  The full sphere of data would 
> include 125,000 reflections.  Since the cube root of 125,000 is 50 you 
> get a range of indices from -25 to +25 which would give you 2 A 
> resolution, which is still far from your hope of 1 A.
> 
>For your test case of 

[ccp4bb] AW: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Herman . Schreuder]

At the bottom line, it is the quality of the image, not only the amount of 
pixels that counts. Adding more megapixels to a digital camera with a poor lens 
(as some manufacturers did), did not result in any sharper or better images.
Herman


-Ursprüngliche Nachricht-
Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von Keller, 
Jacob
Gesendet: Freitag, 10. November 2017 15:48
An: CCP4BB@JISCMAIL.AC.UK
Betreff: [EXTERNAL] Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging 
Conundrum

It seems, then, to be generally agreed that the conversion between voxels and 
Fourier terms was valid, each containing the same amount of information, but 
the problem was in the representation, and there was just trickery of the eye. 
I was thinking and hoping this would be so, since it allows a pretty direct 
comparison of crystal data to microscopic imaging data. I guess a litmus test 
would be to decide whether a voxel version of the electron density map would 
work equivalently well in crystallographic software, which I suspect it would. 
If so, then the same techniques--so effective in extracting information for the 
relatively information-poor crystal structures--could be used on fluorescence 
imaging data, which come in voxels.

Regarding information-wealth, in Dale's example, the whole hkl set was 4.1 MB. 
One frame in a garden-variety XYZT fluorescence image, however, contains about 
2000 x 2000 x 100 voxels at 16-bit, i.e., 400 million bits or 50 MB. In some 
data sets, these frames come at 10 Hz or more. I suspect that the I/sigma is 
also much better in the latter. So, with these data, and keeping a 
data:parameters ratio of ~4, one could model about 100 million parameters. This 
type of modelling, or any type of modelling for that matter, remains almost 
completely absent in the imaging world, perhaps because the data size is 
currently so unwieldy, perhaps also because sometimes people get nervous about 
model biases, perhaps also because people are still improving the imaging 
techniques. But just imagine what could be done with some crystallography-style 
modelling!

Jacob Keller



-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Tristan 
Croll
Sent: Friday, November 10, 2017 8:36 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

Or a nice familiar 2D example: the Ramachandran plot with 7.5 degree binning, 
as a grid (left) or with bicubic smoothing (right). Different visualisations of 
the same data, but the right-hand image uses it better.

On 2017-11-10 08:24, herman.schreu...@sanofi.com wrote:
> In line with Dale's suggestions, I would suggest that you reformat 
> your voxel map into the format of an electron density map and look at 
> it with coot. I am sure it will look much better and much more like 
> the electron density we are used to look at. Alternatively, you could 
> display an bona fide electron density map as voxel blocks and I am 
> sure it will look similar to the voxel map you showed in your first 
> email.
> 
> Best,
> Herman
> 
> -Ursprüngliche Nachricht-
> Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von 
> Dale Tronrud
> Gesendet: Freitag, 10. November 2017 08:08
> An: CCP4BB@JISCMAIL.AC.UK
> Betreff: [EXTERNAL] Re: [ccp4bb] Basic Crystallography/Imaging 
> Conundrum
> 
>Ethan and I apparently agree that anomalous scattering is "normal"
> and Friedel's Law is just an approximation.  I'll presume that your 
> "unique" is assuming otherwise and your 62,500 reflections only 
> include half of reciprocal space.  The full sphere of data would 
> include 125,000 reflections.  Since the cube root of 125,000 is 50 you 
> get a range of indices from -25 to +25 which would give you 2 A 
> resolution, which is still far from your hope of 1 A.
> 
>For your test case of 1 A resolution with 50 A cell lengths you 
> want your indices to run from -50 to +50 giving a box of reflections 
> in reciprocal space 101 spots wide in each direction and a total of
> 101^3 =
> 1,030,301 reflections. (or 515,150.5 reflections for your Friedel 
> unique with the "half" reflection being the F000 which would then be 
> purely real valued.)
> 
>Assuming you can fit your structure factors into 16 bits (You had 
> better not have many more than 10,000 atoms if you don't want your
> F000 to overflow.) the information content will be 1,030,301 * 2 * 16 
> bits (The "2" because they are complex.) giving 32,969,632 bits.
> 
>If you spread this same amount of information across real space you 
> will have 1,030,301 complex density values in a 50x50x50 A space 
> giving a sampling rate along each axis of 101 samples/unit cell.
> 
>Complex density values?  The re

Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Keller, Jacob]

It seems, then, to be generally agreed that the conversion between voxels and 
Fourier terms was valid, each containing the same amount of information, but 
the problem was in the representation, and there was just trickery of the eye. 
I was thinking and hoping this would be so, since it allows a pretty direct 
comparison of crystal data to microscopic imaging data. I guess a litmus test 
would be to decide whether a voxel version of the electron density map would 
work equivalently well in crystallographic software, which I suspect it would. 
If so, then the same techniques--so effective in extracting information for the 
relatively information-poor crystal structures--could be used on fluorescence 
imaging data, which come in voxels.

Regarding information-wealth, in Dale's example, the whole hkl set was 4.1 MB. 
One frame in a garden-variety XYZT fluorescence image, however, contains about 
2000 x 2000 x 100 voxels at 16-bit, i.e., 400 million bits or 50 MB. In some 
data sets, these frames come at 10 Hz or more. I suspect that the I/sigma is 
also much better in the latter. So, with these data, and keeping a 
data:parameters ratio of ~4, one could model about 100 million parameters. This 
type of modelling, or any type of modelling for that matter, remains almost 
completely absent in the imaging world, perhaps because the data size is 
currently so unwieldy, perhaps also because sometimes people get nervous about 
model biases, perhaps also because people are still improving the imaging 
techniques. But just imagine what could be done with some crystallography-style 
modelling!

Jacob Keller



-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Tristan 
Croll
Sent: Friday, November 10, 2017 8:36 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

Or a nice familiar 2D example: the Ramachandran plot with 7.5 degree binning, 
as a grid (left) or with bicubic smoothing (right). Different visualisations of 
the same data, but the right-hand image uses it better.

On 2017-11-10 08:24, herman.schreu...@sanofi.com wrote:
> In line with Dale's suggestions, I would suggest that you reformat 
> your voxel map into the format of an electron density map and look at 
> it with coot. I am sure it will look much better and much more like 
> the electron density we are used to look at. Alternatively, you could 
> display an bona fide electron density map as voxel blocks and I am 
> sure it will look similar to the voxel map you showed in your first 
> email.
> 
> Best,
> Herman
> 
> -Ursprüngliche Nachricht-
> Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von 
> Dale Tronrud
> Gesendet: Freitag, 10. November 2017 08:08
> An: CCP4BB@JISCMAIL.AC.UK
> Betreff: [EXTERNAL] Re: [ccp4bb] Basic Crystallography/Imaging 
> Conundrum
> 
>Ethan and I apparently agree that anomalous scattering is "normal"
> and Friedel's Law is just an approximation.  I'll presume that your 
> "unique" is assuming otherwise and your 62,500 reflections only 
> include half of reciprocal space.  The full sphere of data would 
> include 125,000 reflections.  Since the cube root of 125,000 is 50 you 
> get a range of indices from -25 to +25 which would give you 2 A 
> resolution, which is still far from your hope of 1 A.
> 
>For your test case of 1 A resolution with 50 A cell lengths you 
> want your indices to run from -50 to +50 giving a box of reflections 
> in reciprocal space 101 spots wide in each direction and a total of
> 101^3 =
> 1,030,301 reflections. (or 515,150.5 reflections for your Friedel 
> unique with the "half" reflection being the F000 which would then be 
> purely real valued.)
> 
>Assuming you can fit your structure factors into 16 bits (You had 
> better not have many more than 10,000 atoms if you don't want your
> F000 to overflow.) the information content will be 1,030,301 * 2 * 16 
> bits (The "2" because they are complex.) giving 32,969,632 bits.
> 
>If you spread this same amount of information across real space you 
> will have 1,030,301 complex density values in a 50x50x50 A space 
> giving a sampling rate along each axis of 101 samples/unit cell.
> 
>Complex density values?  The real part of the density is what we 
> call the electron density and the imaginary part we call the anomalous 
> density.  If there is no anomalous scattering then Friedel's Law holds 
> and the number of unique reflections is cut in half and the density 
> values are purely real valued - The information content in both spaces 
> is cut in half and they remain equal.
> 
>By sampling your unit cell with 101 samples their rate is half that 
> of the wavelength of the highest frequency refle

[ccp4bb] AW: Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Herman . Schreuder]

In line with Dale's suggestions, I would suggest that you reformat your voxel 
map into the format of an electron density map and look at it with coot. I am 
sure it will look much better and much more like the electron density we are 
used to look at. Alternatively, you could display an bona fide electron density 
map as voxel blocks and I am sure it will look similar to the voxel map you 
showed in your first email.

Best,
Herman

-Ursprüngliche Nachricht-
Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von Dale 
Tronrud
Gesendet: Freitag, 10. November 2017 08:08
An: CCP4BB@JISCMAIL.AC.UK
Betreff: [EXTERNAL] Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

   Ethan and I apparently agree that anomalous scattering is "normal"
and Friedel's Law is just an approximation.  I'll presume that your "unique" is 
assuming otherwise and your 62,500 reflections only include half of reciprocal 
space.  The full sphere of data would include 125,000 reflections.  Since the 
cube root of 125,000 is 50 you get a range of indices from -25 to +25 which 
would give you 2 A resolution, which is still far from your hope of 1 A.

   For your test case of 1 A resolution with 50 A cell lengths you want your 
indices to run from -50 to +50 giving a box of reflections in reciprocal space 
101 spots wide in each direction and a total of 101^3 =
1,030,301 reflections. (or 515,150.5 reflections for your Friedel unique with 
the "half" reflection being the F000 which would then be purely real valued.)

   Assuming you can fit your structure factors into 16 bits (You had better not 
have many more than 10,000 atoms if you don't want your F000 to overflow.) the 
information content will be 1,030,301 * 2 * 16 bits (The "2" because they are 
complex.) giving 32,969,632 bits.

   If you spread this same amount of information across real space you will 
have 1,030,301 complex density values in a 50x50x50 A space giving a sampling 
rate along each axis of 101 samples/unit cell.

   Complex density values?  The real part of the density is what we call the 
electron density and the imaginary part we call the anomalous density.  If 
there is no anomalous scattering then Friedel's Law holds and the number of 
unique reflections is cut in half and the density values are purely real valued 
- The information content in both spaces is cut in half and they remain equal.

   By sampling your unit cell with 101 samples their rate is half that of the 
wavelength of the highest frequency reflection.  (e.q. a sampling rate of 0.5 A 
for 1 A resolution data)  This is, of course, the Nyquist Theorem which states 
that you have to sample at twice the frequency of the highest resolution 
Fourier coefficient.

  This is exactly how an FFT works.  It allocates the memory required to store 
the structure factors and it returns the map in that same array - The number of 
bytes is unchanged.  It also guarantees that the calculation is reversible as 
no information is lost in either direction.

   So, why does your blocky image look so bad?  First you have sampled too 
coarsely.  You should have twice the sampling rate in each direction.

   The next point is more subtle.  You are displaying each voxel as a block.  
This is not correct.  The sharp lines that occur at the boundaries between the 
blocks is a high frequency feature which is not consistent with a 1 A 
resolution image.  Your sample points should be displayed at discrete points 
since they are not the average density within a block but the value of the 
density at one specific point.

   What is the density of the map between the sampled points?  The Fourier 
series provides all the information needed to calculate them and you can 
calculate values for as fine a sampling rate as you like, just remember that 
you are not adding any more information because these new points are correlated 
with each other.

   If you have only the samples of a map and want to calculate Fourier 
coefficients there are many sets of Fourier coefficients that will reproduce 
the sampled points equally well.  We specify a unique solution in the FFT by 
defining that all reflections of resolution higher than 1 A must be identically 
equal to zero.  When you calculate a map from a set of coefficients that only 
go to 1 A resolution this is guaranteed.

   When you are calculating coefficients from any old map you had better ensure 
that the map you are sampling does not contain information of a higher 
resolution than twice your sampling rate.  This is a problem when calculating 
Fcalc from an atomic model.  You calculate a map from the model and FFT it, but 
you can't sample that map at 1/2 the resolution of your interest.  You must 
sample that map much more finely because an atomic model implies Fourier 
coefficients of very high resolution.
(Otherwise phase extension would be impossible)  This problem was discussed in 
detail in Lynn Ten Eyck'

Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Dale Tronrud]

   Ethan and I apparently agree that anomalous scattering is "normal"
and Friedel's Law is just an approximation.  I'll presume that your
"unique" is assuming otherwise and your 62,500 reflections only include
half of reciprocal space.  The full sphere of data would include 125,000
reflections.  Since the cube root of 125,000 is 50 you get a range of
indices from -25 to +25 which would give you 2 A resolution, which is
still far from your hope of 1 A.

   For your test case of 1 A resolution with 50 A cell lengths you want
your indices to run from -50 to +50 giving a box of reflections in
reciprocal space 101 spots wide in each direction and a total of 101^3 =
1,030,301 reflections. (or 515,150.5 reflections for your Friedel unique
with the "half" reflection being the F000 which would then be purely
real valued.)

   Assuming you can fit your structure factors into 16 bits (You had
better not have many more than 10,000 atoms if you don't want your F000
to overflow.) the information content will be 1,030,301 * 2 * 16 bits
(The "2" because they are complex.) giving 32,969,632 bits.

   If you spread this same amount of information across real space you
will have 1,030,301 complex density values in a 50x50x50 A space giving
a sampling rate along each axis of 101 samples/unit cell.

   Complex density values?  The real part of the density is what we call
the electron density and the imaginary part we call the anomalous
density.  If there is no anomalous scattering then Friedel's Law holds
and the number of unique reflections is cut in half and the density
values are purely real valued - The information content in both spaces
is cut in half and they remain equal.

   By sampling your unit cell with 101 samples their rate is half that
of the wavelength of the highest frequency reflection.  (e.q. a sampling
rate of 0.5 A for 1 A resolution data)  This is, of course, the Nyquist
Theorem which states that you have to sample at twice the frequency of
the highest resolution Fourier coefficient.

  This is exactly how an FFT works.  It allocates the memory required to
store the structure factors and it returns the map in that same array -
The number of bytes is unchanged.  It also guarantees that the
calculation is reversible as no information is lost in either direction.

   So, why does your blocky image look so bad?  First you have sampled
too coarsely.  You should have twice the sampling rate in each direction.

   The next point is more subtle.  You are displaying each voxel as a
block.  This is not correct.  The sharp lines that occur at the
boundaries between the blocks is a high frequency feature which is not
consistent with a 1 A resolution image.  Your sample points should be
displayed at discrete points since they are not the average density
within a block but the value of the density at one specific point.

   What is the density of the map between the sampled points?  The
Fourier series provides all the information needed to calculate them and
you can calculate values for as fine a sampling rate as you like, just
remember that you are not adding any more information because these new
points are correlated with each other.

   If you have only the samples of a map and want to calculate Fourier
coefficients there are many sets of Fourier coefficients that will
reproduce the sampled points equally well.  We specify a unique solution
in the FFT by defining that all reflections of resolution higher than 1
A must be identically equal to zero.  When you calculate a map from a
set of coefficients that only go to 1 A resolution this is guaranteed.

   When you are calculating coefficients from any old map you had better
ensure that the map you are sampling does not contain information of a
higher resolution than twice your sampling rate.  This is a problem when
calculating Fcalc from an atomic model.  You calculate a map from the
model and FFT it, but you can't sample that map at 1/2 the resolution of
your interest.  You must sample that map much more finely because an
atomic model implies Fourier coefficients of very high resolution.
(Otherwise phase extension would be impossible)  This problem was
discussed in detail in Lynn Ten Eyck's 1976 paper on Fcalc FFT's but is
often forgotten.  Gerard Bricogne's papers on NCS averaging from the
1970's also discusses these matters in great depth.

   In summary, your blocky picture (even with double sampling) is not a
valid representation because it is not blurry like a 1 A resolution map
should be.  To create an accurate image you need to oversample the map
sufficiently to prevent the human eye from detecting aliasing artifacts
such as the straight lines visible in your blocky picture.  This
requires very fine sampling because the eye is very sensitive to
straight lines.  When using a map for any purpose other than FFTing you
will need to oversample the map by some amount to prevent aliasing
artifacts and the amount of oversampling will depend on what you are
doing to 

Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Ethan Merritt]

On Friday, 10 November 2017 05:29:09 Keller, Jacob wrote:
> >>62500 is < 40^3, so ±20 indices on each axis.
> 50Å / 20 = 2.5Å,  so not quite 2.5Å resolution
> 
> Nice--thanks for calculating that. Couldn't remember how to do it off-hand, 
> and I guess my over-estimate comes from most protein crystals having some 
> symmetry. I don't really think it affects the question though--do you?
> 
> >>All that proves is that assigning each 1x1x1 voxel a separate density value 
> >>is a very inefficient use of information.  Adjacent voxels are not 
> >>independent, and no possible assignment of values will get around the 
> >>inherent blockiness of the representation.
> 
> Not sure what this means--what is the precise definition or measure of 
> "efficient use of information?" Like a compression algorithm? 

If it helps you to think of it that way, fine.
Suppose it is possible to compress a data set losslessly.
The information content is unchanged, but the compressed representation
is smaller than the original, so the information content per unit of size
is higher - a better use of space - hence "more efficient".

> Are diffraction data sets like compressed data?

Not the diffraction data, no.

But it is true that a truncated Fourier series is one way of compressing data.
Because of the truncation. it is a lossy, rather than lossless, compression.
An infinite series could give infinite resolution, but a truncated series is 
limited by the resolution of terms that are kept after truncation.

For example the compression used in JPEG is a truncated discrete cosine
transform (DCT), making JPEG files smaller than the original pixel-by-pixel 
image.

I'll throw a brain-teaser back at you.

As just noted, encoding the continuous electron density distribution in a
unit cell as a truncated Fourier series is essentially creating a JPEG image of
the original.  It is lossy, but as we know from experience JPEG images are 
pretty good at retaining the "feel" of the origin even with fairly severe
truncation.

But newer compression algorithms like JPEG2000 don't use DCTs,
instead they use wavelets.   I won't get sidetracked by trying to describe
wavelets, but the point is that by switching from a series of cosines to
a series of wavelets you can get higher compression.  They are
more efficient in representing the original data at a selected resolution.   

So here's the brain-teaser:
Why does Nature use Fourier transforms rather than Wavelet transforms?
Or does she?
Have we crystallographers been fooled into describing our experiments
in terms of Fourier transforms when we could do better by using wavelets
or some other transform entirely?

Ethan
 


> Also, the "blockiness" of representation is totally ancillary--you can do all 
> of the smoothing you want, I think, and the voxel map will still be basically 
> lousy. No?

> >>I know!  Let's instead of assigning a magnitude per voxel, let's assign a 
> >>magnitude per something-resolution-sensitive, like a sin wave.   Then for 
> >>each hkl measurement we get one sin wave term.   Add up all the sine waves 
> >>and what do you get?  Ta da.  A nice map.
> 
> It was good of proto-crystallographers to invent diffraction as a way to 
> apply Fourier Series. I don't know--it seems funny to me that somehow 
> diffraction is able to harness "efficient information use," whereas the voxel 
> map is not. I am looking for more insight into this.
> 
> >>Aren't Fourier series marvelous?
> 
> Well, I have always liked FTs, but your explanations are not particularly 
> enlightening to me yet.
> 
> I will re-iterate that the reason I brought this up is that the imaging world 
> might learn a lot from crystallography's incredible extraction of all 
> possible information through the use of priors and modelling.
> 
> Also, I hope you noticed that all of the parameters about the 
> crystallographic data set were extremely optimistic, and in reality the 
> information content would be far less.
> 
> One could compare the information content of the derived structure to that of 
> the measurements to get a metric for "information extraction," perhaps, and 
> this could be applied across many types of experiments in different fields. I 
> nominate crystallography for the best ratio.
> 
> JPK
> 
> 
> 
>  
> > Assuming that it is apt, however: is this a possible way to see the power 
> > of all of our Bayesian modelling? Could one use our modelling tools on such 
> > a grainy picture and arrive at similar results?
> >
> > Are our data sets really this poor in information, and we just model the 
> > heck out of them, as perhaps evidenced by our scarily low data:parameters 
> > ratios?
> > 
> > My underlying motivation in this thought experiment is to illustrate the 
> > richness in information (and poorness of modelling) that one achieves in 
> > fluorescence microscopic imaging. If crystallography is any measure of the 
> > power of modelling, one could really go to town on some of these terabyte 

Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Keller, Jacob]

>>62500 is < 40^3, so ±20 indices on each axis.
50Å / 20 = 2.5Å,  so not quite 2.5Å resolution

Nice--thanks for calculating that. Couldn't remember how to do it off-hand, and 
I guess my over-estimate comes from most protein crystals having some symmetry. 
I don't really think it affects the question though--do you?

>>All that proves is that assigning each 1x1x1 voxel a separate density value 
>>is a very inefficient use of information.  Adjacent voxels are not 
>>independent, and no possible assignment of values will get around the 
>>inherent blockiness of the representation.

Not sure what this means--what is the precise definition or measure of 
"efficient use of information?" Like a compression algorithm? Are diffraction 
data sets like compressed data?

Also, the "blockiness" of representation is totally ancillary--you can do all 
of the smoothing you want, I think, and the voxel map will still be basically 
lousy. No?

>>I know!  Let's instead of assigning a magnitude per voxel, let's assign a 
>>magnitude per something-resolution-sensitive, like a sin wave.   Then for 
>>each hkl measurement we get one sin wave term.   Add up all the sine waves 
>>and what do you get?  Ta da.  A nice map.

It was good of proto-crystallographers to invent diffraction as a way to apply 
Fourier Series. I don't know--it seems funny to me that somehow diffraction is 
able to harness "efficient information use," whereas the voxel map is not. I am 
looking for more insight into this.

>>Aren't Fourier series marvelous?

Well, I have always liked FTs, but your explanations are not particularly 
enlightening to me yet.

I will re-iterate that the reason I brought this up is that the imaging world 
might learn a lot from crystallography's incredible extraction of all possible 
information through the use of priors and modelling.

Also, I hope you noticed that all of the parameters about the crystallographic 
data set were extremely optimistic, and in reality the information content 
would be far less.

One could compare the information content of the derived structure to that of 
the measurements to get a metric for "information extraction," perhaps, and 
this could be applied across many types of experiments in different fields. I 
nominate crystallography for the best ratio.

JPK



 
> Assuming that it is apt, however: is this a possible way to see the power of 
> all of our Bayesian modelling? Could one use our modelling tools on such a 
> grainy picture and arrive at similar results?
>
> Are our data sets really this poor in information, and we just model the heck 
> out of them, as perhaps evidenced by our scarily low data:parameters ratios?
> 
> My underlying motivation in this thought experiment is to illustrate the 
> richness in information (and poorness of modelling) that one achieves in 
> fluorescence microscopic imaging. If crystallography is any measure of the 
> power of modelling, one could really go to town on some of these terabyte 5D 
> functional data sets we see around here at Janelia (and on YouTube).
> 
> What do you think?
> 
> Jacob Keller
> 
> +
> Jacob Pearson Keller
> Research Scientist / Looger Lab
> HHMI Janelia Research Campus
> 19700 Helix Dr, Ashburn, VA 20147
> (571)209-4000 x3159
> +
> 

--
Ethan A Merritt, Dept of Biochemistry
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742

Re: [ccp4bb] Basic Crystallography/Imaging Conundrum

[Ethan Merritt]

On Friday, 10 November 2017 00:10:22 Keller, Jacob wrote:
> Dear Crystallographers,
> 
> I have been considering a thought-experiment of sorts for a while, and wonder 
> what you will think about it:
> 
> Consider a diffraction data set which contains 62,500 unique reflections from 
> a 50 x 50 x 50 Angstrom unit cell, with each intensity measured perfectly 
> with 16-bit depth. (I am not sure what resolution this corresponds to, but it 
> would be quite high even in p1, I think--probably beyond 1.0 Angstrom?).

Meh. 
62500 is < 40^3, so ±20 indices on each axis.
50Å / 20 = 2.5Å,  so not quite 2.5Å resolution


> Thus, there are 62,500 x 16 bits (125 KB) of information in this alone, and 
> there is an HKL index associated with each intensity, so that I suppose 
> contains information as well. One could throw in phases at 16-bit as well, 
> and get a total of 250 KB for this dataset.
> 
> Now consider an parallel (equivalent?) data set, but this time instead of 
> reflection intensities you have a real space voxel map of the same 50 x 50 x 
> 50 unit cell consisting of 125,000 voxels, each of which has a 16-bit 
> electron density value, and an associated xyz index analogous to the hkl 
> above. That makes a total of 250 KB, with each voxel a 1 Angstrom cube. It 
> seems to me this level of graininess would be really hard to interpret, 
> especially for a static picture of a protein structure. (see attached: top is 
> a ~1 Ang/pixel down-sampled version of the image below).

All that proves is that assigning each 1x1x1 voxel a separate density value is 
a very
inefficient use of information.  Adjacent voxels are not independent, and no 
possible
assignment of values will get around the inherent blockiness of the 
representation.

I know!  Let's instead of assigning a magnitude per voxel, let's assign a 
magnitude per
something-resolution-sensitive, like a sin wave.   Then for each hkl 
measurement we get
one sin wave term.   Add up all the sine waves and what do you get?  Ta da.  A 
nice map.
 
> Or, if we wanted smaller voxels still, let's say by half, we would have to 
> reduce the bit depth to 2 bits. But this would still only yield half-Angstrom 
> voxels, each with only four possible electron density values.
> 
> Is this comparison apt? Off the cuff, I cannot see how a 50 x 50 pixel image 
> corresponds at all to the way our maps look, especially at around 1 Ang 
> resolution. Please, if you can shoot down the analogy, do.

Aren't Fourier series marvelous?

 
> Assuming that it is apt, however: is this a possible way to see the power of 
> all of our Bayesian modelling? Could one use our modelling tools on such a 
> grainy picture and arrive at similar results?
>
> Are our data sets really this poor in information, and we just model the heck 
> out of them, as perhaps evidenced by our scarily low data:parameters ratios?
> 
> My underlying motivation in this thought experiment is to illustrate the 
> richness in information (and poorness of modelling) that one achieves in 
> fluorescence microscopic imaging. If crystallography is any measure of the 
> power of modelling, one could really go to town on some of these terabyte 5D 
> functional data sets we see around here at Janelia (and on YouTube).
> 
> What do you think?
> 
> Jacob Keller
> 
> +
> Jacob Pearson Keller
> Research Scientist / Looger Lab
> HHMI Janelia Research Campus
> 19700 Helix Dr, Ashburn, VA 20147
> (571)209-4000 x3159
> +
> 

-- 
Ethan A Merritt, Dept of Biochemistry
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742