Thanks to all who replied both on- and off-list.
A few (such as Charles below) have reported that using general-purpose
FFT algorithms, such as those available in Matlab or NumPy do, in fact,
get the smiley face map back. This is expected because such algorithms
do not have a resolution cutoff. The programs I have found do turn the
smile into a frown are:
phenix.fft and phenix.map_to_structure_factors in Phenix
"fft", "sfall" and "refmac5" in the CCP4 Suite
The reason why this happens is not that these programs are "buggy", but
simply because they require the array of hkls in reciprocal space to be
a sphere, not a cube. That is, if h,k,l = 0,0,15 is at the resolution
limit, then h,k,l = 15,15,15 is going to be outside of it, and
discarded. The smiley face consists entirely of these high-order
structure factors. Rejecting these "corner" regions may seem like a
good idea until you realize that they are the only aspect of the map
that carries happiness. ;)
But all joking aside, the reason why these programs reject the "corners"
is because they "should" be zero. And as long as you only ever work
with maps that are generated from structure factors they will be zero.
If they are not zero, why? And what happens if you rotate the object
before recording it? Those non-zero "corners" will then be out beyond
the faces of the cube, and therefore folding back into the
lower-resolution data, messing it up. This implies, of course, that the
non-rotated object is also being messed up by noise beyond the Nyquist
limit. What I am suggesting is that non-zero "corner" regions may be a
canary in our coal mine.
Paul introduced the interesting possibility of super-sampling, and what
I'm really trying to get at here is: what is the best way to "feather"
an all-or-nothing real-space map (such as a bulk solvent mask, or a
single-pixel detector event), so that it does not spray noise all over
reciprocal space?
Many of you may think this is a trick question and that I already have a
beautiful answer I'm waiting to reveal. That is not really the case.
What I have is a handful of rather unsatisfactory solutions, and I'm
wondering if I'm just not aware of an existing solution to this problem.
Hope everyone had a lovely weekend,
-James Holton
MAD Scientist
On 4/28/2023 1:35 PM, Sindelar, Charles wrote:
Hi James! I’m not sure exactly what cpp4, coot and phenix do for
their FFT’s, but I’m assuming they must mask off the ‘corners’ of the
3D FFT data cube transform (where the frequency is greater than the
Nyquist) during the forwards transform.
If you take your Cheshire map and do the FFT in matlab or octave, this
is not done and I can confirm you get the smiley back. The cryo-EM
software I am familiar with does filter away the FFT corners during
the first 3D reconstruction step, so bypassing this issue.
Fun demo- this would be a great exercise for students!
Chuck
*From: *CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> on behalf of James
Holton <jmhol...@lbl.gov>
*Date: *Friday, April 28, 2023 at 11:49 AM
*To: *CCP4BB@JISCMAIL.AC.UK <CCP4BB@JISCMAIL.AC.UK>
*Subject: *[ccp4bb] happy/sad maps
Its still April, but this one isn't a joke.
The smiley-face electron density in the left panel of the attached image
has the remarkable property that any attempt to sharpen or blur the map
turns it into the frowny-face on the right. If you'd like to try this
yourself, the hidden_frown.map file is available in this tarball:
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbl831.als.lbl.gov%2F~jamesh%2Fbugreports%2Ffft_042423.tgz&data=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jl16hGjaqmChVOHWb%2FAvgtvPTVFOt06wBqBYoxPk%2FZw%3D&reserved=0
<https://bl831.als.lbl.gov/~jamesh/bugreports/fft_042423.tgz>
In fact, any use of an FFT, even with the sharpening B set to zero,
turns the smiley into a frowny face. There is no way to get the smiley
face back (except opening the file again). Yes, that's right, even just
a simple back-and-forth FFT: turning this hidden_frown.map into
structure factors and then back into a map again, gives you a frowny
face. This happens using coot, ccp4 and phenix.
Wait, what!? Isn't a Fourier transform supposed to preserve
information? As in: you can jump back and forth between real and
reciprocal space with impunity? Without introducing error? Well, yes,
it is SUPPOSED to work like that, but the 3D FFT algorithms of
structural biology have a ... quirk. If you start with structure factors
and make a map out of them, you can convert it back-and-forth as often
as you want with 100% preservation of information. However, if you
start with a real-space map (such as from cryoEM), a back-and-forth
conversion gives you a different map. This new map can then be
transformed back-and-forth all you want and be 100% preserved. It has
been "christened" by the FFT, but it is not the same as the starting
map, which is impossible to recover from the FFT-transformed data.
Information has been lost. It is fine for crystallography (which starts
with structure factors), but for techniques such as cryoEM that start
with maps, using an FFT changes the data.
What information is being lost? Sharp edges. These turn into ripples
covering all of real and reciprocal space. Do real-world data have sharp
edges? Well, the all-or-nothing masks we use to model bulk solvent are
one example. Also, if you "mask off" otherwise smooth density with an
all-or-nothing mask, you will get similar ripples. Another example of a
sharp edge might be the large changes between adjacent pixels you see
when a single electron hits a detector. For example, if you make a map
with just one non-zero voxel and run it back-and-forth through FFT you
will find that voxel loses from 50% to 99% of its value (depending on
the size of the map). How much does this actually impact cryo-EM data?
That is my question.
What evil magic did I wield to make this map? Well, I drew a smiley and
frowny face by hand, converted them to maps, and then I generated random
noise within the boundaries of the smiley face. I ran this noisy map
back-and-forth through FFT, and then subtracted the map that survives
the FFT from the pre-FFT map. This cheshire_smile.map has the
interesting property that all of the structure factors calculated from
it are zero. It has an RMSD of 1.4, but after a back-and-forth FFT this
RMSD drops to 1e-7. I generated the hidden_frown.map by simply summing
the frowny.map with cheshire_smile.map.
But isn't this map getting less noisy? Yes it is, but the interpretation
clearly changes as well.
Why does this happen? It is because of a finite resolution cutoff. Oh!
What a relief! You don't have super-high resolution, do you? Well, no,
almost nobody has signal out beyond 1.0 A, but we do have noise. In
diffraction data this noise is removed by simply not measuring it. For
map data, however, the problem is that noise at very high frequencies
(small-number resolutions) is hard to avoid. This is because of another
phenomenon NMR spectroscopists are very familiar with: aliasing, or
"folding". If any high-spatial frequency noise exists above half the
sampling rate (or "Nyquist frequency") it still gets recorded, but shows
up in a lower-frequency Fourier coefficient. It is not possible to
remove such aliasing noise after digitization. Upon discretization of
the signal (FFT or no) all these high frequencies join with lower
frequency terms, and so survive any low-pass filtering. Darn.
Why am I bringing this up? Because if there is noise out beyond the FFT
resolution limit it implies there is also noise out beyond the
Nyquist-Shannon limit as well. If that is the case, direct-space imaging
data may be a lot noisier than it needs to be. In general, in digital
signal processing of things like sound an analog low-pass filter is
always installed at the input of any digitizer. Perhaps this is why
de-focusing works better than being at focus?
What is the solution? Well, for things like the bulk solvent mask I'd
say some real-space "feathering" is called for before performing FFTs.
Same goes for masked density like that used to compute CCmask. It may
also be worth looking into the digitization process of "image first"
structural biology methods?
My question for the BB: can someone explain how Nyquist folding is
handled in cryoEM data processing?
-James Holton
MAD Scientist
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2FWA-JISC.exe%3FSUBED1%3DCCP4BB%26A%3D1&data=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7RJgG1Qghql40Hd6oTuel%2BBQqQiMC3%2FrKNPtho0Rh%2B0%3D&reserved=0
<https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1>
This message was issued to members of
https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jiscmail.ac.uk%2FCCP4BB&data=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RZ3luGFHMgtjYpgSvMRsp%2BQ8ikfw8l%2FX0w0g1ynGvfg%3D&reserved=0
<http://www.jiscmail.ac.uk/CCP4BB>, a mailing list hosted by
https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jiscmail.ac.uk%2F&data=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RsdmQsUbpHz3nzyA2%2BqSw2r6baxoBxSXhSwS2sPMvZQ%3D&reserved=0
<http://www.jiscmail.ac.uk/>, terms & conditions are available at
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fpolicyandsecurity%2F&data=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vYWKPUG8%2B2JFmNx5Ry4aOVhpCXGbPnufwZJ7OiobMMw%3D&reserved=0
<https://www.jiscmail.ac.uk/policyandsecurity/>
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
