Re: [ccp4bb] happy/sad maps

[Sindelar, Charles]

A simple way to visualize what is happening is with UCSF ChimeraX. You can use 
the gaussian filter function to select either the frown or the smile from the 
“hidden_frown.map”, and then you can visualize the FFT structure factor 
magnitudes for each one – here are the commands to do it -

open hidden_frown.map
vol  gaussian #1 sdev 0.5
vol  gaussian #1 sdev 0.5 inv true
vol fourier #1
vol fourier #2
vol fourier #3

Fun stuff- your example is a great way of illuminating what is going on with 
filtering and fourier transforms!

Cheers, Chuck

From: CCP4 bulletin board  on behalf of James Holton 

Date: Sunday, April 30, 2023 at 5:16 PM
To: CCP4BB@JISCMAIL.AC.UK 
Subject: Re: [ccp4bb] happy/sad maps
Thank you Paul.  This is interesting!

I have not played with super-sampling yet.  I am assuming you mean
creating a new map 8x the size? If so, did you fill the interstitial
grid with zeroes? Local maximum? Linear interpolation?  Tricubic spline?

And when you say "sharpen/blur" with a factor of 4. Is 4 a scale factor?
or a B factor?

Cheers, and I hope you had a pleasant weekend,

-James

On 4/28/2023 5:29 PM, Paul Emsley wrote:
> What fun!
>
> super-sampled by a factor of 2 then sharpen/blur with a factor of 4
> gives a superposition
>
> Paul.
>



To unsubscribe from the CCP4BB list, click the following link:
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2FWA-JISC.exe%3FSUBED1%3DCCP4BB%26A%3D1=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cacedb11c93ac4900f10408db49c02e38%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638184862139535122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=lfAErsKbQtKLBx8Xmmn%2BC7zG%2Bt6N43ChGLcniZrSyps%3D=0

This message was issued to members of 
https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jiscmail.ac.uk%2FCCP4BB=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cacedb11c93ac4900f10408db49c02e38%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638184862139535122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Nn%2BNpEPMzHrExi7899MakiwAJ8IJ5ZdgMsRIbuXjfZg%3D=0,
 a mailing list hosted by 
https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jiscmail.ac.uk%2F=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cacedb11c93ac4900f10408db49c02e38%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638184862139535122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=AMn8ylqVwAO0sg%2BjlTi07Q20HVTD5ogHy2rQk1BPqfY%3D=0,
 terms & conditions are available at 
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fpolicyandsecurity%2F=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cacedb11c93ac4900f10408db49c02e38%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638184862139535122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=R1Iq6MMuDWk2p%2FHKtmr9lWF7Vuzk%2FrZLMn7IVuZA4tI%3D=0



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] happy/sad maps

[Sindelar, Charles]

Hi James!  I’m not sure exactly what cpp4, coot and phenix do for their FFT’s, 
but I’m assuming they must mask off the ‘corners’ of the 3D FFT data cube 
transform (where the frequency is greater than the Nyquist) during the forwards 
transform.

If you take your Cheshire map and do the FFT in matlab or octave, this is not 
done and I can confirm you get the smiley back. The cryo-EM software I am 
familiar with does filter away the FFT corners during the first 3D 
reconstruction step, so bypassing this issue.

Fun demo- this would be a great exercise for students!

Chuck


From: CCP4 bulletin board  on behalf of James Holton 

Date: Friday, April 28, 2023 at 11:49 AM
To: CCP4BB@JISCMAIL.AC.UK 
Subject: [ccp4bb] happy/sad maps
Its still April, but this one isn't a joke.

The smiley-face electron density in the left panel of the attached image
has the remarkable property that any attempt to sharpen or blur the map
turns it into the frowny-face on the right.  If you'd like to try this
yourself, the hidden_frown.map file is available in this tarball:
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbl831.als.lbl.gov%2F~jamesh%2Fbugreports%2Ffft_042423.tgz=05%7C01%7Ccharles.sindelar%40YALE.EDU%7Cc059178a4b774611f6ec08db48001b68%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638182937669477928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=jl16hGjaqmChVOHWb%2FAvgtvPTVFOt06wBqBYoxPk%2FZw%3D=0

In fact, any use of an FFT, even with the sharpening B set to zero,
turns the smiley into a frowny face. There is no way to get the smiley
face back (except opening the file again).  Yes, that's right, even just
a simple back-and-forth FFT: turning this hidden_frown.map into
structure factors and then back into a map again, gives you a frowny
face.  This happens using coot, ccp4 and phenix.

Wait, what!?  Isn't a Fourier transform supposed to preserve
information? As in: you can jump back and forth between real and
reciprocal space with impunity? Without introducing error?  Well, yes,
it is SUPPOSED to work like that, but the 3D FFT algorithms of
structural biology have a ... quirk. If you start with structure factors
and make a map out of them, you can convert it back-and-forth as often
as you want with 100% preservation of information.  However, if you
start with a real-space map (such as from cryoEM), a back-and-forth
conversion gives you a different map. This new map can then be
transformed back-and-forth all you want and be 100% preserved. It has
been "christened" by the FFT, but it is not the same as the starting
map, which is impossible to recover from the FFT-transformed data.
Information has been lost.  It is fine for crystallography (which starts
with structure factors), but for techniques such as cryoEM that start
with maps, using an FFT changes the data.

What information is being lost?  Sharp edges. These turn into ripples
covering all of real and reciprocal space. Do real-world data have sharp
edges?  Well, the all-or-nothing masks we use to model bulk solvent are
one example. Also, if you "mask off" otherwise smooth density with an
all-or-nothing mask, you will get similar ripples.  Another example of a
sharp edge might be the large changes between adjacent pixels you see
when a single electron hits a detector. For example, if you make a map
with just one non-zero voxel and run it back-and-forth through FFT you
will find that voxel loses from 50% to 99% of its value (depending on
the size of the map).  How much does this actually impact cryo-EM data?
That is my question.

What evil magic did I wield to make this map?  Well, I drew a smiley and
frowny face by hand, converted them to maps, and then I generated random
noise within the boundaries of the smiley face. I ran this noisy map
back-and-forth through FFT, and then subtracted the map that survives
the FFT from the pre-FFT map.  This cheshire_smile.map has the
interesting property that all of the structure factors calculated from
it are zero. It has an RMSD of 1.4, but after a back-and-forth FFT this
RMSD drops to 1e-7.  I generated the hidden_frown.map by simply summing
the frowny.map with cheshire_smile.map.

But isn't this map getting less noisy? Yes it is, but the interpretation
clearly changes as well.

Why does this happen?  It is because of a finite resolution cutoff. Oh!
What a relief! You don't have super-high resolution, do you? Well, no,
almost nobody has signal out beyond 1.0 A, but we do have noise.  In
diffraction data this noise is removed by simply not measuring it.  For
map data, however, the problem is that noise at very high frequencies
(small-number resolutions) is hard to avoid. This is because of another
phenomenon NMR spectroscopists are very familiar with: aliasing, or
"folding".  If any high-spatial frequency noise exists above half the
sampling rate (or "Nyquist frequency") it still gets