Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
If there are no other noise sources, then then final signal to noise of measuring the photons you describe is always EXACTLY the same. This is why photon-counting is such a useful error currency: doesn't matter how you slice them up or lump them together. Photons are photons, and the square root of their count is the error. Think of a detector that not only counts them one at a time, but stores them in individual files. You get a lot of files, but if you add them all together into one or into a million the signal-to-noise is the same. In reality, however, there is ALWAYS another source of error, and if that error changes up with each acquisition, then yes, you do average over the extra error with multiplicity. Examples of such error are shutter jitter, beam flicker, sample vibration, and read-out noise. Detector calibration is also one of these errors, provided you never use the same pixel twice. That's the nature of systematic errors, you can turn them into random errors if you can find a way to keep changing their source. But if you use the same pixels over and over again to measure the same thing you might be only fooling yourself into thinking you are reducing your total error. The number of photons sets a lower limit on the total error. You can't do anything about that. Profile fitting allows you to reduce the error incurred from not clearly knowing the boundary of a spot, but there is no way to get around shot noise (aka photon counting error). But yes, in reality multiplicity is definitely your friend. The trick is making it true multiplicity, where all sources of error have been changed up. Personally, since there is so much contention about using the term multiplicity or redundancy, I think it should be called multiplicity when you are actually averaging over errors, but redundancy when you are not. -James Holton MAD Scientist On 1/15/2015 4:14 PM, Keller, Jacob wrote: I think a summary is that: Background levels and errors thereof can be estimated very precisely as a percentage of the level, but as an absolute number, it nevertheless swamps out the signal contained in a Bragg peak. For example, if background = 10^6 photons +/- 1000 (very good estimate as a percentage--0.1%), that still adds an absolute number of +/- 1000 photons to a Bragg peak which might represent 1000 photons or so, so this noise is a big problem. One would, however, gain a lot by having many pixels in each spot and profile-fitting them, as you mentioned, as this would improve sampling and reduce error. What about this comparison, though: either measure a photon count of 10^6 once on a background of 10^7 or measure the same reflection 1000 times independently at 10^3 photons on a background of 10^4. Assuming no readout noise or other noise source, wouldn't the latter obviously be better? In a sense, each pixel in the Bragg peak in the first case would really be 1000 pixels in the second, and the noise from the background could be cancelled much more effectively? Admittedly this is an orthogonal issue to the background subtraction one, since obviously as you demonstrated the background makes I/sig worse, but this does make background subtraction markedly better, maybe even enough to warrant always erring on the side of too much non-crystal stuff? And definitely always arguing to measure data at low intensities multiple times rather than once at high intensity, for a given total dose! Jacob -Original Message- From: James Holton [mailto:jmhol...@lbl.gov] Sent: Thursday, January 15, 2015 12:00 PM To: Keller, Jacob;CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?) Jacob, Background subtraction is unfortunately not as forgiving as you think it is. You can subtract the background, but you can't subtract the noise. This is because noise (by definition) is never the same twice. Yes, the average or true background under a spot may be flat, but any given observation of it will be noisy, and there is no way to separate the noise that came from the Bragg-scattered photons from the background-scattered photons that hit the same pixel. Each photon is an independent event, after all. Consider an example: if you see 4 photons in an isolated spot after 1 second and there is zero background then sigma(I) = sqrt(I) = sqrt(4) = 2, and your I/sigma is most likely 2. I write most likely because the true photon arrival rate (the thing that is proportional to F^2) doesn't have to be 4 photons/s just because you counted four in one observation. The long-term average could easily be something else, such as 3 photons/s, 5 photons/s or even 3.2 photons/s (on average). Observing 4 photons is not unlikely in all these scenarios. However, if you consider all possible true rates, simulate millions of trials and isolate all the instances where you counted 4 photons you will find
Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
10 micron crystal is embedded in a block of vitrified water, for the same dose to the crystal (photons/area) you will still get 16 Bragg photons but 192 background photons in the spot area pixels, reducing your I/sigma to 1.1. Most people would rather not do that. So, yes, smaller beams are better if your crystal is actually small, and it is surrounded by stuff of similar thickness, density and elemental composition. You also need to bear in mind the stuff that is in the beam path before and after your crystal because this is part of the illuminated volume too. Ideally, you want your crystal sticking out into the air, then any beam size is more or less equivalent because air/N2 scatters 1000x less than the solid stuff in the loop. The only problem with that is a lot of crystals hate surface tension. This is why I recommend having surrounding stuff, but no too much. A factor of 2 in volume is a good compromise. Yes, there is such a thing as big crystals with a sweet spot that can only be accessed with a small beam and there is definitely a lot of excitement about that. What I keep wondering is: what went wrong in the rest of that crystal? Brighter beams are better for getting your experiment over more quickly, but if you're attenuating then those extra photons are going to waste. Faster detectors can help with this, but if they are too fast they will start picking up high-frequency noise in the beam. This is a problem for anomalous, but not for resolution. 5% error when you are only counting 4 photons is a minor consideration. Background is the primary enemy of resolution. After disorder, of course! As for where to put resources, I try not to think about which aspect of structural biology should be cut in favor of some other part because I think structural biology as a whole is important! Especially when you consider how much money is wasted on insert random political statement here. Detectors can always be better, but at the moment low-end noise levels and speed are not limiting factors. The challenges are either 1) detecting weak spots (aka resolution) or 2) accurately measuring small differences between strong spots (aka anomalous differences). 1) is limited by pixel count and 2) by calibration. I say pixel count because larger active areas are always better for background reduction (inverse square law), but only if your spots take up more than a few pixels. If your spots are all smaller than a pixel then your pixels are too big. Colin Nave (JSR, 2014) has calculated that the ideal MX detector would have about 1e9 pixels in it. Only problem with that is the going rate for a pixel these days is ~$0.25 each. For anomalous, the biggest problem with detectors is calibration, which is a lot harder to deal with than you might think. The best evidence of this fact is that if you simulate data with every kind of noise you can think of you still get low-resolution R-meas values of ~0.5% (Holton et al, FEBS 2014, Diederichs, 2009). I have never seen a real dataset like that. Nevertheless, if you count 1,000,000 photons, the sigma of that count is 1000, or 0.1% error. Something else is getting in the way. Unfortunate really, because if we could routinely get R-meas = 0.1% we would never need to make metal derivatives again. -James Holton MAD Scientist On 1/8/2015 9:47 AM, Keller, Jacob wrote: Yes, this is great info and thoughts. What I still do not understand, however, is why the noise from air/loop scattering is so bad--why not make sure only the top of the Gaussian is engulfing the crystal, and the tails can hit air or loop? Isn't the air scattering noise easily subtractable, being essentially flat over time, whereas uneven illumination of the crystal is highly difficult to correct? Also, in light of these considerations, it would seem to me a much better use of resources not to make brighter and smaller beams but instead concentrate on making better low-intensity big beam profiles (top-hats?) and lower-noise, faster detectors (like Pilatus and the new ADSC). Jacob -Original Message- From: James Holton [mailto:jmhol...@lbl.gov] Sent: Tuesday, December 30, 2014 3:57 PM To: Keller, Jacob;CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] How far does rad dam travel? Yes, bigger is okay, and perhaps a little better if you consider the effects of beam/crystal vibration and two sharp-edged boundaries dancing over each other. But bigger is better only to a point. That point is when the illuminated area of non-good-stuff is about equal to the area of the good stuff. This is because the total background noise is equal to the square root of the number of photons and equal volumes of any given stuff (good or non-good) yield about the same number of background-scattered photons. So, since you're taking the square root, completely eliminating the non-good-stuff only buys you a gain of 40% in total noise. Given
Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
Dear Jacob, To add one more thing to the many ones from James, despite air scattering noise being flat and easily subtractable over time, that noise wont be the same if you take images with and without the sample. Since some of the photons will be absorbed and diffracted by the crystal, the air noise will be different from what you recorded without the sample (with or without the loop/stuff) and you would like to subtract. D -Original Message- From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of James Holton Sent: 15 January 2015 17:00 To: ccp4bb Subject: Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?) Jacob, Background subtraction is unfortunately not as forgiving as you think it is. You can subtract the background, but you can't subtract the noise. This is because noise (by definition) is never the same twice. Yes, the average or true background under a spot may be flat, but any given observation of it will be noisy, and there is no way to separate the noise that came from the Bragg-scattered photons from the background-scattered photons that hit the same pixel. Each photon is an independent event, after all. Consider an example: if you see 4 photons in an isolated spot after 1 second and there is zero background then sigma(I) = sqrt(I) = sqrt(4) = 2, and your I/sigma is most likely 2. I write most likely because the true photon arrival rate (the thing that is proportional to F^2) doesn't have to be 4 photons/s just because you counted four in one observation. The long-term average could easily be something else, such as 3 photons/s, 5 photons/s or even 3.2 photons/s (on average). Observing 4 photons is not unlikely in all these scenarios. However, if you consider all possible true rates, simulate millions of trials and isolate all the instances where you counted 4 photons you will find that the true rate of 4.0 photons/s turns up more often than any other, so that's your best guess. Nevertheless, if your true rate really is 4.0 photons/s, then the probability of actually seeing 4 photons is only ~20%! The other 80% of the time you will see something else. 20% of the time you will count 3, there is a 15% chance you will only see 2, and a ~2% chance you will see zero. There is also a 2% chance of seeing 11 photons in 1 second when the true rate is really 4. You just don't know. Of course, with more observations you can narrow it down. If you do 100 trials and see an average of 4.0, then you are obviously a lot more confident in the true rate of 4 photons/s. But this isn't observing 4 photons, this is observing 400 photons and dividing that number by 100. Because the error in counting 400 photons is sqrt(400)=20, your signal-to-noise is 400/sqrt(400) = 20! This is why multiplicity is a good thing. However, if all you have is one sample of 4 photons your best guess is I = 4 and sigma(I) = 2. Now consider the case where there is background. Say the true rate for the background is 10 photons/pixel/s, and for simplicity lets say your 4 photon/s spot lies entirely within one pixel. What is your signal-to-noise now? Well, if you take 100 pixels in the neighborhood around your spot you will count ~1000 photons, giving you an excellent estimate of the true background rate: 10 photons/pixel/s with a sigma of sqrt(1000)/100 = 0.3, or a signal-to-noise of 31.6 for the estimate of the background level. Looking at the one pixel containing your spot, let's say you saw 14 photons in it, that means you have a sigma of 2 from the 4 spot photons and a sigma of 0.3 from the background photons for a total sigma of sqrt(2^2+0.3^2) = 2.02 and I/sigma = 1.97, right? Wrong. The reality of the situation is the 14 photons that landed in your spot pixel were independent events from all the photons that landed in the nearby background pixels. Yes, you know that there should be 10, but the probability of actually getting 10 exactly is only 12.4%. You are just as likely to see 11 or 9, and there is a 5% chance of 14 background photons hitting the spot area. In that case the Bragg intensity could easily be zero, and the 14 photons you saw were just a random fluctuation from the average of 10. You just don't know! And your error bars should reflect that. The correct value for sigma(I) turns out to be the square root of the TOTAL number of photons that hit the spot area: sqrt(14) = 3.7 and the I/sigma of your 4-photon spot is now 1.07. With 100 background photons/pixel, your I/sigma = 0.4. This is how background degrades your resolution limit. The bright side of it is that the degradation of I/sigma rises only with the square root of the background level, not the background level itself. As a general rule: it takes 3x the Bragg photons to cut I/sigma in half, 15x the Bragg photons to cut it to 1/4, etc. So, a factor of 2 in background is at worst a 40% hit in I/sigma
Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
-Original Message- From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Dom Bellini Sent: Thursday, January 15, 2015 12:25 PM To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?) Dear Jacob, To add one more thing to the many ones from James, despite air scattering noise being flat and easily subtractable over time, that noise wont be the same if you take images with and without the sample. Since some of the photons will be absorbed and diffracted by the crystal, the air noise will be different from what you recorded without the sample (with or without the loop/stuff) and you would like to subtract. Right--I meant to extrapolate the background from non-spot pixels on the detector. I am still processing James's message, however (always a good idea to consider them well). BTW, I should add that many of these statistical discussions on CCP4BB have informed a lot of (all?) other research I have been doing, so thanks very much everyone for the enlightenment! JPK
Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
I think a summary is that: Background levels and errors thereof can be estimated very precisely as a percentage of the level, but as an absolute number, it nevertheless swamps out the signal contained in a Bragg peak. For example, if background = 10^6 photons +/- 1000 (very good estimate as a percentage--0.1%), that still adds an absolute number of +/- 1000 photons to a Bragg peak which might represent 1000 photons or so, so this noise is a big problem. One would, however, gain a lot by having many pixels in each spot and profile-fitting them, as you mentioned, as this would improve sampling and reduce error. What about this comparison, though: either measure a photon count of 10^6 once on a background of 10^7 or measure the same reflection 1000 times independently at 10^3 photons on a background of 10^4. Assuming no readout noise or other noise source, wouldn't the latter obviously be better? In a sense, each pixel in the Bragg peak in the first case would really be 1000 pixels in the second, and the noise from the background could be cancelled much more effectively? Admittedly this is an orthogonal issue to the background subtraction one, since obviously as you demonstrated the background makes I/sig worse, but this does make background subtraction markedly better, maybe even enough to warrant always erring on the side of too much non-crystal stuff? And definitely always arguing to measure data at low intensities multiple times rather than once at high intensity, for a given total dose! Jacob -Original Message- From: James Holton [mailto:jmhol...@lbl.gov] Sent: Thursday, January 15, 2015 12:00 PM To: Keller, Jacob; CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?) Jacob, Background subtraction is unfortunately not as forgiving as you think it is. You can subtract the background, but you can't subtract the noise. This is because noise (by definition) is never the same twice. Yes, the average or true background under a spot may be flat, but any given observation of it will be noisy, and there is no way to separate the noise that came from the Bragg-scattered photons from the background-scattered photons that hit the same pixel. Each photon is an independent event, after all. Consider an example: if you see 4 photons in an isolated spot after 1 second and there is zero background then sigma(I) = sqrt(I) = sqrt(4) = 2, and your I/sigma is most likely 2. I write most likely because the true photon arrival rate (the thing that is proportional to F^2) doesn't have to be 4 photons/s just because you counted four in one observation. The long-term average could easily be something else, such as 3 photons/s, 5 photons/s or even 3.2 photons/s (on average). Observing 4 photons is not unlikely in all these scenarios. However, if you consider all possible true rates, simulate millions of trials and isolate all the instances where you counted 4 photons you will find that the true rate of 4.0 photons/s turns up more often than any other, so that's your best guess. Nevertheless, if your true rate really is 4.0 photons/s, then the probability of actually seeing 4 photons is only ~20%! The other 80% of the time you will see something else. 20% of the time you will count 3, there is a 15% chance you will only see 2, and a ~2% chance you will see zero. There is also a 2% chance of seeing 11 photons in 1 second when the true rate is really 4. You just don't know. Of course, with more observations you can narrow it down. If you do 100 trials and see an average of 4.0, then you are obviously a lot more confident in the true rate of 4 photons/s. But this isn't observing 4 photons, this is observing 400 photons and dividing that number by 100. Because the error in counting 400 photons is sqrt(400)=20, your signal-to-noise is 400/sqrt(400) = 20! This is why multiplicity is a good thing. However, if all you have is one sample of 4 photons your best guess is I = 4 and sigma(I) = 2. Now consider the case where there is background. Say the true rate for the background is 10 photons/pixel/s, and for simplicity lets say your 4 photon/s spot lies entirely within one pixel. What is your signal-to-noise now? Well, if you take 100 pixels in the neighborhood around your spot you will count ~1000 photons, giving you an excellent estimate of the true background rate: 10 photons/pixel/s with a sigma of sqrt(1000)/100 = 0.3, or a signal-to-noise of 31.6 for the estimate of the background level. Looking at the one pixel containing your spot, let's say you saw 14 photons in it, that means you have a sigma of 2 from the 4 spot photons and a sigma of 0.3 from the background photons for a total sigma of sqrt(2^2+0.3^2) = 2.02 and I/sigma = 1.97, right? Wrong. The reality of the situation is the 14 photons that landed in your spot pixel were independent events from all
Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Jacob, both George Sheldrick and Andrew Leslie explained to me that the machine I had in mind - a sealed tube generator with a graphite monochromator - is not really state of the art and merely a technology from 20 years ago. Hence my comment about the top hat profile of inhouse machines adding to the high quality data they produce was inappropriate. Modern inhouse machines usually don't show a top hat profile. The quote from Bruker I referred to addressed a project to check crystals before collecting neutron data, where such a machine is indeed appropriate. However, most of us hardly ever see crystals with a volume in the mm^3 region. Sorry if I caused any confusion - I felt I should set this straight for everyone to know. Cheers, Tim On 01/12/2015 11:32 PM, Keller, Jacob wrote: at the beginning of my experience of S-SAD about 10 years ago, it was not too difficult to do S-SAD phasing with inhouse data provided the resolution was better than 2.0A, while it did not always work with synchrotron data. Purely personal experience. I assume that the synchrotron data were collected at similarly-low energy? However, the inhouse machines I am familiar with have three circles, so that you get much better real redundancy with equivalent reflections recorded at different settings. This reduces systematic errors, I think. The most sophisticated synchrotron beamline I have been to offered a mini-kappa with 30degree range - that's not much compared to 10-20 different settings with varying phi- omega- and distance settings. Yes, I haven't seen much about people collecting multiple orientations of the same crystal, since I think people generally roast their crystals really fast to see higher-resolution spots. I am thinking recently that the best option might really be home sources with pixel-array detectors... The top-hat comes from a quote I received from Bruker, and I have no reason to believe the person acted purely with a salesperson's intent. Pretty interesting--wonder what's the best way to confirm this for our home source...? JPK Best, Tim On 01/12/2015 09:05 PM, Keller, Jacob wrote: the top-hat profile is one of the reasons why inhouse machines produce better quality data than synchrotrons. However, the often much increased resolution you achieve at the synchrotron is generally worth more than the quality of the data at restricted resolution. Cheers, Tim Several surprises to me: -Data from in-house sources is better? I have not heard of this--is there any systematic examination of this? I saw nothing about this in a very brief Google foray. -In-house beam profiles are top-hats? Is there a place which shows such measurements? Does not pop out of Google for me, but I would love to be shown that this is true. -Resolution at the synchrotron is better? This does not really seem right to me theoretically, although in practice it does seem to happen. I think it is just a question of waiting for enough exposure time, as the CCP4BB response quoted at bottom describes. JPK === Date: Tue, 12 Oct 2010 09:04:05 -0700 From: James Holton jmhol...@lbl.gov Re: Re: Lousy diffraction at home but fantastic at the synchrotron? There are a few things that synchrotron beamlines generally do better than home sources, but the most important are flux, collimation and absorption. Flux is in photons/s and simply scales down the amount of time it takes to get a given amount of photons onto the crystal. Contrary to popular belief, there is nothing magical about having more photons/s: it does not somehow make your protein molecules behave and line up in a more ordered way. However, it does allow you to do the equivalent of a 24-hour exposure in a few seconds (depending on which beamline and which home source you are comparing), so it can be hard to get your brain around the comparison. Collimation, in a nutshell, is putting all the incident photons through the crystal, preferably in a straight line. Illuminating anything that isn't the crystal generates background, and background buries weak diffraction spots (also known as high-resolution spots). Now, when I say crystal I mean the thing you want to shoot, so this includes the best part of a bent, cracked or otherwise inhomogeneous crystal. The amount of background goes as the square of the beam size, so a 0.5 mm beam can produce up to 25 times more background than a 0.1 mm beam (for a fixed spot intensity). Also, if the beam has high divergence (the range of incidence angles onto the crystal), then the spots on the detector will be more spread out than if the beam had low divergence, and the more spread-out the spots are the easier it is for them to fade into the background. Now, even at home sources, one can cut down the beam to have very low divergence and a very small size at the sample
Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
Hi Tim, Jacob, I must admit that I was very surprised by the suggestion of a top-hat profile for a in-house rotating anode. We have a Rigaku Fr-E generator, and Rigaku provided a plot of the beam profile for that (with VariMax-HR Optic) and it is very far from being top hat, much more Gaussian-like, which really is what I would have expected for this type of source and optic. Without significantly truncating the full profile (i.e. by selecting the very central part of a Gaussian, which results in a significant loss of flux) I don't know how they would achieve a top hat profile, but perhaps someone from Bruker could respond to this ? I guess my point is that certainly not all in house generators provide a beam with a top hat profile. Best wishes, Andrew On 12 Jan 2015, at 21:38, Tim Gruene t...@shelx.uni-ac.gwdg.de wrote: Hi Jacob, at the beginning of my experience of S-SAD about 10 years ago, it was not too difficult to do S-SAD phasing with inhouse data provided the resolution was better than 2.0A, while it did not always work with synchrotron data. Purely personal experience. However, the inhouse machines I am familiar with have three circles, so that you get much better real redundancy with equivalent reflections recorded at different settings. This reduces systematic errors, I think. The most sophisticated synchrotron beamline I have been to offered a mini-kappa with 30degree range - that's not much compared to 10-20 different settings with varying phi- omega- and distance settings. The top-hat comes from a quote I received from Bruker, and I have no reason to believe the person acted purely with a salesperson's intent. Best, Tim On 01/12/2015 09:05 PM, Keller, Jacob wrote: the top-hat profile is one of the reasons why inhouse machines produce better quality data than synchrotrons. However, the often much increased resolution you achieve at the synchrotron is generally worth more than the quality of the data at restricted resolution. Cheers, Tim Several surprises to me: -Data from in-house sources is better? I have not heard of this--is there any systematic examination of this? I saw nothing about this in a very brief Google foray. -In-house beam profiles are top-hats? Is there a place which shows such measurements? Does not pop out of Google for me, but I would love to be shown that this is true. -Resolution at the synchrotron is better? This does not really seem right to me theoretically, although in practice it does seem to happen. I think it is just a question of waiting for enough exposure time, as the CCP4BB response quoted at bottom describes. JPK === Date: Tue, 12 Oct 2010 09:04:05 -0700 From: James Holton jmhol...@lbl.gov Re: Re: Lousy diffraction at home but fantastic at the synchrotron? There are a few things that synchrotron beamlines generally do better than home sources, but the most important are flux, collimation and absorption. Flux is in photons/s and simply scales down the amount of time it takes to get a given amount of photons onto the crystal. Contrary to popular belief, there is nothing magical about having more photons/s: it does not somehow make your protein molecules behave and line up in a more ordered way. However, it does allow you to do the equivalent of a 24-hour exposure in a few seconds (depending on which beamline and which home source you are comparing), so it can be hard to get your brain around the comparison. Collimation, in a nutshell, is putting all the incident photons through the crystal, preferably in a straight line. Illuminating anything that isn't the crystal generates background, and background buries weak diffraction spots (also known as high-resolution spots). Now, when I say crystal I mean the thing you want to shoot, so this includes the best part of a bent, cracked or otherwise inhomogeneous crystal. The amount of background goes as the square of the beam size, so a 0.5 mm beam can produce up to 25 times more background than a 0.1 mm beam (for a fixed spot intensity). Also, if the beam has high divergence (the range of incidence angles onto the crystal), then the spots on the detector will be more spread out than if the beam had low divergence, and the more spread-out the spots are the easier it is for them to fade into the background. Now, even at home sources, one can cut down the beam to have very low divergence and a very small size at the sample position, but this comes at the expense of flux. Another tenant of collimation (in my book) is the DEPTH of non-crystal stuff in the primary x-ray beam that can be seen by the detector. This includes the air space between the collimator and the beam stop. One millimeter of air generates about as much background as 1 micron of crystal, water, or plastic. Some home sources have ridiculously large air
[ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
the top-hat profile is one of the reasons why inhouse machines produce better quality data than synchrotrons. However, the often much increased resolution you achieve at the synchrotron is generally worth more than the quality of the data at restricted resolution. Cheers, Tim Several surprises to me: -Data from in-house sources is better? I have not heard of this--is there any systematic examination of this? I saw nothing about this in a very brief Google foray. -In-house beam profiles are top-hats? Is there a place which shows such measurements? Does not pop out of Google for me, but I would love to be shown that this is true. -Resolution at the synchrotron is better? This does not really seem right to me theoretically, although in practice it does seem to happen. I think it is just a question of waiting for enough exposure time, as the CCP4BB response quoted at bottom describes. JPK === Date: Tue, 12 Oct 2010 09:04:05 -0700 From: James Holton jmhol...@lbl.gov Re: Re: Lousy diffraction at home but fantastic at the synchrotron? There are a few things that synchrotron beamlines generally do better than home sources, but the most important are flux, collimation and absorption. Flux is in photons/s and simply scales down the amount of time it takes to get a given amount of photons onto the crystal. Contrary to popular belief, there is nothing magical about having more photons/s: it does not somehow make your protein molecules behave and line up in a more ordered way. However, it does allow you to do the equivalent of a 24-hour exposure in a few seconds (depending on which beamline and which home source you are comparing), so it can be hard to get your brain around the comparison. Collimation, in a nutshell, is putting all the incident photons through the crystal, preferably in a straight line. Illuminating anything that isn't the crystal generates background, and background buries weak diffraction spots (also known as high-resolution spots). Now, when I say crystal I mean the thing you want to shoot, so this includes the best part of a bent, cracked or otherwise inhomogeneous crystal. The amount of background goes as the square of the beam size, so a 0.5 mm beam can produce up to 25 times more background than a 0.1 mm beam (for a fixed spot intensity). Also, if the beam has high divergence (the range of incidence angles onto the crystal), then the spots on the detector will be more spread out than if the beam had low divergence, and the more spread-out the spots are the easier it is for them to fade into the background. Now, even at home sources, one can cut down the beam to have very low divergence and a very small size at the sample position, but this comes at the expense of flux. Another tenant of collimation (in my book) is the DEPTH of non-crystal stuff in the primary x-ray beam that can be seen by the detector. This includes the air space between the collimator and the beam stop. One millimeter of air generates about as much background as 1 micron of crystal, water, or plastic. Some home sources have ridiculously large air paths (like putting the backstop on the detector surface), and that can give you a lot of background. As a rule of thumb, you want you air path in mm to be less than or equal to your crystal size in microns. In this situation, the crystal itself is generating at least as much background as the air, and so further reducing the air path has diminishing returns. For example, going from 100 mm air and 100 um crystal to completely eliminating air will only get you about a 40% reduction in background noise (it goes as the square root). Now, this rule of thumb also goes for the support material around your crystal: one micron of cryoprotectant generates about as much background as one micron of crystal. So, if you have a 10 micron crystal mounted in a 1 mm thick drop, and manage to hit the crystal with a 10 micron beam, you still have 100 times more background coming from the drop than you do from the crystal. This is why in-situ diffraction is so difficult: it is hard to come by a crystal tray that is the same thickness as the crystals. Absorption differences between home and beamline are generally because beamlines operate at around 1 A, where a 200 um thick crystal or a 200 mm air path absorbs only about 4% of the x-rays, and home sources generally operate at CuKa, where the same amount of crystal or air absorbs ~20%. The absorption correction due to different paths taken through the sample must always be less than the total absorption, so you can imagine the relative difficulty of trying to measure a ~3% anomalous difference. Lower absorption also accentuates the benefits of putting the detector further away. By the way, there IS a good reason why we spend so much money on large-area detectors. Background falls off with the square of distance, but the spots don't (assuming good
Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
Hi Jacob, at the beginning of my experience of S-SAD about 10 years ago, it was not too difficult to do S-SAD phasing with inhouse data provided the resolution was better than 2.0A, while it did not always work with synchrotron data. Purely personal experience. However, the inhouse machines I am familiar with have three circles, so that you get much better real redundancy with equivalent reflections recorded at different settings. This reduces systematic errors, I think. The most sophisticated synchrotron beamline I have been to offered a mini-kappa with 30degree range - that's not much compared to 10-20 different settings with varying phi- omega- and distance settings. The top-hat comes from a quote I received from Bruker, and I have no reason to believe the person acted purely with a salesperson's intent. Best, Tim On 01/12/2015 09:05 PM, Keller, Jacob wrote: the top-hat profile is one of the reasons why inhouse machines produce better quality data than synchrotrons. However, the often much increased resolution you achieve at the synchrotron is generally worth more than the quality of the data at restricted resolution. Cheers, Tim Several surprises to me: -Data from in-house sources is better? I have not heard of this--is there any systematic examination of this? I saw nothing about this in a very brief Google foray. -In-house beam profiles are top-hats? Is there a place which shows such measurements? Does not pop out of Google for me, but I would love to be shown that this is true. -Resolution at the synchrotron is better? This does not really seem right to me theoretically, although in practice it does seem to happen. I think it is just a question of waiting for enough exposure time, as the CCP4BB response quoted at bottom describes. JPK === Date: Tue, 12 Oct 2010 09:04:05 -0700 From: James Holton jmhol...@lbl.gov Re: Re: Lousy diffraction at home but fantastic at the synchrotron? There are a few things that synchrotron beamlines generally do better than home sources, but the most important are flux, collimation and absorption. Flux is in photons/s and simply scales down the amount of time it takes to get a given amount of photons onto the crystal. Contrary to popular belief, there is nothing magical about having more photons/s: it does not somehow make your protein molecules behave and line up in a more ordered way. However, it does allow you to do the equivalent of a 24-hour exposure in a few seconds (depending on which beamline and which home source you are comparing), so it can be hard to get your brain around the comparison. Collimation, in a nutshell, is putting all the incident photons through the crystal, preferably in a straight line. Illuminating anything that isn't the crystal generates background, and background buries weak diffraction spots (also known as high-resolution spots). Now, when I say crystal I mean the thing you want to shoot, so this includes the best part of a bent, cracked or otherwise inhomogeneous crystal. The amount of background goes as the square of the beam size, so a 0.5 mm beam can produce up to 25 times more background than a 0.1 mm beam (for a fixed spot intensity). Also, if the beam has high divergence (the range of incidence angles onto the crystal), then the spots on the detector will be more spread out than if the beam had low divergence, and the more spread-out the spots are the easier it is for them to fade into the background. Now, even at home sources, one can cut down the beam to have very low divergence and a very small size at the sample position, but this comes at the expense of flux. Another tenant of collimation (in my book) is the DEPTH of non-crystal stuff in the primary x-ray beam that can be seen by the detector. This includes the air space between the collimator and the beam stop. One millimeter of air generates about as much background as 1 micron of crystal, water, or plastic. Some home sources have ridiculously large air paths (like putting the backstop on the detector surface), and that can give you a lot of background. As a rule of thumb, you want you air path in mm to be less than or equal to your crystal size in microns. In this situation, the crystal itself is generating at least as much background as the air, and so further reducing the air path has diminishing returns. For example, going from 100 mm air and 100 um crystal to completely eliminating air will only get you about a 40% reduction in background noise (it goes as the square root). Now, this rule of thumb also goes for the support material around your crystal: one micron of cryoprotectant generates about as much background as one micron of crystal. So, if you have a 10 micron crystal mounted in a 1 mm thick drop, and manage to hit the crystal with a 10 micron beam, you still have 100 times more background
Re: [ccp4bb] X-ray Source Differences (WAS: RE: [ccp4bb] How far does rad dam travel?)
at the beginning of my experience of S-SAD about 10 years ago, it was not too difficult to do S-SAD phasing with inhouse data provided the resolution was better than 2.0A, while it did not always work with synchrotron data. Purely personal experience. I assume that the synchrotron data were collected at similarly-low energy? However, the inhouse machines I am familiar with have three circles, so that you get much better real redundancy with equivalent reflections recorded at different settings. This reduces systematic errors, I think. The most sophisticated synchrotron beamline I have been to offered a mini-kappa with 30degree range - that's not much compared to 10-20 different settings with varying phi- omega- and distance settings. Yes, I haven't seen much about people collecting multiple orientations of the same crystal, since I think people generally roast their crystals really fast to see higher-resolution spots. I am thinking recently that the best option might really be home sources with pixel-array detectors... The top-hat comes from a quote I received from Bruker, and I have no reason to believe the person acted purely with a salesperson's intent. Pretty interesting--wonder what's the best way to confirm this for our home source...? JPK Best, Tim On 01/12/2015 09:05 PM, Keller, Jacob wrote: the top-hat profile is one of the reasons why inhouse machines produce better quality data than synchrotrons. However, the often much increased resolution you achieve at the synchrotron is generally worth more than the quality of the data at restricted resolution. Cheers, Tim Several surprises to me: -Data from in-house sources is better? I have not heard of this--is there any systematic examination of this? I saw nothing about this in a very brief Google foray. -In-house beam profiles are top-hats? Is there a place which shows such measurements? Does not pop out of Google for me, but I would love to be shown that this is true. -Resolution at the synchrotron is better? This does not really seem right to me theoretically, although in practice it does seem to happen. I think it is just a question of waiting for enough exposure time, as the CCP4BB response quoted at bottom describes. JPK === Date: Tue, 12 Oct 2010 09:04:05 -0700 From: James Holton jmhol...@lbl.gov Re: Re: Lousy diffraction at home but fantastic at the synchrotron? There are a few things that synchrotron beamlines generally do better than home sources, but the most important are flux, collimation and absorption. Flux is in photons/s and simply scales down the amount of time it takes to get a given amount of photons onto the crystal. Contrary to popular belief, there is nothing magical about having more photons/s: it does not somehow make your protein molecules behave and line up in a more ordered way. However, it does allow you to do the equivalent of a 24-hour exposure in a few seconds (depending on which beamline and which home source you are comparing), so it can be hard to get your brain around the comparison. Collimation, in a nutshell, is putting all the incident photons through the crystal, preferably in a straight line. Illuminating anything that isn't the crystal generates background, and background buries weak diffraction spots (also known as high-resolution spots). Now, when I say crystal I mean the thing you want to shoot, so this includes the best part of a bent, cracked or otherwise inhomogeneous crystal. The amount of background goes as the square of the beam size, so a 0.5 mm beam can produce up to 25 times more background than a 0.1 mm beam (for a fixed spot intensity). Also, if the beam has high divergence (the range of incidence angles onto the crystal), then the spots on the detector will be more spread out than if the beam had low divergence, and the more spread-out the spots are the easier it is for them to fade into the background. Now, even at home sources, one can cut down the beam to have very low divergence and a very small size at the sample position, but this comes at the expense of flux. Another tenant of collimation (in my book) is the DEPTH of non-crystal stuff in the primary x-ray beam that can be seen by the detector. This includes the air space between the collimator and the beam stop. One millimeter of air generates about as much background as 1 micron of crystal, water, or plastic. Some home sources have ridiculously large air paths (like putting the backstop on the detector surface), and that can give you a lot of background. As a rule of thumb, you want you air path in mm to be less than or equal to your crystal size in microns. In this situation, the crystal itself is generating at least as much background as the air, and so further reducing the air path has diminishing returns. For example, going from 100 mm air and 100 um crystal
Re: [ccp4bb] How far does rad dam travel?
Yes, this is great info and thoughts. What I still do not understand, however, is why the noise from air/loop scattering is so bad--why not make sure only the top of the Gaussian is engulfing the crystal, and the tails can hit air or loop? Isn't the air scattering noise easily subtractable, being essentially flat over time, whereas uneven illumination of the crystal is highly difficult to correct? Also, in light of these considerations, it would seem to me a much better use of resources not to make brighter and smaller beams but instead concentrate on making better low-intensity big beam profiles (top-hats?) and lower-noise, faster detectors (like Pilatus and the new ADSC). Jacob -Original Message- From: James Holton [mailto:jmhol...@lbl.gov] Sent: Tuesday, December 30, 2014 3:57 PM To: Keller, Jacob; CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] How far does rad dam travel? Yes, bigger is okay, and perhaps a little better if you consider the effects of beam/crystal vibration and two sharp-edged boundaries dancing over each other. But bigger is better only to a point. That point is when the illuminated area of non-good-stuff is about equal to the area of the good stuff. This is because the total background noise is equal to the square root of the number of photons and equal volumes of any given stuff (good or non-good) yield about the same number of background-scattered photons. So, since you're taking the square root, completely eliminating the non-good-stuff only buys you a gain of 40% in total noise. Given that other sources of noise come into play when the beam and crystal are exactly matched (flicker), 40% is a reasonable compromise. This is why I recommend loop sizes that are about 40% bigger than the crystal itself. Much less risk of surface-tension injury, and the air around the loop scatters 1000x less than the non-crystal stuff in the loop: effectively defining the beam size. As for what beam profiles look like at different beamlines, there are some sobering mug-shots in this paper: http://dx.doi.org/10.1107/S0909049511008235 Some interesting quirks in a few of them, but in general optimally focused beams are Gaussian. Almost by definition! (central limit theorem and all that). It is when you de-focus that things get really embarrassing. X-ray mirrors all have a fingerprint in the de-focused region that leads to striations and other distortions. The technology is improving, but good solutions for de focusing are still not widely available. Perhaps because they are hard to fund. Genuine top-hat beams are rare, but there are a few of them. Petra-III is particularly proud of theirs. But top-hats are usually defined by collimation of a Gaussian and the more x-rays you have hitting the back of the aperture the more difficult it is to control the background generated by the collimator. If you can see the shadow of your pin on the detector, then you know there is a significant amount of background that is coming from upstream of your crystal! My solution is to collimate at roughly the FWHM. This chops off the tails and gives you a tolerably flat beam in the middle. How much more intense is the peak than the tails? Well, at the FWHM, the intensity is, well, half of that at the center. At twice that distance from the center, you are down to 6.2%. The equation is exp(-log(16)*(x/hwhm)**2) where hwhm is 1/2 of the FHWM. HTH! -James Holton MAD Scientist On 12/30/2014 12:10 PM, Keller, Jacob wrote: Yes, it gets complicated, doesn't it? This is why I generally recommend trying to use a beam that matches your crystal size. ...or is bigger, right? Diffuse scattering, yes, but more even illumination might be worth it? Generally, James, I have a question: what is the nature of the intensity cross-sections at most beamlines--are they usually Gaussian, or are some flatter? Or I guess, if Gaussian, how much more intense is the peak than the tails? JPK
Re: [ccp4bb] How far does rad dam travel?
Translate it by 13 microns. And use enough attenuation to get 180 degrees at each position. The track length of photoelectrons from 1 A X-rays in water, protein, plastic, and other materials with density close to 1 g/cm^3 and atomic numbers close to 7 is about 3 microns (Cole, Rad. Res. 1969). This defines the effective maximum range of the radiolytic chemistry. So, 10+3 = 13 microns from center-to-center if you want to avoid the damage of the last shot. That said, if you blast the living daylights out of one spot you will eventually be able to see it grow in size, and the uneven expansion produces stress that can propagate into the unilluminated areas of your sample. It stands to reason that stress is not good for diffraction, so you could consider this dose contrast effect as a mechanism of damage spreading. Nevertheless, it has been shown that at moderate doses (spots fading noticeably, but not disappearing entirely) properly accounting for the dose to the illuminated volume under different dose contrast situations leads to similar decay curves (Zeldin et al 2013), indicating that dose itself is a lot more important than dose contrast. Perhaps the main reason why damage spreading is still not all that well understood is because it is really really hard to produce an X-ray beam with edges sharp enough to study it! This is because all X-ray beams have some divergence (aka crossfire), and it is generally unwise to put a collimator inside the cryo stream. At 1 cm from the sample, even with the relatively low divergence of 100 microRadian (0.006 deg) the X-ray beam will be 1 micron bigger at the sample than it was at the collimator, blurring at the edges. You can reduce the divergence, but that will cost you flux. Balancing all these considerations for making a small beam generally results in a Gaussian shape, so I'm willing to bet your 10 micron beam is Gaussian. For any Gaussian beam half of the incident photons fall outside the full-width-at-half-max (FWHM) contour level generally quoted to define the size of the beam. No doubt a lot of people who think they are seeing damage spreading into regions outside the beam-box are actually seeing nothing more than damage caused by the tails of the main beam itself. Without collimation, these tails formally extend to infinity, so the question of how far to translate becomes not one of how to completely avoid damage, but how much damage you are willing to put up with. Is 10% okay? 5%? 20%? Remember, that even your first shot on a fresh part of the crystal is not going to be damage-free because damage is going on during each exposure, including the first one! (unless, of course, you are using an XFEL). You can do a lot of math trying to optimize diffracted photons vs damage (see Zeldin et al. 2013), but at the end of it all you find that the best way to utilize a given volume of good scattering matter is to use a beam that evenly illuminates that volume. This is because any bit of good stuff that never sees beam is wasted, and over-exposing one bit over another doesn't gain you anything. You also don't want to shoot things that are not good stuff because that corrupts your data with background and/or unwanted spots. Unfortunately, adjusting beam size to match each crystal shape exactly is a major engineering challenge and even if you could do this the sample has to rotate, making avoiding at least some unwanted material impossible. So, in reality, your beam size tends to be fixed and you must paint with it on the canvas of your large, rotating crystal. You can run simulations of such strategies at http://www.raddo.se/, and there are some tricks like off-setting the beam from the rotation axis to better approach even illumination, but in the end you cannot escape the even-illumination optimum. To that end, a train of Gaussian profiles separated by their FWHM forms a profile that is flat on top to within 10%. So, once again, since the damage from a 10 micron beam is 16 microns wide, a translation of 13 microns per wedge is a decent compromise. Hence my recommendation above. The next, question, of course, is how many shots you can get per wedge. I have written a web jiffy for answering questions like this: http://bl831.als.lbl.gov/xtallife.html Since you mention metals in your crystal, I'm going to assume this is a metalloprotein, and metalloprotein active sites can be particularly dose-sensitive. For example the water-splitting complex in Photosystem-II has been shown to decay with a half dose of 500 kGy (Yano, 2004), but the standing world-record is myoglobin, reducing half its iron with only 20 kGy (Denisov, 2007). Taking 500 kGy as your dose limit, and assuming you are using 1 A X-rays, I can type in the parameters you describe into the above web page and I get ... an error message. This is because the beam you are using delivers 5 MGy/s, so your first 0.1 s
Re: [ccp4bb] How far does rad dam travel?
Yes, it gets complicated, doesn't it? This is why I generally recommend trying to use a beam that matches your crystal size. ...or is bigger, right? Diffuse scattering, yes, but more even illumination might be worth it? Generally, James, I have a question: what is the nature of the intensity cross-sections at most beamlines--are they usually Gaussian, or are some flatter? Or I guess, if Gaussian, how much more intense is the peak than the tails? JPK
Re: [ccp4bb] How far does rad dam travel?
Yes, bigger is okay, and perhaps a little better if you consider the effects of beam/crystal vibration and two sharp-edged boundaries dancing over each other. But bigger is better only to a point. That point is when the illuminated area of non-good-stuff is about equal to the area of the good stuff. This is because the total background noise is equal to the square root of the number of photons and equal volumes of any given stuff (good or non-good) yield about the same number of background-scattered photons. So, since you're taking the square root, completely eliminating the non-good-stuff only buys you a gain of 40% in total noise. Given that other sources of noise come into play when the beam and crystal are exactly matched (flicker), 40% is a reasonable compromise. This is why I recommend loop sizes that are about 40% bigger than the crystal itself. Much less risk of surface-tension injury, and the air around the loop scatters 1000x less than the non-crystal stuff in the loop: effectively defining the beam size. As for what beam profiles look like at different beamlines, there are some sobering mug-shots in this paper: http://dx.doi.org/10.1107/S0909049511008235 Some interesting quirks in a few of them, but in general optimally focused beams are Gaussian. Almost by definition! (central limit theorem and all that). It is when you de-focus that things get really embarrassing. X-ray mirrors all have a fingerprint in the de-focused region that leads to striations and other distortions. The technology is improving, but good solutions for de focusing are still not widely available. Perhaps because they are hard to fund. Genuine top-hat beams are rare, but there are a few of them. Petra-III is particularly proud of theirs. But top-hats are usually defined by collimation of a Gaussian and the more x-rays you have hitting the back of the aperture the more difficult it is to control the background generated by the collimator. If you can see the shadow of your pin on the detector, then you know there is a significant amount of background that is coming from upstream of your crystal! My solution is to collimate at roughly the FWHM. This chops off the tails and gives you a tolerably flat beam in the middle. How much more intense is the peak than the tails? Well, at the FWHM, the intensity is, well, half of that at the center. At twice that distance from the center, you are down to 6.2%. The equation is exp(-log(16)*(x/hwhm)**2) where hwhm is 1/2 of the FHWM. HTH! -James Holton MAD Scientist On 12/30/2014 12:10 PM, Keller, Jacob wrote: Yes, it gets complicated, doesn't it? This is why I generally recommend trying to use a beam that matches your crystal size. ...or is bigger, right? Diffuse scattering, yes, but more even illumination might be worth it? Generally, James, I have a question: what is the nature of the intensity cross-sections at most beamlines--are they usually Gaussian, or are some flatter? Or I guess, if Gaussian, how much more intense is the peak than the tails? JPK
[ccp4bb] How far does rad dam travel?
Dear all In a metal-containing crystal of (say) 200 um x 200 um, and a beam size of 10 um x 10 um, how far will I need to move away from an irradiated part to a fresh part to obtain an undamaged dataset? Exposure conditions: 100 % transmission at 10^12 ph/s, 0.1 s exposure, fine sliced at 0.1 degree/frame with a total 180 degrees. Obviously it will be crystal dependent but I would like to have a rule of thumb. I could use fresh crystals altogether, but not all crystals diffract well unfortunately. Thanks. Mohamed
