Hi Ian, James, I have a strong feeling that I have seen this result before, and it was due to Andy Hammersley at ESRF. I’ve done a literature search and there is a paper relating to errors in analysis of counting statistics (se below), but I had a quick look at this and could not find the (N+1) correction, so it must have been somewhere else. I Have cc’d Andy on this Email (hoping that this Email address from 2016 still works) and maybe he can throw more light on this. What I remember at the time I saw this was the simplicity of the correction.
Cheers, Andrew Reducing bias in the analysis of counting statistics data Hammersley, AP <https://www.webofscience.com/wos/author/record/2665675> (Hammersley, AP) Antoniadis, A <https://www.webofscience.com/wos/author/record/13070551> (Antoniadis, A) NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT Volume 394 Issue 1-2 Page 219-224 DOI 10.1016/S0168-9002(97)00668-2 Published JUL 11 1997 > On 12 Oct 2021, at 18:55, Ian Tickle <ianj...@gmail.com> wrote: > > > Hi James > > What the Poisson distribution tells you is that if the true count is N then > the expectation and variance are also N. That's not the same thing as saying > that for an observed count N the expectation and variance are N. Consider > all those cases where the observed count is exactly zero. That can arise > from any number of true counts, though as you noted larger values become > increasingly unlikely. However those true counts are all >= 0 which means > that the mean and variance of those true counts must be positive and > non-zero. From your results they are both 1 though I haven't been through > the algebra to prove it. > > So what you are saying seems correct: for N observed counts we should be > taking the best estimate of the true value and variance as N+1. For > reasonably large N the difference is small but if you are concerned with weak > images it might start to become significant. > > Cheers > > -- Ian > > > On Tue, 12 Oct 2021 at 17:56, James Holton <jmhol...@lbl.gov > <mailto:jmhol...@lbl.gov>> wrote: > All my life I have believed that if you're counting photons then the > error of observing N counts is sqrt(N). However, a calculation I just > performed suggests its actually sqrt(N+1). > > My purpose here is to understand the weak-image limit of data > processing. Question is: for a given pixel, if one photon is all you > got, what do you "know"? > > I simulated millions of 1-second experiments. For each I used a "true" > beam intensity (Itrue) between 0.001 and 20 photons/s. That is, for > Itrue= 0.001 the average over a very long exposure would be 1 photon > every 1000 seconds or so. For a 1-second exposure the observed count (N) > is almost always zero. About 1 in 1000 of them will see one photon, and > roughly 1 in a million will get N=2. I do 10,000 such experiments and > put the results into a pile. I then repeat with Itrue=0.002, > Itrue=0.003, etc. All the way up to Itrue = 20. At Itrue > 20 I never > see N=1, not even in 1e7 experiments. With Itrue=0, I also see no N=1 > events. > Now I go through my pile of results and extract those with N=1, and > count up the number of times a given Itrue produced such an event. The > histogram of Itrue values in this subset is itself Poisson, but with > mean = 2 ! If I similarly count up events where 2 and only 2 photons > were seen, the mean Itrue is 3. And if I look at only zero-count events > the mean and standard deviation is unity. > > Does that mean the error of observing N counts is really sqrt(N+1) ? > > I admit that this little exercise assumes that the distribution of Itrue > is uniform between 0.001 and 20, but given that one photon has been > observed Itrue values outside this range are highly unlikely. The > Itrue=0.001 and N=1 events are only a tiny fraction of the whole. So, I > wold say that even if the prior distribution is not uniform, it is > certainly bracketed. Now, Itrue=0 is possible if the shutter didn't > open, but if the rest of the detector pixels have N=~1, doesn't this > affect the prior distribution of Itrue on our pixel of interest? > > Of course, two or more photons are better than one, but these days with > small crystals and big detectors N=1 is no longer a trivial situation. > I look forward to hearing your take on this. And no, this is not a trick. > > -James Holton > MAD Scientist > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > <https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1> > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB > <http://www.jiscmail.ac.uk/CCP4BB>, a mailing list hosted by > www.jiscmail.ac.uk <http://www.jiscmail.ac.uk/>, terms & conditions are > available at https://www.jiscmail.ac.uk/policyandsecurity/ > <https://www.jiscmail.ac.uk/policyandsecurity/> > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > <https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1> ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/