I think I deleted the duplicate posts relevant to data gaps in deviation statistics, and apologize if I took anybody’s out.
In my personal software I don’t deal with outliers or data gaps by interpolating, spline-fitting, or any other fancier things. Instead I compute the standard statistical measures by simply ignoring terms in the summation that hit upon a data gap or outlier. I don’t think Stable32 does that, so I’d like to pose this challenge to anyone interested in serving the community: Assuming IEEE will make the source code available, can the approach I use be made an option? And yes, I know that the uncertainties of the deviation points get hard to compute, but such issues can be warned about in the documentation, if not addressed. > -------------------------------------------- > > Message: 1 > Date: Tue, 19 Jan 2021 08:49:13 -0500 > From: Bob kb8tq <kb...@n1k.org> > To: Discussion of precise time and frequency measurement > <time-nuts@lists.febo.com> > Subject: Re: [time-nuts] Long Wave Radio-Frequency standard testing > Message-ID: <30728187-dbfe-4110-a595-d1497b5f4...@n1k.org> > Content-Type: text/plain; charset=utf-8 > > Hi > > The normal approach to filling a gap is to put in a point that is the average > of the two adjacent points. The assumption is that this is a ?safe? value that > will not blow up the result. That?s probably ok if it is done rarely. The > risk is > that you are running a filter process (averaging is a low pass filter). > > If you pull out a *lot* of outliers and replace them, you are doing a lot of > filtering. > Since you are measuring noise, filtering is very likely to improve the > result. > The question becomes - how representative is the result after a lot of this > or > that has been done? > > Obviously the answer to all this depends on what you are trying to do. If you > are running a control loop and the output improves, that?s fine. If you are > trying to provide an accurate measure of noise ?. maybe not so much :) > > Bob > >> On Jan 19, 2021, at 2:15 AM, Gilles Clement <clemg...@gmail.com> wrote: >> >> Hi, >> Yes outliers removal creates gap in Stable32. >> The ? fill ? function can fills gaps with interpolated values. >> It does not change much the graphs, except in the low Tau area (see >> attached). >> Do you know a discussion of impact of outliers removal ? >> Gilles. >> >> >> >>> Le 18 janv. 2021 ? 22:06, Bob kb8tq <kb...@n1k.org> a ?crit : >>> >>> Hi >>> >>> As you throw away samples that are far off the mean, you reduce the sample >>> rate ( or at least create gaps in the record). Dealing with that could be >>> difficult. >>> >>> Bob >>> >>>> On Jan 18, 2021, at 1:33 PM, Gilles Clement <clemg...@gmail.com> wrote: >>>> >>>>> Hi >>>>> >>>>> Very cool !!! >>>>> >>>>> The red trace is obviously the one to focus on. Some sort of digital loop >>>>> that >>>>> only operates under the ?known good? conditions would seem to make sense. >>>>> >>>>> Thanks for sharing >>>>> >>>>> Bob >>>> >>>> Hi, >>>> I tried something with the idea to consider night records fluctuations as >>>> ? outliers ? as compared to day records. >>>> Indeed the 3 days record mean value is flat and the histogram quite >>>> gaussian. >>>> So I processed the 3 days record (green trace) with Stable32?s ? Check >>>> Function ?, >>>> while removing outliers with decreasing values of the Sigma Factor. The >>>> graph below shows the outcome. >>>> The graph with Sigma=0.8 (blue trace) connects rather well with the 1Day >>>> record (red trace). >>>> Would this be a workable approach ? >>>> Best, >>>> Gilles. >>>> >>>> >>>> > > > > ------------------------------ > > Message: 2 > Date: Tue, 19 Jan 2021 13:37:53 -0500 > From: Dave Daniel <kc0...@gmail.com> > To: Discussion of precise time and frequency measurement > <time-nuts@lists.febo.com> > Subject: Re: [time-nuts] Long Wave Radio-Frequency standard testing > Message-ID: <2235bf02-8fff-4000-85a5-119bd358a...@gmail.com> > Content-Type: text/plain; charset=utf-8 > > Or one can replace those values with zero. That eliminates them; averaging > then proceeds without those values altering the most probable correct average. > > DaveD > >> >>> >>>> Le 18 janv. 2021 ? 22:06, Bob kb8tq <kb...@n1k.org> a ?crit : >>>> >>>> Hi >>>> >>>> As you throw away samples that are far off the mean, you reduce the sample >>>> rate ( or at least create gaps in the record). Dealing with that could be >>>> difficult. >>>> >>>> Bob >>>> >>>>>> On Jan 18, 2021, at 1:33 PM, Gilles Clement <clemg...@gmail.com> wrote: >>>>>> >>>>>> Hi >>>>>> >>>>>> Very cool !!! >>>>>> >>>>>> The red trace is obviously the one to focus on. Some sort of digital >>>>>> loop that >>>>>> only operates under the ?known good? conditions would seem to make >>>>>> sense. >>>>>> >>>>>> Thanks for sharing >>>>>> >>>>>> Bob >>>>> >>>>> Hi, >>>>> I tried something with the idea to consider night records fluctuations as >>>>> ? outliers ? as compared to day records. >>>>> Indeed the 3 days record mean value is flat and the histogram quite >>>>> gaussian. >>>>> So I processed the 3 days record (green trace) with Stable32?s ? Check >>>>> Function ?, >>>>> while removing outliers with decreasing values of the Sigma Factor. The >>>>> graph below shows the outcome. >>>>> The graph with Sigma=0.8 (blue trace) connects rather well with the 1Day >>>>> record (red trace). >>>>> Would this be a workable approach ? >>>>> Best, >>>>> Gilles. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> time-nuts mailing list -- time-nuts@lists.febo.com >>>>> To unsubscribe, go to >>>>> http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com >>>>> and follow the instructions there. >>>> >>>> >>>> _______________________________________________ >>>> time-nuts mailing list -- time-nuts@lists.febo.com >>>> To unsubscribe, go to >>>> http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com >>>> and follow the instructions there. >>> >>> _______________________________________________ >>> time-nuts mailing list -- time-nuts@lists.febo.com >>> To unsubscribe, go to >>> http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com >>> and follow the instructions there. >> >> > > Message: 5 > Date: Tue, 19 Jan 2021 16:27:35 -0500 > From: Bob kb8tq <kb...@n1k.org> > To: Discussion of precise time and frequency measurement > <time-nuts@lists.febo.com> > Subject: Re: [time-nuts] Long Wave Radio-Frequency standard testing > Message-ID: <601c2ae8-4750-49dc-801b-b1c6c8d22...@n1k.org> > Content-Type: text/plain; charset=utf-8 > > Hi > > Assuming the goal is a normal ADEV or xDEV sort of calculation: > > If you replace the raw phase values with zero that can mess things up > > 0 seconds +20 ns > 1 seconds +22 ns > 2 seconds +23 ns > 3 seconds +25 ns > 4 seconds +27ns > 5 seconds +29 ns > > If you ?loose? one of those 20 to 30 ns values and replace it with zero, you > have significantly > changed the data set. > > Even if you are looking at deltas, zero stuffing would be problematic with > that > (contrived) phase data set. > > 1 seconds +2 > 2 seconds +1 > 3 seconds +2 > 4 seconds +2 > 5 seconds +2 > > If the objective is something like a PLL then ?hold at the last value? is the > only practical > answer to the question. You don?t *have* the next value and you need to stuff > something > into the control loop computation. > > Bob > > >> On Jan 19, 2021, at 1:37 PM, Dave Daniel <kc0...@gmail.com> wrote: >> >> Or one can replace those values with zero. That eliminates them; averaging >> then proceeds without those values altering the most probable correct >> average. >> >> DaveD >> >>>>> >>>>> _______________________________________________ >>>>> time-nuts mailing list -- time-nuts@lists.febo.com >>>>> To unsubscribe, go to >>>>> http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com >>>>> and follow the instructions there. >>>> >>>> _______________________________________________ >>>> time-nuts mailing list -- time-nuts@lists.febo.com >>>> To unsubscribe, go to >>>> http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com >>>> and follow the instructions there. >>> >>> >>> _______________________________________________ >>> time-nuts mailing list -- time-nuts@lists.febo.com >>> To unsubscribe, go to >>> http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com >>> and follow the instructions there. >> >> _______________________________________________ >> time-nuts mailing list -- time-nuts@lists.febo.com >> To unsubscribe, go to >> http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com >> and follow the instructions there. > > > > > ------------------------------ > > Message: 6 > Date: Tue, 19 Jan 2021 17:19:30 -0500 > From: Dave Daniel <kc0...@gmail.com> > To: Discussion of precise time and frequency measurement > <time-nuts@lists.febo.com> > Subject: Re: [time-nuts] Long Wave Radio-Frequency standard testing > Message-ID: <dd608f22-023e-48fd-bd8e-821da0a5d...@gmail.com> > Content-Type: text/plain; charset=utf-8 > > Answers inline > >> On Jan 19, 2021, at 16:27, Bob kb8tq <kb...@n1k.org> wrote: >> >> Hi >> >> Assuming the goal is a normal ADEV or xDEV sort of calculation: >> >> If you replace the raw phase values with zero that can mess things up >> >> 0 seconds +20 ns >> 1 seconds +22 ns >> 2 seconds +23 ns >> 3 seconds +25 ns >> 4 seconds +27ns >> 5 seconds +29 ns >> >> If you ?loose? one of those 20 to 30 ns values and replace it with zero, you >> have significantly >> changed the data set. > > Ok > >> >> Even if you are looking at deltas, > > Nope. It doesn?t work for deltas. > _______________________________________________ time-nuts mailing list -- time-nuts@lists.febo.com To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com and follow the instructions there.
