Re: [Discuss-gnuradio] Calculating additive noise power for known signal

[Andy Walls]

On Wed, 2016-08-24 at 13:55 -0400, Andy Walls wrote:
> On Wed, 2016-08-24 at 12:00 -0400, discuss-gnuradio-requ...@gnu.org
> wrote:
> > Date: Tue, 23 Aug 2016 16:42:33 -0700 (MST)
> > From: Paul Creaser
> > To: Discuss-gnuradio@gnu.org
> 
> > In a real system the signal would be a preamble, which would normally be 
> > used
> > for synchronization purposes at the receiver end (For example PLC system).
> > 
> > "I'm not quite sure what you mean by "cyclic noise", but the example you
> > give is 50 Hz (or 60 Hz) hum, so a narrowband interference."
> > 
> > Cyclic noise, perhaps my understanding is incorrect. In a PLC system, the
> > noise rejection system uses a Zero Cross detection to detect the beginning
> > of a power cycle (50Hz/60Hz). During the cycle at the same phase point,
> > noise repeatedly occurs. So on a scope you would see a nice sine wave for
> > the power and noise which repeats at the same phase point in the power
> > cycle.
> 
> Ah, now I understand: a periodic noise impulse.  So definitely not an
> AWGN.
> 
> 
> > This is different from the 50 Hz noise I suggested previously. However the
> > idea/hope is that this noise is narrow band noise and because of its
> > repetitive behaviour,
> 
> Impulsive noise is usually broadband.  Signals that are concentrated in
> the time domain are spread out in the frequency domain, and vice versa
> (this falls out from the Fourier transform).
> 
> A periodic, broadband noise impulse is essentially going bump up the
> noise floor for a short time every 50th (or 60th) of a second.
> 
> 
> 
> > .i.e. occurring in the same point in the cycle, and
> > relatively stable amplitude, it should be possible to remove it from a known
> > signal.
> > 
> > I will take on board all the useful advice and continue my studies.
> > 

This paper on impulsive noise mitigation in PLC systems seems apropos:
https://arxiv.org/pdf/1303.1217.pdf

-Andy


___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] Calculating additive noise power for known signal

[Marcus Müller]

Hi Andy,

> 3. HF systems are the only other systems I'm familiar with that deal
> with impulsive noise.  HF communications systems often employ FEC and
> interleavers to combat impulsive noise.
Powerline comms typically suffer from that kind of noise. Think of
someone switching on lights. Or a commutator in the motor of the washing
mashine that someone carelessly attached to your communication channel!

Best regards,
Marcus

On 24.08.2016 19:55, Andy Walls wrote:
> On Wed, 2016-08-24 at 12:00 -0400, discuss-gnuradio-requ...@gnu.org
> wrote:
>> Date: Tue, 23 Aug 2016 16:42:33 -0700 (MST)
>> From: Paul Creaser
>> To: Discuss-gnuradio@gnu.org
>> In a real system the signal would be a preamble, which would normally be used
>> for synchronization purposes at the receiver end (For example PLC system).
>>
>> "I'm not quite sure what you mean by "cyclic noise", but the example you
>> give is 50 Hz (or 60 Hz) hum, so a narrowband interference."
>>
>> Cyclic noise, perhaps my understanding is incorrect. In a PLC system, the
>> noise rejection system uses a Zero Cross detection to detect the beginning
>> of a power cycle (50Hz/60Hz). During the cycle at the same phase point,
>> noise repeatedly occurs. So on a scope you would see a nice sine wave for
>> the power and noise which repeats at the same phase point in the power
>> cycle.
> Ah, now I understand: a periodic noise impulse.  So definitely not an
> AWGN.
>
>
>> This is different from the 50 Hz noise I suggested previously. However the
>> idea/hope is that this noise is narrow band noise and because of its
>> repetitive behaviour,
> Impulsive noise is usually broadband.  Signals that are concentrated in
> the time domain are spread out in the frequency domain, and vice versa
> (this falls out from the Fourier transform).
>
> A periodic, broadband noise impulse is essentially going bump up the
> noise floor for a short time every 50th (or 60th) of a second.
>
>
>
>> .i.e. occurring in the same point in the cycle, and
>> relatively stable amplitude, it should be possible to remove it from a known
>> signal.
>>
>> I will take on board all the useful advice and continue my studies.
>>
> So I have a couple of thoughts:
>
> 1. The LMS adaptive filter I suggested earlier won't do much good in the
> face of broadband periodic noise impulses.  It still may help with
> equalizing a poor channel, since PLC transmission lines are usually of
> uncontrolled and unknown quality.
>
> 2. Your communication system should still use appropriate channel
> filters and pulse matched filters, to mitigate the effects of broadband
> noise that shows up outside your channel of interest.
>
> 3. HF systems are the only other systems I'm familiar with that deal
> with impulsive noise.  HF communications systems often employ FEC and
> interleavers to combat impulsive noise.
>
> 4. Adaptive noise cancellation techniques may help here, but I'm no
> expert on them.  A quick Google turns up some decent material.
>
> 5. If you track the timing of the impulsive noise bursts, is it feasible
> to have your system schedule its communications between the bursts? (I
> don't know your topology.)  Think of it as a TDMA timeslot clock. :)
> 1/60th of a second at 9600 symbols/second is 160 symbols.
>
> -Andy 
>
>
> ___
> Discuss-gnuradio mailing list
> Discuss-gnuradio@gnu.org
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio


___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] Calculating additive noise power for known signal

[Andy Walls]

On Wed, 2016-08-24 at 12:00 -0400, discuss-gnuradio-requ...@gnu.org
wrote:
> Date: Tue, 23 Aug 2016 16:42:33 -0700 (MST)
> From: Paul Creaser
> To: Discuss-gnuradio@gnu.org

> In a real system the signal would be a preamble, which would normally be used
> for synchronization purposes at the receiver end (For example PLC system).
> 
> "I'm not quite sure what you mean by "cyclic noise", but the example you
> give is 50 Hz (or 60 Hz) hum, so a narrowband interference."
> 
> Cyclic noise, perhaps my understanding is incorrect. In a PLC system, the
> noise rejection system uses a Zero Cross detection to detect the beginning
> of a power cycle (50Hz/60Hz). During the cycle at the same phase point,
> noise repeatedly occurs. So on a scope you would see a nice sine wave for
> the power and noise which repeats at the same phase point in the power
> cycle.

Ah, now I understand: a periodic noise impulse.  So definitely not an
AWGN.


> This is different from the 50 Hz noise I suggested previously. However the
> idea/hope is that this noise is narrow band noise and because of its
> repetitive behaviour,

Impulsive noise is usually broadband.  Signals that are concentrated in
the time domain are spread out in the frequency domain, and vice versa
(this falls out from the Fourier transform).

A periodic, broadband noise impulse is essentially going bump up the
noise floor for a short time every 50th (or 60th) of a second.



> .i.e. occurring in the same point in the cycle, and
> relatively stable amplitude, it should be possible to remove it from a known
> signal.
> 
> I will take on board all the useful advice and continue my studies.
> 

So I have a couple of thoughts:

1. The LMS adaptive filter I suggested earlier won't do much good in the
face of broadband periodic noise impulses.  It still may help with
equalizing a poor channel, since PLC transmission lines are usually of
uncontrolled and unknown quality.

2. Your communication system should still use appropriate channel
filters and pulse matched filters, to mitigate the effects of broadband
noise that shows up outside your channel of interest.

3. HF systems are the only other systems I'm familiar with that deal
with impulsive noise.  HF communications systems often employ FEC and
interleavers to combat impulsive noise.

4. Adaptive noise cancellation techniques may help here, but I'm no
expert on them.  A quick Google turns up some decent material.

5. If you track the timing of the impulsive noise bursts, is it feasible
to have your system schedule its communications between the bursts? (I
don't know your topology.)  Think of it as a TDMA timeslot clock. :)
1/60th of a second at 9600 symbols/second is 160 symbols.

-Andy 


___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] Calculating additive noise power for known signal

[Andy Walls]

On Tue, 2016-08-23 at 15:36 -0700, Richard Bell wrote:
> Great answer. I wish I could upvote it! 

Thank you! You're too kind, in light of my typos (e.g. your -> you're,
of -> on).  :) 


> There should be a GNU Radio Stack Exchange type thing. 

There is :)

http://stackoverflow.com/questions/tagged/gnuradio

Regards,
Andy



___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] Calculating additive noise power for known signal

[Marcus Müller]

Hi Rich!

HA! You've come to the *right* neighborhood!

There's actually more [tag:gnuradio] stuff on StackOverflow than I can
keep check on:

http://stackoverflow.com/questions/tagged/gnuradio

A lot of mathematically more interesting questions happen on dsp.SE

http://dsp.stackexchange.com/

and specifically

http://dsp.stackexchange.com/questions/tagged/gnuradio

I'd love if anyone could pick up on those – the DSP.SE community is
really great at helping people understand and improve their own
questions, so these Q/A pairs and discussions are often pretty nice! For
example, Phil Frost has a question I've not found the time addressing
yet. It'd be really nice if someone picked up that slack :D

Talking of Phil: Then there's the amateur radio SE site, and that has
very mixed-topic questions; many are very operation-centric, other would
belong on dsp.SE, and yet other are discussion of basic Radio concept. I
like the mix, but I clearly don't check these often, so if everyone
could have a look every few months:

http://ham.stackexchange.com

Cheers,
Marcus


On 24.08.2016 00:36, Richard Bell wrote:
> Great answer. I wish I could upvote it! There should be a GNU Radio
> Stack Exchange type thing.
>
> Rich
>
> On Tue, Aug 23, 2016 at 3:07 PM, Andy Walls
> mailto:a...@silverblocksystems.net>> wrote:
>
> On Tue, 2016-08-23 at 12:00 -0400,
> discuss-gnuradio-requ...@gnu.org
> 
> wrote:
> > Message: 7
> > Date: Mon, 22 Aug 2016 18:14:29 -0700 (MST)
> > From: Paul Creaser  >
> >
> > I've just started studying methods used to detect and then
> filter out/remove
> > cyclic noise from known signals.
> >
> > I have a signal of 256 samples which repeats itself. I take this
> signal,
> > attenuate it and add noise at a specific band (frequency band),
> for example
> > 50 Hz Sine Wave. In the simplest case this is none varying.
> However in the
> > future it will vary slowly over time.
>
> I'm not quite sure what you mean by "cyclic noise", but the
> example you
> give is 50 Hz (or 60 Hz) hum, so a narrowband interference.
>
>
> > What I would like to do is find the power level of the additive
> cyclic noise
> > (, which should be the difference between the two signals) and
> where in the
> > frequency spectrum this noise exists. Using this information, I
> would hope
> > to use weighting to recover the original signal.
>
> If the noise is always out of the channel of your signal of interest,
> then a bandpass filter will do the job and your done.
>
> If the noise is in the channel of your signal of interest, then it
> sounds like what you really want in the end is an adaptive
> equalizer or
> filter.
>
> If you're not afraid of a lot of work:
> Just dive into implementing a Least Mean Squares (LMS) adaptive
> filter.
>
> You can either make it Data-Aided (DA), adapting the filter when it
> detects and operates on a known preamble; or Decision Directed (DD),
> adapting the filter every time it makes a decision about what a
> data bit
> should be.
>
> I prefer using a Data-Aided LMS adaptive filter, as I often work with
> signals that have known preambles.
>
> Such a system would look something like:
>
> received samples source -> channel filter -> automatic gain control ->
> correlator to detect and mark the preamble -> LMS DA adaptive filter
> -> ...
>
> Translating that to example GNURadio blocks:
>
> USRP Source -> Freq Xlating FIR Filter -> Feed Forward AGC ->
> Correlation Estimator -> (Your custom LMS DA filter block) -> ...
>
>
> > *Steps*
> >
> > 1 I take the original and modified signal and rescale the
> modified signal to
> > match the original.
> >
> > At the moment I use a very naive approach which is to take the
> absolute sum
> > of the 256 samples for both signals and from this calculate a
> simple scale
> > factor. I think this should be OK where I have narrow band
> noise, but it may
> > fail badly in other cases where the noise levels are high.
> >
> > 2 Next I take the FFT of the two signals (256 samples).
> >
> > 3 Calculate the noise
> >
> > Using the difference between the FFTs, I then calculate the
> noise power.
> >
> > *Two questions?*
> >
> > 1 The rescaling method is very basic, using absolute accumulated
> sums. Does
> > GNU radio have any blocks, which could perform this auto-scaling
> more
> > effectively?
>
> GNURadio has several AGC blocks.  They all have their quirks. 
> Pick one
> an try to make it work.
>
>
> > 2 Using the basic difference between the FFT's, such as the absolute
> > magnitude difference, should provide a starting point for
> calculating the
> > noise pow

Re: [Discuss-gnuradio] Calculating additive noise power for known signal

[Paul Creaser]

In a real system the signal would be a preamble, which would normally be used
for synchronization purposes at the receiver end (For example PLC system).

"I'm not quite sure what you mean by "cyclic noise", but the example you
give is 50 Hz (or 60 Hz) hum, so a narrowband interference."

Cyclic noise, perhaps my understanding is incorrect. In a PLC system, the
noise rejection system uses a Zero Cross detection to detect the beginning
of a power cycle (50Hz/60Hz). During the cycle at the same phase point,
noise repeatedly occurs. So on a scope you would see a nice sine wave for
the power and noise which repeats at the same phase point in the power
cycle.

This is different from the 50 Hz noise I suggested previously. However the
idea/hope is that this noise is narrow band noise and because of its
repetitive behaviour, .i.e. occurring in the same point in the cycle, and
relatively stable amplitude, it should be possible to remove it from a known
signal.

I will take on board all the useful advice and continue my studies.




--
View this message in context: 
http://gnuradio.4.n7.nabble.com/Calculating-additive-noise-power-for-known-signal-tp61281p61302.html
Sent from the GnuRadio mailing list archive at Nabble.com.

___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] Calculating additive noise power for known signal

[Richard Bell]

Great answer. I wish I could upvote it! There should be a GNU Radio Stack
Exchange type thing.

Rich

On Tue, Aug 23, 2016 at 3:07 PM, Andy Walls 
wrote:

> On Tue, 2016-08-23 at 12:00 -0400, discuss-gnuradio-requ...@gnu.org
> wrote:
> > Message: 7
> > Date: Mon, 22 Aug 2016 18:14:29 -0700 (MST)
> > From: Paul Creaser 
> >
> > I've just started studying methods used to detect and then filter
> out/remove
> > cyclic noise from known signals.
> >
> > I have a signal of 256 samples which repeats itself. I take this signal,
> > attenuate it and add noise at a specific band (frequency band), for
> example
> > 50 Hz Sine Wave. In the simplest case this is none varying. However in
> the
> > future it will vary slowly over time.
>
> I'm not quite sure what you mean by "cyclic noise", but the example you
> give is 50 Hz (or 60 Hz) hum, so a narrowband interference.
>
>
> > What I would like to do is find the power level of the additive cyclic
> noise
> > (, which should be the difference between the two signals) and where in
> the
> > frequency spectrum this noise exists. Using this information, I would
> hope
> > to use weighting to recover the original signal.
>
> If the noise is always out of the channel of your signal of interest,
> then a bandpass filter will do the job and your done.
>
> If the noise is in the channel of your signal of interest, then it
> sounds like what you really want in the end is an adaptive equalizer or
> filter.
>
> If you're not afraid of a lot of work:
> Just dive into implementing a Least Mean Squares (LMS) adaptive
> filter.
>
> You can either make it Data-Aided (DA), adapting the filter when it
> detects and operates on a known preamble; or Decision Directed (DD),
> adapting the filter every time it makes a decision about what a data bit
> should be.
>
> I prefer using a Data-Aided LMS adaptive filter, as I often work with
> signals that have known preambles.
>
> Such a system would look something like:
>
> received samples source -> channel filter -> automatic gain control ->
> correlator to detect and mark the preamble -> LMS DA adaptive filter
> -> ...
>
> Translating that to example GNURadio blocks:
>
> USRP Source -> Freq Xlating FIR Filter -> Feed Forward AGC ->
> Correlation Estimator -> (Your custom LMS DA filter block) -> ...
>
>
> > *Steps*
> >
> > 1 I take the original and modified signal and rescale the modified
> signal to
> > match the original.
> >
> > At the moment I use a very naive approach which is to take the absolute
> sum
> > of the 256 samples for both signals and from this calculate a simple
> scale
> > factor. I think this should be OK where I have narrow band noise, but it
> may
> > fail badly in other cases where the noise levels are high.
> >
> > 2 Next I take the FFT of the two signals (256 samples).
> >
> > 3 Calculate the noise
> >
> > Using the difference between the FFTs, I then calculate the noise power.
> >
> > *Two questions?*
> >
> > 1 The rescaling method is very basic, using absolute accumulated sums.
> Does
> > GNU radio have any blocks, which could perform this auto-scaling more
> > effectively?
>
> GNURadio has several AGC blocks.  They all have their quirks.  Pick one
> an try to make it work.
>
>
> > 2 Using the basic difference between the FFT's, such as the absolute
> > magnitude difference, should provide a starting point for calculating the
> > noise power. Again is this naive?
>
> Noise power and noise density have specific meanings which I don't think
> match what you're thinking about here.  AFAICT, you want to know the
> power of an in-channel narrowband interference (so that you can
> ultimately filter it out).
>
> Looking at FFT's will give you a feel for the situation, but it's kind
> of a blunt instrument, if you plan of filtering by direct FFT bin
> scaling or excision.
>
> It really sounds like what you want is an adaptive equalizer (aka
> adaptive filter).
>
> There's lots of existing literature on equalizers.
> This lecture is still a little too advanced for most folks, but it has
> the basic concepts covered clearer than most others I could find on
> Google:
> http://www.eecg.toronto.edu/~johns/nobots/courses/ece1392/
> equalization2.pdf
>
>
> Section 14.6 of this book describes the LMS algorithm:
> https://www.amazon.com/Mathematical-Methods-Algorithms-Signal-Processing/
> dp/0201361868
>
> And here is a PDF copy I spotted on the internet (click at your own
> risk):
> https://www.u-cursos.cl/usuario/834c0e46b93fd72fd8408c492af56f
> 8d/mi_blog/r/4%29_Todd_Moon_Mathematical_Methods_and_
> Algorithms_for_Signal_Processing.pdf
>
> -Andy
>
>
>
> ___
> Discuss-gnuradio mailing list
> Discuss-gnuradio@gnu.org
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] Calculating additive noise power for known signal

[Andy Walls]

On Tue, 2016-08-23 at 12:00 -0400, discuss-gnuradio-requ...@gnu.org
wrote:
> Message: 7
> Date: Mon, 22 Aug 2016 18:14:29 -0700 (MST)
> From: Paul Creaser 
> 
> I've just started studying methods used to detect and then filter out/remove
> cyclic noise from known signals. 
> 
> I have a signal of 256 samples which repeats itself. I take this signal,
> attenuate it and add noise at a specific band (frequency band), for example
> 50 Hz Sine Wave. In the simplest case this is none varying. However in the
> future it will vary slowly over time. 

I'm not quite sure what you mean by "cyclic noise", but the example you
give is 50 Hz (or 60 Hz) hum, so a narrowband interference.


> What I would like to do is find the power level of the additive cyclic noise
> (, which should be the difference between the two signals) and where in the
> frequency spectrum this noise exists. Using this information, I would hope
> to use weighting to recover the original signal.

If the noise is always out of the channel of your signal of interest,
then a bandpass filter will do the job and your done.

If the noise is in the channel of your signal of interest, then it
sounds like what you really want in the end is an adaptive equalizer or
filter.

If you're not afraid of a lot of work:
Just dive into implementing a Least Mean Squares (LMS) adaptive
filter.  

You can either make it Data-Aided (DA), adapting the filter when it
detects and operates on a known preamble; or Decision Directed (DD),
adapting the filter every time it makes a decision about what a data bit
should be.

I prefer using a Data-Aided LMS adaptive filter, as I often work with
signals that have known preambles.
 
Such a system would look something like:

received samples source -> channel filter -> automatic gain control ->
correlator to detect and mark the preamble -> LMS DA adaptive filter
-> ...

Translating that to example GNURadio blocks:

USRP Source -> Freq Xlating FIR Filter -> Feed Forward AGC ->
Correlation Estimator -> (Your custom LMS DA filter block) -> ...


> *Steps*
> 
> 1 I take the original and modified signal and rescale the modified signal to
> match the original.
> 
> At the moment I use a very naive approach which is to take the absolute sum
> of the 256 samples for both signals and from this calculate a simple scale
> factor. I think this should be OK where I have narrow band noise, but it may
> fail badly in other cases where the noise levels are high.
> 
> 2 Next I take the FFT of the two signals (256 samples).
> 
> 3 Calculate the noise
> 
> Using the difference between the FFTs, I then calculate the noise power.
> 
> *Two questions?*
> 
> 1 The rescaling method is very basic, using absolute accumulated sums. Does
> GNU radio have any blocks, which could perform this auto-scaling more
> effectively?

GNURadio has several AGC blocks.  They all have their quirks.  Pick one
an try to make it work.


> 2 Using the basic difference between the FFT's, such as the absolute
> magnitude difference, should provide a starting point for calculating the
> noise power. Again is this naive?

Noise power and noise density have specific meanings which I don't think
match what you're thinking about here.  AFAICT, you want to know the
power of an in-channel narrowband interference (so that you can
ultimately filter it out).

Looking at FFT's will give you a feel for the situation, but it's kind
of a blunt instrument, if you plan of filtering by direct FFT bin
scaling or excision.

It really sounds like what you want is an adaptive equalizer (aka
adaptive filter).

There's lots of existing literature on equalizers.
This lecture is still a little too advanced for most folks, but it has
the basic concepts covered clearer than most others I could find on
Google: 
http://www.eecg.toronto.edu/~johns/nobots/courses/ece1392/equalization2.pdf


Section 14.6 of this book describes the LMS algorithm:
https://www.amazon.com/Mathematical-Methods-Algorithms-Signal-Processing/dp/0201361868

And here is a PDF copy I spotted on the internet (click at your own
risk):
https://www.u-cursos.cl/usuario/834c0e46b93fd72fd8408c492af56f8d/mi_blog/r/4%29_Todd_Moon_Mathematical_Methods_and_Algorithms_for_Signal_Processing.pdf

-Andy



___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio