RE: DMARC ViolationAS21299 - 46.42.196.0/24 ASN prepending 255 times

[Adam Thompson]

I partially agree with you, but… 10 is way too aggressive.  I’m already seeing 
a “true” internet diameter of up to 13 AS hops today.  Here’s the data I’m 
using to form that opinion.

[ NOTE: my tooling didn’t handle BGP confederations in the RIB dump properly.  
It’s sufficiently rare (only 1009 routes in total) that I’m not going back to 
square one to compensate for that, the data’s not noticeably skewed because of 
it, with one exception: the len=15 and len=16 entries in the de-prepended tale 
are both bogus.  I left them in in the spirit of full disclosure. ]

This is the AS path-length distribution in my RIB as of a few minutes ago.  
Like everyone else, I do some moderately strange things for  including 
artificially locally prepending routes from some of my upstreams, so I’m quite 
certain no-one else’s will exactly match mine.  The overall shape of the 
distribution, though, should be very similar, probably with the peak shifted 
left or right slightly.

Pathlen
Count
1
1864
2
77314
3
223030
4
248878
5
729240
6
627685
7
224092
8
105225
9
60962
10
50201
11
22856
12
11914
13
7224
14
5423
15
4421
16
2756
17
1577
18
945
19
422
20
669
21
477
22
116
23
70
24
75
25
163
26
33
27
40
28
5
29
41
30
5
31
5
33
1
35
12
38
5
39
2
40
1
55
1

I took a look at some of the outliers, and they are, in the extreme cases, 
somewhat insane, with over 20 prepends or more in those last few cases.  
However, at least a few of the pathlen>=20 entries could be legit.  Could be.  
Not saying they are, just that there are some paths that aren’t immediately 
obvious BS and it would take a lot more time & effort to figure that out.

I then eliminated ALL the prepends, remote, and local and transit, and that 
length distribution changes to look like this, which should reflect the “true” 
diameter of the internet as seen from here:

Pathlen
Count
1
18950
2
223149
3
563843
4
824648
5
532985
6
165056
7
47559
8
16759
9
8430
10
3746
11
519
12
233
13
6
15
1
16
2
(Interestingly, the shape of the distribution doesn’t change significantly.  
That hints that most people could be doing prepending in roughly the same way.)

Using your suggestion of as AS-PATH length of 10 as a cutoff, you’d be dropping 
~4.5% (109,460 routes here) of the total RIB.  But even in my artificially 
de-prepended dataset, I still see 4,507 presumably-legitimate routes with a 
path length >= 10.

I’m not saying a threshold is a bad idea.  (I’m on the fence.)  I AM saying 
that using 10 as your cutoff point is too aggressive, and in my opinion, way 
too aggressive.
Based on the de-prepended dataset, the approximate diameter of my internet (not 
necessarily yours) is 13 AS hops.  (Not 15 or 16, see note above.)

Since it’s entirely public data, here’s the raw paths from me to AS 270606, 
prepends and all, that generated my “true” diameter of 13, as recorded in my 
looking glass:

  *   I* N 177.37.16.0/22 206.211.216.51 100 0 7122 7122 577 6461 52320 53087 
262740 262740 262740 262740 262740 262740 266097 268868 61568 26615 10429 
263144 270606
  *   I* N 177.37.16.0/22 206.211.216.52 100 0 7122 7122 577 6461 52320 53087 
262740 262740 262740 262740 262740 262740 266097 268868 61568 26615 10429 
263144 270606
  *   I* N 177.37.16.0/22 216.73.71.131 100 0 6327 6327 6327 6461 52320 53087 
262740 262740 262740 262740 262740 262740 266097 268868 61568 26615 10429 
263144 270606
(Formatting didn’t translate well… the locally-prepended AS path for these 3 
examples starts with 7122 7122 577, or 6327 6327 6327.)

I do not have any alternate path to those prefixes in my RIB.  The deduped 
13-long path is “7122 577 6461 52320 53087 262740 266097 268868 61568 26615 
10429 263144 270606”.

Do I really care if education users in Manitoba can reach R3 Telecom users in 
Brazil?  Probably not (although there are quite a lot of Brazilian 
international students here, so maybe?) but this demonstrates that the 
“diameter of the internet” is absolutely not well under 10.

I’ll point out, since I speculated about this in a previous email, the most 
extreme case discussed here was purely on commercial internet, and did not 
involve NREN paths at all.  I went looking for NREN-style prepending and found 
several examples buried in the middle of the distribution.  That presumptive 
root cause doesn’t appear, at least at first glance, to contribute much to the 
extreme path lengths I see.

Imposing a limit like this has a strong precedent: Sprint(?)’s cap at accepting 
only /24 and larger, waay back.  Like that action, this max-path-len 
proposal is likely to be discriminatory because it implicitly favours ASNs 
located in areas with good connectivity to the “core”, i.e. mainly the eastern 
US and some areas of western Europe.  (I also did not examine the entire path 
to AS270606 for sanity – it’s not always about simple availability, sometimes 
perverse incentives play a strong role, too.)
If I start looking at the 233 routes of “true” distance 12, where will they be 
located?  Or the 5

Re: Let's Focus on Moving Forward Re: V6 still not supported

[Masataka Ohta]

Owen DeLong wrote:


As I repeatedly pointed out, end to end NAT is clean preserving
the universal peer to peer nature of the Internet.


Nope… It really isn’t.


Wrong.


The problem of audit trail opacity is still a major issue with any form
of stateful NAT.


How poorly you understand NAT.

As I wrote in my draft:

   Depending on how port numbers are shared, there are static and
   dynamic E2ENAT or combinations of them. With static E2ENAT, an end
   host is assigned port numbers statically, which is necessary for a
   server with a stable IP address and a port number.

static E2ENAT is not, with your questionable terminology, stateful.

It is even possible to construct legacy NAT which dynamically,
thus statefully, assign ports only from some static range,
which does not need state maintenance, for each private IP
address.

Masataka Ohta

Re: PoE, Comcast Modems, and Service Outages

[Randy Bush]

> he said I needed to disable PoE because it messes with the Comcast
> modems and he can see "buildups" in his graphs that show power is
> "leaking" to the Comcast modem every 24 hours.

revealing the critical failure with comcast support; they do not share
what they are smoking

randy

---
ra...@psg.com
`gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
signatures are back, thanks to dmarc header butchery

Re: PoE, Comcast Modems, and Service Outages

[Sabri Berisha]

- On Mar 29, 2022, at 2:46 PM, Joe Greco jgr...@ns.sol.net wrote:

Hi,

> So if you want the $100 test to eliminate PoE electrical effects, get
> a pair of media converters and run fiber between them.  Put the CPE on
> the far end.  Optimize as appropriate if you have SFP-capable switches.

But now the modem will suffer from excessive gas...

https://www.newscientist.com/article/2313629-a-gas-made-from-light-becomes-easier-to-compress-as-you-squash-it/

"Did you hear that?" -"What?" "That's your modem, farting. Time to reboot 
again!"

Thanks,

Sabri

Re: PoE, Comcast Modems, and Service Outages

[Blake Hudson]

Generally anything inside the customer premise (including wiring) is the 
customer’s responsibility. If your coax runs across a fluorescent light 
fixture, that’s on you. If your coax is RG59, it’s on you to replace with RG6 
quad shield. 

Maybe the cable operator will work with you, maybe not. I’ve had some techs 
replace splitters or make custom length user cables. 

And yes, the RF stats I mentioned are all on the HFC interface of the cable 
modem (or the CMTS). Customer facing Ethernet stats may not even be tracked by 
Comcast. 

--Blake 

> On Mar 29, 2022, at 5:29 PM, Aaron de Bruyn  wrote:
> 
> 
> Thanks Blake,
> 
> As I understand it all that stuff is on the "cable provider" side of the CPE 
> and (within reason) it's up to the provider to deal with the signals arriving 
> on the cable side of the modem.
> i.e. if it was a blower or something in our suite that was causing RF 
> interference, the provider might work with us to move the modem or the cable 
> run.
> 
> -A
> 
> On Tue Mar 29, 2022, 09:59 PM GMT, Blake Hudson wrote:
> 
> On 3/29/2022 3:24 PM, Joe Greco wrote:
> He's got graphs showing it every 24 hours? Liar, liar, pants on fire,
> lazy SOB is looking for an excuse to clear you off the line. Where the
> heck does this "24 hour" cycle even come from? What SNMP OID is there
> for "ghostly PoE build-up"? What crontab is there that would clear out
> such buildups in the router's daily run? What capacitor would store up
> juice for precisely 24 hours? What's the mechanism here? CURIOUS MINDS
> WANT TO KNOW!
> 
> 
> Taken at face value, I assume the tech would be looking at historical 
> signal graphs (we keep them for cable networks for each CM) that record 
> stats like FEC, SNR, and signal strength. For aerial runs it's common to 
> see some change throughout the day due to warming and cooling. These 
> look like waves with peaks and valleys around 4PM/4AM and generally 
> affect all customers in a service area equally. Sometimes there will be 
> a device at a customer premise that causes interference with a CM, 
> something like a motor or tool. These could absolutely be on a 24hr 
> cycle (think of a programmable thermostat kicking on the blower fan in 
> your HVAC at the same time every day).
> 
> As Joe said, there's no SNMP MIB for PoE buildup. There are well 
> documented MIBs for DOCSIS to cover standard signal level, quality, or 
> similar. The cause of that signal strength or quality can be myriad. 
> This Comcast tech has likely climbed the ladder of inference several 
> steps too far.
> 
> 

Internet Storm Center says Russia hijacking Twitter's BGP

[Jay R. Ashworth]

https://isc.sans.edu/diary/rss/28488

-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: PoE, Comcast Modems, and Service Outages

[Aaron de Bruyn via NANOG]

Thanks Blake,

As I understand it all that stuff is on the "cable provider" side of the
CPE and (within reason) it's up to the provider to deal with the signals
arriving on the cable side of the modem.
i.e. if it was a blower or something in our suite that was causing RF
interference, the provider might work with us to move the modem or the
cable run.

-A

On Tue Mar 29, 2022, 09:59 PM GMT, Blake Hudson  wrote:


On 3/29/2022 3:24 PM, Joe Greco wrote:

He's got graphs showing it every 24 hours? Liar, liar, pants on fire,
lazy SOB is looking for an excuse to clear you off the line. Where the
heck does this "24 hour" cycle even come from? What SNMP OID is there
for "ghostly PoE build-up"? What crontab is there that would clear out
such buildups in the router's daily run? What capacitor would store up
juice for precisely 24 hours? What's the mechanism here? CURIOUS MINDS
WANT TO KNOW!


Taken at face value, I assume the tech would be looking at historical
signal graphs (we keep them for cable networks for each CM) that record
stats like FEC, SNR, and signal strength. For aerial runs it's common to
see some change throughout the day due to warming and cooling. These
look like waves with peaks and valleys around 4PM/4AM and generally
affect all customers in a service area equally. Sometimes there will be
a device at a customer premise that causes interference with a CM,
something like a motor or tool. These could absolutely be on a 24hr
cycle (think of a programmable thermostat kicking on the blower fan in
your HVAC at the same time every day).

As Joe said, there's no SNMP MIB for PoE buildup. There are well
documented MIBs for DOCSIS to cover standard signal level, quality, or
similar. The cause of that signal strength or quality can be myriad.
This Comcast tech has likely climbed the ladder of inference several
steps too far.

Re: PoE, Comcast Modems, and Service Outages

[Blake Hudson]

On 3/29/2022 3:24 PM, Joe Greco wrote:

He's got graphs showing it every 24 hours?  Liar, liar, pants on fire,
lazy SOB is looking for an excuse to clear you off the line.  Where the
heck does this "24 hour" cycle even come from?  What SNMP OID is there
for "ghostly PoE build-up"?  What crontab is there that would clear out
such buildups in the router's daily run?  What capacitor would store up
juice for precisely 24 hours?  What's the mechanism here?  CURIOUS MINDS
WANT TO KNOW!




Taken at face value, I assume the tech would be looking at historical 
signal graphs (we keep them for cable networks for each CM) that record 
stats like FEC, SNR, and signal strength. For aerial runs it's common to 
see some change throughout the day due to warming and cooling. These 
look like waves with peaks and valleys around 4PM/4AM and generally 
affect all customers in a service area equally. Sometimes there will be 
a device at a customer premise that causes interference with a CM, 
something like a motor or tool. These could absolutely be on a 24hr 
cycle (think of a programmable thermostat kicking on the blower fan in 
your HVAC at the same time every day).


As Joe said, there's no SNMP MIB for PoE buildup. There are well 
documented MIBs for DOCSIS to cover standard signal level, quality, or 
similar. The cause of that signal strength or quality can be myriad. 
This Comcast tech has likely climbed the ladder of inference several 
steps too far.

Re: PoE, Comcast Modems, and Service Outages

[Brie]

On 3/29/22 2:24 PM, Aaron C. de Bruyn wrote:
On Tue, Mar 29, 2022 at 12:20 PM Brie > wrote:


Unifi/EdgeSwitch?


Yeah.  Unfortunately.  USW-24-250.



Oh, I have quite a few of them in service.  They work great in my 
experience as long as you don't shove them in hot closets or cabinets 
and cause them to overheat.  Also, firmware has gotten a bit better over 
the years.


I have a lot of background in Ubnt stuff, so I'm a bit biased here.




Yeah, you know when 24v passive POE is turned on because it kills the
port on the other end that aren't designed to handle it.  Your router
would likely have a dead eth port on it.


I've never tested it with one of my routers, but a tech did accidentally 
test it on a UniFi AP.  I'm still not sure how he ignored the warning 
about sending 24 volts down the line, but the WAP didn't like it and 
decided to refuse to work with anyone ever again. ;)



Yup, that's normal if its a more recent AP that wasn't designed for 24V 
passive (pretty much anything recent AP wise).


I've seen techs destroy ethernet ports on brand new laptops doing that. 
 Very expensive lesson :)


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Let's Focus on Moving Forward Re: V6 still not supported

[Owen DeLong via NANOG]

> On Mar 26, 2022, at 17:30 , Masataka Ohta  
> wrote:
> 
> Owen DeLong via NANOG wrote:
> 
>> It still looks like NAT to me.
> 
> Almost all the people, perhaps other than you, accept NAT
> as is to keep IPv4 Internet or as part of transition
> plan from IPv4 to IPv6.
> 
>> NAT is a disgusting hack and destroys the universal peer to peer
>> nature of the internet in favor of a consumer/provider model.
> 
> As I repeatedly pointed out, end to end NAT is clean preserving
> the universal peer to peer nature of the Internet.

Nope… It really isn’t.

The problem of audit trail opacity is still a major issue with any form
of stateful NAT.

Owen

Re: IPv6 Only - was Re: Let's Focus on Moving Forward Re: V6 still not supported re: 202203261833.AYC

[jim deleskie]

If then industry still hasn't adopted v6 full in 25 years maybe it's v6
that should be given up it, that it clearly wasn't what customers wanted.
Perhaps we should should have a small group working on the next iteration.

-jim

On Tue, Mar 29, 2022, 5:54 PM Jacques Latour  wrote:

> So, in 25, 50 or 100 years from now, are we still going to be dual stack
> IPv4/IPv6?
>
> When are we going to give up on IPv4?
>
> People can run IPv4 all they want inside their networks for 1000s of years.
>
> What will it take to be IPv6 only?
>
>
>
> 😊
>
>
>
> *From:* NANOG  *On Behalf
> Of *Owen DeLong via NANOG
> *Sent:* March 29, 2022 3:52 PM
> *To:* Abraham Y. Chen 
> *Cc:* NANOG 
> *Subject:* [EXT] Re: Let's Focus on Moving Forward Re: V6 still not
> supported re: 202203261833.AYC
>
>
>
> Submit an Internet draft, same as any other IP related enhancement gets
> introduced.
>
>
>
> What you’re really complaining about is that it’s been virtually
> impossible to gain consensus to move anything IPv4 related forward in the
> IETF since at least 2015.
>
>
>
> Well… It’s a consensus process. If your idea isn’t getting consensus, then
> perhaps it’s simply that the group you are seeking consensus from doesn’t
> like your idea.
>
>
>
> Your inability to convince the members of the various working groups that
> your idea has merit isn’t necessarily a defect in the IETF process… It
> might simply be a lack of merit in your ideas.
>
>
>
> Owen
>
>
>
>
>
> On Mar 26, 2022, at 15:43 , Abraham Y. Chen  wrote:
>
>
>
> Hi, Justin:
>
>
>
> 1)"... no one is stopping anyone from working on IPv4 ... ":
> After all these discussions, are you still denying this basic issue? For
> example, there has not been any straightforward way to introduce IPv4
> enhancement ideas to IETF since at least 2015. If you know the way, please
> make it public. I am sure that many are eager to learn about it. Thanks.
>
>
>
> Regards,
>
>
>
>
>
> Abe (2022-03-26 18:42)
>
>
>
>
>
>
>
>
>
> On 2022-03-26 11:20, Justin Streiner wrote:
>
> While the Internet is intended to allow the free exchange of information,
> the means of getting that information from place to place is and has to be
> defined by protocols that are implemented in a consistent manner (see: BGP,
> among many other examples).  It's important to separate the ideas from the
> plumbing.
>
>
>
> That said, no one is stopping anyone from working on IPv4, so what
> personal freedoms are being impacted by working toward deploying IPv6, with
> an eye toward sunsetting IPv4 in the future?
>
>
>
> Keep in mind that IPv4 started out as an experiment that found its way
> into wider use.  It's a classic case of a test deployment that suddenly
> mutated into a production service.  Why should we continue to expend effort
> to perpetuate the sins of the past, rather work toward getting v6 into
> wider use?
>
>
>
> Is IPv6 a perfect protocol?  Absolutely not, but it addresses the key pain
> point of IPv4 - address space exhaustion.
>
>
>
> Thank you
>
> jms
>
>
>
> On Sat, Mar 26, 2022 at 9:35 AM Abraham Y. Chen  wrote:
>
>
>
> 3)Re: Ur. Pts. 5) & 6):I believe that there is a philosophic /
> logic baseline that we need to sort out, first. That is, we must keep in
> mind that the Internet community strongly promotes "*personal freedom*".
> Assuming that by stopping others from working on IPv4 will shift their
> energy to IPv6 is totally contradicting such a principle. A project
> attracts contributors by its own merits, not by relying on artificial
> barriers to the competitions. Based on my best understanding, IPv6 failed
> right after the decision of "not emphasizing the backward compatibility
> with IPv4". It broke one of the golden rules in the system engineering
> discipline. After nearly three decades, still evading such fact, but
> defusing IPv6 issues by various tactics is the real impedance to progress,
> not only to IPv4 but also to IPv6.
>
>
>
>
>

IPv6 Only - was Re: Let's Focus on Moving Forward Re: V6 still not supported re: 202203261833.AYC

[Jacques Latour]

So, in 25, 50 or 100 years from now, are we still going to be dual stack 
IPv4/IPv6?
When are we going to give up on IPv4?
People can run IPv4 all they want inside their networks for 1000s of years.
What will it take to be IPv6 only?

😊

From: NANOG  On Behalf Of Owen 
DeLong via NANOG
Sent: March 29, 2022 3:52 PM
To: Abraham Y. Chen 
Cc: NANOG 
Subject: [EXT] Re: Let's Focus on Moving Forward Re: V6 still not supported re: 
202203261833.AYC

Submit an Internet draft, same as any other IP related enhancement gets 
introduced.

What you’re really complaining about is that it’s been virtually impossible to 
gain consensus to move anything IPv4 related forward in the IETF since at least 
2015.

Well… It’s a consensus process. If your idea isn’t getting consensus, then 
perhaps it’s simply that the group you are seeking consensus from doesn’t like 
your idea.

Your inability to convince the members of the various working groups that your 
idea has merit isn’t necessarily a defect in the IETF process… It might simply 
be a lack of merit in your ideas.

Owen



On Mar 26, 2022, at 15:43 , Abraham Y. Chen 
mailto:ayc...@avinta.com>> wrote:

Hi, Justin:

1)"... no one is stopping anyone from working on IPv4 ... ":   After 
all these discussions, are you still denying this basic issue? For example, 
there has not been any straightforward way to introduce IPv4 enhancement ideas 
to IETF since at least 2015. If you know the way, please make it public. I am 
sure that many are eager to learn about it. Thanks.

Regards,


Abe (2022-03-26 18:42)




On 2022-03-26 11:20, Justin Streiner wrote:
While the Internet is intended to allow the free exchange of information, the 
means of getting that information from place to place is and has to be defined 
by protocols that are implemented in a consistent manner (see: BGP, among many 
other examples).  It's important to separate the ideas from the plumbing.

That said, no one is stopping anyone from working on IPv4, so what personal 
freedoms are being impacted by working toward deploying IPv6, with an eye 
toward sunsetting IPv4 in the future?

Keep in mind that IPv4 started out as an experiment that found its way into 
wider use.  It's a classic case of a test deployment that suddenly mutated into 
a production service.  Why should we continue to expend effort to perpetuate 
the sins of the past, rather work toward getting v6 into wider use?

Is IPv6 a perfect protocol?  Absolutely not, but it addresses the key pain 
point of IPv4 - address space exhaustion.

Thank you
jms

On Sat, Mar 26, 2022 at 9:35 AM Abraham Y. Chen 
mailto:ayc...@avinta.com>> wrote:

3)Re: Ur. Pts. 5) & 6):I believe that there is a philosophic / logic 
baseline that we need to sort out, first. That is, we must keep in mind that 
the Internet community strongly promotes "personal freedom". Assuming that by 
stopping others from working on IPv4 will shift their energy to IPv6 is totally 
contradicting such a principle. A project attracts contributors by its own 
merits, not by relying on artificial barriers to the competitions. Based on my 
best understanding, IPv6 failed right after the decision of "not emphasizing 
the backward compatibility with IPv4". It broke one of the golden rules in the 
system engineering discipline. After nearly three decades, still evading such 
fact, but defusing IPv6 issues by various tactics is the real impedance to 
progress, not only to IPv4 but also to IPv6.

Re: PoE, Comcast Modems, and Service Outages

[Matt Ertle]

> On Mar 29, 2022, at 5:46 PM, Joe Greco  wrote:
> 
> So if you want the $100 test to eliminate PoE electrical effects, get
> a pair of media converters and run fiber between them.  Put the CPE on
> the far end.  Optimize as appropriate if you have SFP-capable switches.
> 
> ... JG
> — 
Bingo ^^^

As a few others have said, I’d be more inclined to feel like this was a real 
thing if the tech had mentioned checking grounding, etc and not just said all 
the PoE has to go away for the modem to function correctly.  If it is truly a 
power issue there are a bunch of ways to isolate/troubleshoot/fix it.  
Thanks,
Matt

Matt Ertle
Manager - Network Operations
Eastern Shore of Virginia Broadband Authority
(w) 757 414.0304
(f)  757 656.7066
mer...@esvba.com

Re: PoE, Comcast Modems, and Service Outages

[Niels Bakker]

On Tue, Mar 29, 2022 at 11:21:28AM -0700, Aaron de Bruyn via NANOG wrote:
When I said "yes", he said I needed to disable PoE because it 
messes with the Comcast modems and he can see "buildups" in his 
graphs that show power is "leaking" to the Comcast modem every 24 
hours.


This guy must have a side business selling crystals, or those Faraday cages 
to contain the bad type of WiFi or 3/4/5G radiation, or some other scam.



* jgr...@ns.sol.net (Joe Greco) [Tue 29 Mar 2022, 20:52 CEST]:
People who can, do.  People who can't, tech support. Worth 
remembering.


I think this smearing of an entire line of work is uncalled for.


-- Niels.

Re: PoE, Comcast Modems, and Service Outages

[Kord Martin]

On 2022-03-29 5:46 p.m., Joe Greco wrote:

On Tue, Mar 29, 2022 at 03:42:54PM -0400, Josh Luthman wrote:

There's a certain manufacturer of TDD radio where the CPU clock is at the
same frequency as what Verizon's enodeB will transmit.  Even at miles away,
it can and will cause PIM issues.  Again, don't rule it out.

I'm not ruling anything out, but on the flip side, here in this group
of professional networkers, you'd think lots of people would have piped
up by now with "me too"'s if PoE ghosts killing cable CPE on a 24 hour
cycle were a common thing.


As a small DOCSIS operator, I suppose my sample set isn't large enough 
to be significant but I can tell you PoE has never been an issue with 
our modems.


We've had a significantly higher failure rate due to things such as 
customers driving nails through the modem in an attempt to mount them to 
walls, and concerned citizens shooting birdshot into our overhead 
distribution in an attempt to curb the rodent population.


K

Re: PoE, Comcast Modems, and Service Outages

[Aaron C. de Bruyn via NANOG]

On Tue, Mar 29, 2022 at 1:12 PM Joe Greco  wrote:

> So if you want the $100 test to eliminate PoE electrical effects, get
> a pair of media converters and run fiber between them.  Put the CPE on
> the far end.  Optimize as appropriate if you have SFP-capable switches.


Sure--that would shoot down the "leaking non-existent PoE across a
motherboard and out another NIC" theory, but I was more thinking along the
lines of something like PoE causing RF interference or something.
I mean it's DC not AC so...it wouldn't be putting out a modulating signal
that interferes? ...honestly that's outside my knowledge domain.

-A

Re: PoE, Comcast Modems, and Service Outages

[Jay Hennigan]

On 3/29/22 11:21, Aaron de Bruyn via NANOG wrote:
I just got off the phone with a Comcast tech, and wanted to double-check 
my sanity.


Somehow in the last 6 months I've managed to reach the exact same rep 
twice when dealing with an outage or a degraded service event.


I asked him to remotely reboot the modem because there was high packet loss.

Both times I've talked with him, he noted the high packet loss, started 
to reboot the modem, and then asked me point-blank if we had any PoE 
switches on our network.


When I said "yes", he said I needed to disable PoE because it messes 
with the Comcast modems and he can see "buildups" in his graphs that 
show power is "leaking" to the Comcast modem every 24 hours.


Obligatory relevant xkcd: https://xkcd.com/806/

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: PoE, Comcast Modems, and Service Outages

[Aaron C. de Bruyn via NANOG]

On Tue, Mar 29, 2022 at 12:20 PM Brie  wrote:

> Unifi/EdgeSwitch?
>

Yeah.  Unfortunately.  USW-24-250.


> Yeah, you know when 24v passive POE is turned on because it kills the
> port on the other end that aren't designed to handle it.  Your router
> would likely have a dead eth port on it.
>

I've never tested it with one of my routers, but a tech did accidentally
test it on a UniFi AP.  I'm still not sure how he ignored the warning about
sending 24 volts down the line, but the WAP didn't like it and decided to
refuse to work with anyone ever again. ;)


-A

Re: PoE, Comcast Modems, and Service Outages

[Aaron de Bruyn via NANOG]

Just to be clear Josh, I'm not insulting him.

I find the situation extremely difficult to believe based on my (possibly
incorrect) understanding of how PoE works and very (very!) basic knowledge
of things like RF interference—especially when it comes to Cable networks.

I mean, the call literally went like this:
"Thank you for calling Comcast this is , how can I help you?"
"Hey, can you remotely reboot the modem on account 12345? We're seeing high
packet loss and latency starting about 10 minutes ago."
"Yeah...uh...do you have a PoE switch at that location?"

When you hear hoof beats, look for horses, not zebras. As a first
troubleshooting step, I certainly wouldn't jump to "it's PoE". Granted, I
have no idea if Comcast has "PoE Buildup" graphs in their internal tools,
but based on my conversations with tons of other Comcast reps about tons of
other Comcast connections and never hearing one of them mention those
graphs, I'm leaning towards him lying through his teeth.

Lastly, the reboot of the Comcast modem "fixed" the issue.

I saw one of the IT guys from another office in the complex a few minutes
ago and he said their internet had problems at the same time. Comcast has
been out to the equipment room in the facility ~5 times over the last few
years to "adjust" things...so I'm still leaning towards this being
something more common like faulty equipment, bad signal levels, etc...and
not "It's because you have a PoE switch".

-A

On Tue Mar 29, 2022, 07:42 PM GMT, Josh Luthman
 wrote:

There's a certain manufacturer of TDD radio where the CPU clock is at the
same frequency as what Verizon's enodeB will transmit.  Even at miles away,
it can and will cause PIM issues.  Again, don't rule it out.

Maybe he's just looking for a simple answer that 99% of callers will accept
and it makes them happy.  When a customer of mine tells me they think it's
something and I know it's off, I just let them believe in their statement.
There's no reason to go after this tech and insult him, all that's doing is
making everyone miserable.

On Tue, Mar 29, 2022 at 3:26 PM Joe Greco  wrote:

> On Tue, Mar 29, 2022 at 03:07:47PM -0400, Josh Luthman wrote:
> > We've routinely seen where lines not even connected to the same circuit
> in
> > any way (ie an OTA antenna coax line and cat5 POE) cause issues with one
> > another.  As much as we would all love to have a perfect line in the
> sand,
> > there isn't.  Don't rule anything out until the issue is resolved.
> >
> > As someone that sees this in the field and watches people simply hate on
> > someone because there's a frustrating situation, it's worth taking a
> breath
> > before too upset.
>
> You can run cable lines next to A/C wiring and get problems too.  Or
> ethernet lines next to A/C wiring.  That does not justify wild claims
> about PoE such as what this tech was making, and until someone shows
> me a graph of "PoE buildups" observable via SNMP or whatever the
> cable company is using to graph trends, it seems pretty clear that
> this is a bogus answer.
>
> There's a lot of difference between "we observed this very specific kind
> of interference related to PoE in a particular circumstance" and the
> crazy generalizations being made by the tech.  Asking to please make sure
> your switch is grounded properly?  That'd be good.  Asking for PoE to be
> disabled on the port?  Yeah fine.  Suggesting separation of cables?
> Sure.  Checking for proper grounding of the ground block (on the cable
> inlet)?  Sure.  There's room for things to happen.
>
> I'm all for investigating with an open mind, but I draw the line at crazy.
>
> Given that so much of the world works on PoE, it seems like the other
> potential resolution would be to note that there's an implication here
> by the tech that Comcast's hardware is standards noncompliant and ask
> them what they plan to replace their cheap CPE with.
>
> ... JG
> --
> Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
> "The strain of anti-intellectualism has been a constant thread winding its
> way
> through our political and cultural life, nurtured by the false notion that
> democracy means that 'my ignorance is just as good as your
> knowledge.'"-Asimov
>

Re: PoE, Comcast Modems, and Service Outages

[Joe Greco]

On Tue, Mar 29, 2022 at 03:42:54PM -0400, Josh Luthman wrote:
> There's a certain manufacturer of TDD radio where the CPU clock is at the
> same frequency as what Verizon's enodeB will transmit.  Even at miles away,
> it can and will cause PIM issues.  Again, don't rule it out.

I'm not ruling anything out, but on the flip side, here in this group 
of professional networkers, you'd think lots of people would have piped
up by now with "me too"'s if PoE ghosts killing cable CPE on a 24 hour
cycle were a common thing.

> Maybe he's just looking for a simple answer that 99% of callers will accept
> and it makes them happy.  When a customer of mine tells me they think it's
> something and I know it's off, I just let them believe in their statement.

I'm unclear on how this is making the caller happy.

I'm trying to envision under what circumstances a customer site that has
purchased PoE switches, presumably to power PoE gear, would be delighted
to hear that their not-directly-connected PoE gear would need to be
removed, presumably replaced, and then, what?  Run extension cords and
bricks to all the access points, IP phones, cameras, door terminals, 
and other PoE-powered gear?

> There's no reason to go after this tech and insult him, 

I'd agree it isn't sporting, but on the other hand, a poster here asked
for an evaluation.  I did not immediately blow it off, but instead
tossed out some thoughts for consideration.  Then blew it off.  But I
am still pondering the issue.

> all that's doing is making everyone miserable.

I am guessing lots of people laughed.  It's an El Reg grade tale of
woe.  I have to assume the poster who asked is frustrated but trying
to resolve a real issue.

So if you want the $100 test to eliminate PoE electrical effects, get
a pair of media converters and run fiber between them.  Put the CPE on
the far end.  Optimize as appropriate if you have SFP-capable switches.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"The strain of anti-intellectualism has been a constant thread winding its way
through our political and cultural life, nurtured by the false notion that
democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

Re: Let's Focus on Moving Forward Re: V6 still not supported re: 202203261833.AYC

[Owen DeLong via NANOG]

Submit an Internet draft, same as any other IP related enhancement gets 
introduced.

What you’re really complaining about is that it’s been virtually impossible to 
gain consensus to move anything IPv4 related forward in the IETF since at least 
2015.

Well… It’s a consensus process. If your idea isn’t getting consensus, then 
perhaps it’s simply that the group you are seeking consensus from doesn’t like 
your idea.

Your inability to convince the members of the various working groups that your 
idea has merit isn’t necessarily a defect in the IETF process… It might simply 
be a lack of merit in your ideas.

Owen


> On Mar 26, 2022, at 15:43 , Abraham Y. Chen  wrote:
> 
> Hi, Justin:
> 
> 1)"... no one is stopping anyone from working on IPv4 ... ":   After 
> all these discussions, are you still denying this basic issue? For example, 
> there has not been any straightforward way to introduce IPv4 enhancement 
> ideas to IETF since at least 2015. If you know the way, please make it 
> public. I am sure that many are eager to learn about it. Thanks.
> 
> Regards,
> 
> 
> Abe (2022-03-26 18:42)
> 
> 
> 
> 
> On 2022-03-26 11:20, Justin Streiner wrote:
>> While the Internet is intended to allow the free exchange of information, 
>> the means of getting that information from place to place is and has to be 
>> defined by protocols that are implemented in a consistent manner (see: BGP, 
>> among many other examples).  It's important to separate the ideas from the 
>> plumbing.
>> 
>> That said, no one is stopping anyone from working on IPv4, so what personal 
>> freedoms are being impacted by working toward deploying IPv6, with an eye 
>> toward sunsetting IPv4 in the future?
>> 
>> Keep in mind that IPv4 started out as an experiment that found its way into 
>> wider use.  It's a classic case of a test deployment that suddenly mutated 
>> into a production service.  Why should we continue to expend effort to 
>> perpetuate the sins of the past, rather work toward getting v6 into wider 
>> use?
>> 
>> Is IPv6 a perfect protocol?  Absolutely not, but it addresses the key pain 
>> point of IPv4 - address space exhaustion.
>> 
>> Thank you
>> jms
>> 
>> On Sat, Mar 26, 2022 at 9:35 AM Abraham Y. Chen > > wrote:
>> 
>> 3)Re: Ur. Pts. 5) & 6):I believe that there is a philosophic / logic 
>> baseline that we need to sort out, first. That is, we must keep in mind that 
>> the Internet community strongly promotes "personal freedom". Assuming that 
>> by stopping others from working on IPv4 will shift their energy to IPv6 is 
>> totally contradicting such a principle. A project attracts contributors by 
>> its own merits, not by relying on artificial barriers to the competitions. 
>> Based on my best understanding, IPv6 failed right after the decision of "not 
>> emphasizing the backward compatibility with IPv4". It broke one of the 
>> golden rules in the system engineering discipline. After nearly three 
>> decades, still evading such fact, but defusing IPv6 issues by various 
>> tactics is the real impedance to progress, not only to IPv4 but also to IPv6.
>> 
>> 
>>  

Re: PoE, Comcast Modems, and Service Outages

[Josh Luthman]

There's a certain manufacturer of TDD radio where the CPU clock is at the
same frequency as what Verizon's enodeB will transmit.  Even at miles away,
it can and will cause PIM issues.  Again, don't rule it out.

Maybe he's just looking for a simple answer that 99% of callers will accept
and it makes them happy.  When a customer of mine tells me they think it's
something and I know it's off, I just let them believe in their statement.
There's no reason to go after this tech and insult him, all that's doing is
making everyone miserable.

On Tue, Mar 29, 2022 at 3:26 PM Joe Greco  wrote:

> On Tue, Mar 29, 2022 at 03:07:47PM -0400, Josh Luthman wrote:
> > We've routinely seen where lines not even connected to the same circuit
> in
> > any way (ie an OTA antenna coax line and cat5 POE) cause issues with one
> > another.  As much as we would all love to have a perfect line in the
> sand,
> > there isn't.  Don't rule anything out until the issue is resolved.
> >
> > As someone that sees this in the field and watches people simply hate on
> > someone because there's a frustrating situation, it's worth taking a
> breath
> > before too upset.
>
> You can run cable lines next to A/C wiring and get problems too.  Or
> ethernet lines next to A/C wiring.  That does not justify wild claims
> about PoE such as what this tech was making, and until someone shows
> me a graph of "PoE buildups" observable via SNMP or whatever the
> cable company is using to graph trends, it seems pretty clear that
> this is a bogus answer.
>
> There's a lot of difference between "we observed this very specific kind
> of interference related to PoE in a particular circumstance" and the
> crazy generalizations being made by the tech.  Asking to please make sure
> your switch is grounded properly?  That'd be good.  Asking for PoE to be
> disabled on the port?  Yeah fine.  Suggesting separation of cables?
> Sure.  Checking for proper grounding of the ground block (on the cable
> inlet)?  Sure.  There's room for things to happen.
>
> I'm all for investigating with an open mind, but I draw the line at crazy.
>
> Given that so much of the world works on PoE, it seems like the other
> potential resolution would be to note that there's an implication here
> by the tech that Comcast's hardware is standards noncompliant and ask
> them what they plan to replace their cheap CPE with.
>
> ... JG
> --
> Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
> "The strain of anti-intellectualism has been a constant thread winding its
> way
> through our political and cultural life, nurtured by the false notion that
> democracy means that 'my ignorance is just as good as your
> knowledge.'"-Asimov
>

Re: SentryPeer: A distributed peer to peer list of bad IP addresses and phone numbers collected via a SIP Honeypot

[Gavin Henry]

Hi all,

Come a long way since Nov:

https://github.com/SentryPeer/SentryPeer/releases/tag/v1.4.0

Peer to peer bad_actor replication is now released. Deutsche Telekom
"T-Pot - The All In One Honeypot Platform" included SentryPeer
(https://github.com/telekom-security/tpotce/tree/22.x) and Kali Linux
is coming - https://bugs.kali.org/view.php?id=7523#c15939

Would love to have some testers onboard!

Thanks,
Gavin.

Re: Let's Focus on Moving Forward Re: V6 still not supported Re: 20220326125.AYC

[Owen DeLong via NANOG]

Just because there is a small code snippet you found that prevents casting 
240/4 as unicast on an interface doesn’t mean that removing that code will 
magically make 240/4 usable in the entire stack.

It’s also important to note that there are at least a dozen IPv4 stacks in 
common use with differing code implementations and underlying assumptions baked 
into the code in various places. Your Q.E.D. is, in fact, a demonstration of 
the fact that you are still lack some background and experience in networking 
software.

The code you found may just be a safety valve to prevent the user from doing 
something stupid that will have undefined consequences down the line if 
executed. What you appear to have found is, quite likely, the equivalent of a 
buffer bounds check on input. Removing the check doesn’t inherently make the 
buffer infinite.

Owen

> On Mar 26, 2022, at 09:38 , Abraham Y. Chen  wrote:
> 
> Hi, Paul:
> 
> 1)" ...  may be in fact: /writing/* and */deploying/* the code  ... ":
> Having no idea why and how the 240/4 netblock became so mysteriously kept 
> away from being used while the IPv4 was officially already on its way to "Sun 
> Set", we started the conventional approach as you stated. It was quite 
> frustrating since we did not have the background in networking software. One 
> day, we came across a short program code fragment that did the function of 
> disabling 240/4 addressed IP packets. It is the "there exists an example" 
> moment for us, like proofing a mathematics theorem. After all, there was no 
> magic separating 240/4 from the rest of the IPv4 address pool to start with. 
> It cleared our mind about this particular task. Now, we only cite 
> this reference to challenge those software engineers who may state that using 
> the 240/4 in their code is a lot more involved.    Q.E.D.  😉
> 
> Regards,
> 
> 
> Abe (2022-03-26 12:37 EDT)
> 
> 
> 
> 
> On 2022-03-26 09:52, Paul Rolland wrote:
>> Hello,
>> 
>> On Sat, 26 Mar 2022 09:35:30 -0400
>> "Abraham Y. Chen"   wrote:
>> 
>>> touching the hardware, by implementing the EzIP technique (*/disabling/* 
>>> the program code that has been */disabling/* the use of the 240/4 
>>> netblock), an existing CG-NAT module becomes a RAN! As to universal 
>> Have you ever considered that this may be in fact:
>> 
>> */writing/* and */deploying/* the code that will allow the use of 240/4 the
>> way you expect 
>> 
>> 
>> Paul
> 
> 
>  
> 
>   Virus-free. www.avast.com 
> 
>  

Re: PoE, Comcast Modems, and Service Outages

[Joe Greco]

On Tue, Mar 29, 2022 at 03:07:47PM -0400, Josh Luthman wrote:
> We've routinely seen where lines not even connected to the same circuit in
> any way (ie an OTA antenna coax line and cat5 POE) cause issues with one
> another.  As much as we would all love to have a perfect line in the sand,
> there isn't.  Don't rule anything out until the issue is resolved.
> 
> As someone that sees this in the field and watches people simply hate on
> someone because there's a frustrating situation, it's worth taking a breath
> before too upset.

You can run cable lines next to A/C wiring and get problems too.  Or
ethernet lines next to A/C wiring.  That does not justify wild claims
about PoE such as what this tech was making, and until someone shows
me a graph of "PoE buildups" observable via SNMP or whatever the
cable company is using to graph trends, it seems pretty clear that
this is a bogus answer.

There's a lot of difference between "we observed this very specific kind
of interference related to PoE in a particular circumstance" and the
crazy generalizations being made by the tech.  Asking to please make sure
your switch is grounded properly?  That'd be good.  Asking for PoE to be
disabled on the port?  Yeah fine.  Suggesting separation of cables? 
Sure.  Checking for proper grounding of the ground block (on the cable
inlet)?  Sure.  There's room for things to happen.

I'm all for investigating with an open mind, but I draw the line at crazy.

Given that so much of the world works on PoE, it seems like the other
potential resolution would be to note that there's an implication here
by the tech that Comcast's hardware is standards noncompliant and ask
them what they plan to replace their cheap CPE with.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"The strain of anti-intellectualism has been a constant thread winding its way
through our political and cultural life, nurtured by the false notion that
democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

Re: Let's Focus on Moving Forward Re: V6 still not supported

[Owen DeLong via NANOG]

> On Mar 26, 2022, at 09:37 , Tom Beecher  wrote:
> 
> Have you ever considered that this may be in fact:
> 
> */writing/* and */deploying/* the code that will allow the use of 240/4 the
> way you expect
> 
> While Mr. Chen may have considered that, he has repeatedly hand waved that 
> it's 'not that big a deal.', so I don't think he adequately grasps the scale 
> of that challenge.

It’s certainly clear that he does not understand that in terms of cost-benefit 
ratio, the benefit of deploying his idea divided by the cost is a significantly 
lower number (in my estimation) than the much larger benefit of deploying IPv6 
divided by the rather limited remaining costs involved in doing so.

Owen

>  
> 
> On Sat, Mar 26, 2022 at 9:53 AM Paul Rolland  > wrote:
> Hello,
> 
> On Sat, 26 Mar 2022 09:35:30 -0400
> "Abraham Y. Chen" mailto:ayc...@avinta.com>> wrote:
> 
> > touching the hardware, by implementing the EzIP technique (*/disabling/* 
> > the program code that has been */disabling/* the use of the 240/4 
> > netblock), an existing CG-NAT module becomes a RAN! As to universal 
> 
> Have you ever considered that this may be in fact:
> 
> */writing/* and */deploying/* the code that will allow the use of 240/4 the
> way you expect 
> 
> 
> Paul

Re: PoE, Comcast Modems, and Service Outages

[Brie]

On 3/29/22 12:21 PM, Aaron de Bruyn via NANOG wrote:


Both times I've talked with him, he noted the high packet loss, started 
to reboot the modem, and then asked me point-blank if we had any PoE 
switches on our network.


This sounds like a guy who has created his own script for 'improving' 
his call resolution stats.  Lucky you!





When I said "yes", he said I needed to disable PoE because it messes 
with the Comcast modems and he can see "buildups" in his graphs that 
show power is "leaking" to the Comcast modem every 24 hours.



I just choked on my ice tea reading that.



For reference, our setup is:

Internal Network ←→ PoE Switch ←→ My Router (FreeBSD Box) ←→ Comcast Modem

I told him the Comcast modem isn't plugged into the PoE Switch, it's 
plugged into My Router (FreeBSD box) and My Router does not negotiate 
PoE+ and the switch shows PoE isn't being send to My Router's LAN port. 
While the switch is capable of outputting old-school 24v PoE, it must be 
specifically turned on for a port, and it's not enabled or used anywhere 
on the networks I manage.


Unifi/EdgeSwitch?

Yeah, you know when 24v passive POE is turned on because it kills the 
port on the other end that aren't designed to handle it.  Your router 
would likely have a dead eth port on it.





When provided with that information, the Comcast tech still insisted 
that the switch was sending PoE to My Router and it was "leaking 
through" to the Comcast modem and that's why every 4-6 weeks the Comcast 
modem needs to be reset. The tech insisted that switches that 
/are/PoE-capable /always/send PoE even if the device doesn't request it 
or negotiate it. Attempts to explain the difference between the old 
24-volt PoE and PoE+/++ were met with arguing that he's been in the 
industry for decades and I don't know what I'm talking about...and that 
all my problems would go away if I just disabled PoE everywhere on the 
switch.





Last time I checked, ethernet ports require magnetics for normal 
operation, so they're isolated anyway.


And, you are right on the POE negotiation - there is no POE until 
certain conditions are met on the line to signal to the switch that the 
device requires POE.


Yes, in theory there could be interference if POE was enabled, it's not 
likely these days, and there's no interference from POE if its not been 
negotiated.






Anyways, am I insane for thinking the tech was flat-out wrong? I 
mean...occasionally some really bizarre stuff happens in IT...but this 
seems extremely far-fetched and contrary to everything I know about the 
PoE standard.





Nope, not insane.  Tech is full of it and really needs to be retrained 
properly by a supervisor.  I'd be asking for said supervisor next time 
you call in and get him.



--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: PoE, Comcast Modems, and Service Outages

[Josh Luthman]

We've routinely seen where lines not even connected to the same circuit in
any way (ie an OTA antenna coax line and cat5 POE) cause issues with one
another.  As much as we would all love to have a perfect line in the sand,
there isn't.  Don't rule anything out until the issue is resolved.

As someone that sees this in the field and watches people simply hate on
someone because there's a frustrating situation, it's worth taking a breath
before too upset.

On Tue, Mar 29, 2022 at 2:50 PM Joe Greco  wrote:

> On Tue, Mar 29, 2022 at 11:21:28AM -0700, Aaron de Bruyn via NANOG wrote:
> > I just got off the phone with a Comcast tech, and wanted to double-check
> my
> > sanity.
> >
> > Somehow in the last 6 months I've managed to reach the exact same rep
> twice
> > when dealing with an outage or a degraded service event.
> >
> > I asked him to remotely reboot the modem because there was high packet
> loss.
> >
> > Both times I've talked with him, he noted the high packet loss, started
> to
> > reboot the modem, and then asked me point-blank if we had any PoE
> switches
> > on our network.
> >
> > When I said "yes", he said I needed to disable PoE because it messes with
> > the Comcast modems and he can see "buildups" in his graphs that show
> power
> > is "leaking" to the Comcast modem every 24 hours.
> >
> > For reference, our setup is:
> >
> > Internal Network ?? PoE Switch ?? My Router (FreeBSD Box) ??
> Comcast Modem
> >
> > I told him the Comcast modem isn't plugged into the PoE Switch, it's
> > plugged into My Router (FreeBSD box) and My Router does not negotiate
> PoE+
> > and the switch shows PoE isn't being send to My Router's LAN port. While
> > the switch is capable of outputting old-school 24v PoE, it must be
> > specifically turned on for a port, and it's not enabled or used anywhere
> on
> > the networks I manage.
> >
> > When provided with that information, the Comcast tech still insisted that
> > the switch was sending PoE to My Router and it was "leaking through" to
> the
> > Comcast modem and that's why every 4-6 weeks the Comcast modem needs to
> be
> > reset. The tech insisted that switches that *are* PoE-capable *always*
> send
> > PoE even if the device doesn't request it or negotiate it. Attempts to
> > explain the difference between the old 24-volt PoE and PoE+/++ were met
> > with arguing that he's been in the industry for decades and I don't know
> > what I'm talking about...and that all my problems would go away if I just
> > disabled PoE everywhere on the switch.
> >
> > Again, I double-checked the port and said "It's not sending PoE to my
> > router, but even if I were, I highly doubt PoE would leak through a PCI
> > card to the opposite side of the chassis to the on-board NIC and out to
> > your modem".
> >
> > He insisted it happened "all the time" and he had previously fried
> > equipment by plugging it into a PoE switch. He insisted that he's also
> > handled quite a few calls relating to this magic PoE problem over the
> years
> > and Comcast has internal tools that show graphs of how much PoE power
> > "builds up" inside their modems and he "can see a buildup in my router
> that
> > resets every 24 hours".
> >
> > I didn't have the heart to tell him that I manage about 40 networks that
> > have Comcast connections...and they *all* have identical FreeBSD boxes
> > acting as their router, and they are *all* using the exact same PoE
> > switches at every location with all ports set to PoE+...and we only have
> > degraded service or outages after ~30 days at ~3 locations.
> >
> > Slightly off-topic, but if I call Comcast about outages or degraded
> service
> > and any *other* tech but this guy answers, they all say "you need to
> unplug
> > your Comcast modem and plug it back in once every 3-4 weeks" and they act
> > like it's normal to reboot the modems every few weeks. In fact, last
> week I
> > wanted Comcast to check on a modem setting at one location and they said
> > the modem had been up for over 127 days and it should be rebooted. I said
> > "it's up and working fine, why would I reboot it?".
> >
> > Anyways, am I insane for thinking the tech was flat-out wrong? I
> > mean...occasionally some really bizarre stuff happens in IT...but this
> > seems extremely far-fetched and contrary to everything I know about the
> PoE
> > standard.
>
> That's ridiculous, as you already know.
>
> Their crappy equipment needing rebooting every few weeks, not ridiculous.
>
> Their purchasing gear from incompetent vendors who cannot be standards
> compliant for PoE PD negotiation, tragically plausible.
>
> Ethernet electronic differential signalling not being handled properly
> with respect to grounding or other issues, not unheard-of.
>
> Ghosts of PoE floating around a network through other devices, causing
> weird problems on the far side of properly installed and standards
> compliant
> gear, ah, super-unlikely, I'll go so far as to say nah, but in this
> audience
> I am positiv

Re: PoE, Comcast Modems, and Service Outages

[Joe Greco]

On Tue, Mar 29, 2022 at 11:21:28AM -0700, Aaron de Bruyn via NANOG wrote:
> I just got off the phone with a Comcast tech, and wanted to double-check my
> sanity.
> 
> Somehow in the last 6 months I've managed to reach the exact same rep twice
> when dealing with an outage or a degraded service event.
> 
> I asked him to remotely reboot the modem because there was high packet loss.
> 
> Both times I've talked with him, he noted the high packet loss, started to
> reboot the modem, and then asked me point-blank if we had any PoE switches
> on our network.
> 
> When I said "yes", he said I needed to disable PoE because it messes with
> the Comcast modems and he can see "buildups" in his graphs that show power
> is "leaking" to the Comcast modem every 24 hours.
> 
> For reference, our setup is:
> 
> Internal Network ?? PoE Switch ?? My Router (FreeBSD Box) ?? 
> Comcast Modem
> 
> I told him the Comcast modem isn't plugged into the PoE Switch, it's
> plugged into My Router (FreeBSD box) and My Router does not negotiate PoE+
> and the switch shows PoE isn't being send to My Router's LAN port. While
> the switch is capable of outputting old-school 24v PoE, it must be
> specifically turned on for a port, and it's not enabled or used anywhere on
> the networks I manage.
> 
> When provided with that information, the Comcast tech still insisted that
> the switch was sending PoE to My Router and it was "leaking through" to the
> Comcast modem and that's why every 4-6 weeks the Comcast modem needs to be
> reset. The tech insisted that switches that *are* PoE-capable *always* send
> PoE even if the device doesn't request it or negotiate it. Attempts to
> explain the difference between the old 24-volt PoE and PoE+/++ were met
> with arguing that he's been in the industry for decades and I don't know
> what I'm talking about...and that all my problems would go away if I just
> disabled PoE everywhere on the switch.
> 
> Again, I double-checked the port and said "It's not sending PoE to my
> router, but even if I were, I highly doubt PoE would leak through a PCI
> card to the opposite side of the chassis to the on-board NIC and out to
> your modem".
> 
> He insisted it happened "all the time" and he had previously fried
> equipment by plugging it into a PoE switch. He insisted that he's also
> handled quite a few calls relating to this magic PoE problem over the years
> and Comcast has internal tools that show graphs of how much PoE power
> "builds up" inside their modems and he "can see a buildup in my router that
> resets every 24 hours".
> 
> I didn't have the heart to tell him that I manage about 40 networks that
> have Comcast connections...and they *all* have identical FreeBSD boxes
> acting as their router, and they are *all* using the exact same PoE
> switches at every location with all ports set to PoE+...and we only have
> degraded service or outages after ~30 days at ~3 locations.
> 
> Slightly off-topic, but if I call Comcast about outages or degraded service
> and any *other* tech but this guy answers, they all say "you need to unplug
> your Comcast modem and plug it back in once every 3-4 weeks" and they act
> like it's normal to reboot the modems every few weeks. In fact, last week I
> wanted Comcast to check on a modem setting at one location and they said
> the modem had been up for over 127 days and it should be rebooted. I said
> "it's up and working fine, why would I reboot it?".
> 
> Anyways, am I insane for thinking the tech was flat-out wrong? I
> mean...occasionally some really bizarre stuff happens in IT...but this
> seems extremely far-fetched and contrary to everything I know about the PoE
> standard.

That's ridiculous, as you already know.

Their crappy equipment needing rebooting every few weeks, not ridiculous.

Their purchasing gear from incompetent vendors who cannot be standards
compliant for PoE PD negotiation, tragically plausible.

Ethernet electronic differential signalling not being handled properly
with respect to grounding or other issues, not unheard-of.

Ghosts of PoE floating around a network through other devices, causing
weird problems on the far side of properly installed and standards compliant
gear, ah, super-unlikely, I'll go so far as to say nah, but in this audience
I am positive someone has a counterexample SOMEWHERE, cuz, you know, the
world's a strange place and there's broken stuff out there.

He's got graphs showing it every 24 hours?  Liar, liar, pants on fire,
lazy SOB is looking for an excuse to clear you off the line.  Where the
heck does this "24 hour" cycle even come from?  What SNMP OID is there
for "ghostly PoE build-up"?  What crontab is there that would clear out
such buildups in the router's daily run?  What capacitor would store up
juice for precisely 24 hours?  What's the mechanism here?  CURIOUS MINDS
WANT TO KNOW!

Been doing PoE everywhere for years and this is the stupidest thing I've
heard this year so far in the networking category.

Ne

PoE, Comcast Modems, and Service Outages

[Aaron de Bruyn via NANOG]

I just got off the phone with a Comcast tech, and wanted to double-check my
sanity.

Somehow in the last 6 months I've managed to reach the exact same rep twice
when dealing with an outage or a degraded service event.

I asked him to remotely reboot the modem because there was high packet loss.

Both times I've talked with him, he noted the high packet loss, started to
reboot the modem, and then asked me point-blank if we had any PoE switches
on our network.

When I said "yes", he said I needed to disable PoE because it messes with
the Comcast modems and he can see "buildups" in his graphs that show power
is "leaking" to the Comcast modem every 24 hours.

For reference, our setup is:

Internal Network ←→ PoE Switch ←→ My Router (FreeBSD Box) ←→ Comcast Modem

I told him the Comcast modem isn't plugged into the PoE Switch, it's
plugged into My Router (FreeBSD box) and My Router does not negotiate PoE+
and the switch shows PoE isn't being send to My Router's LAN port. While
the switch is capable of outputting old-school 24v PoE, it must be
specifically turned on for a port, and it's not enabled or used anywhere on
the networks I manage.

When provided with that information, the Comcast tech still insisted that
the switch was sending PoE to My Router and it was "leaking through" to the
Comcast modem and that's why every 4-6 weeks the Comcast modem needs to be
reset. The tech insisted that switches that *are* PoE-capable *always* send
PoE even if the device doesn't request it or negotiate it. Attempts to
explain the difference between the old 24-volt PoE and PoE+/++ were met
with arguing that he's been in the industry for decades and I don't know
what I'm talking about...and that all my problems would go away if I just
disabled PoE everywhere on the switch.

Again, I double-checked the port and said "It's not sending PoE to my
router, but even if I were, I highly doubt PoE would leak through a PCI
card to the opposite side of the chassis to the on-board NIC and out to
your modem".

He insisted it happened "all the time" and he had previously fried
equipment by plugging it into a PoE switch. He insisted that he's also
handled quite a few calls relating to this magic PoE problem over the years
and Comcast has internal tools that show graphs of how much PoE power
"builds up" inside their modems and he "can see a buildup in my router that
resets every 24 hours".

I didn't have the heart to tell him that I manage about 40 networks that
have Comcast connections...and they *all* have identical FreeBSD boxes
acting as their router, and they are *all* using the exact same PoE
switches at every location with all ports set to PoE+...and we only have
degraded service or outages after ~30 days at ~3 locations.

Slightly off-topic, but if I call Comcast about outages or degraded service
and any *other* tech but this guy answers, they all say "you need to unplug
your Comcast modem and plug it back in once every 3-4 weeks" and they act
like it's normal to reboot the modems every few weeks. In fact, last week I
wanted Comcast to check on a modem setting at one location and they said
the modem had been up for over 127 days and it should be rebooted. I said
"it's up and working fine, why would I reboot it?".

Anyways, am I insane for thinking the tech was flat-out wrong? I
mean...occasionally some really bizarre stuff happens in IT...but this
seems extremely far-fetched and contrary to everything I know about the PoE
standard.

-A

Budget news: Proposed Puerto Rico FEMA distribution center (warehouse)

[Sean Donelan]

Program Change 1 – Caribbean Area Office and Warehouse Support:
Description

The FY 2023 Budget includes an increase of $1.4M for the staffing and 
operational costs of the Puerto Rico Distribution Centers (DC). Staff will
manage readiness, disaster response operations and the distribution of 
commodities for the FEMA DCs in Puerto Rico.


[...]
As a result of lesson learned responding during the 2017 and 2018 
hurricane seasons, analysis of observed maritime parameters has determined 
the requirement for one week of supply necessary to bridge a tailored 
disaster supply chain from CONUS.


[...]
To meet these readiness targets, the sites require staffing that are 
prepared to respond to all future major declarations, in lieu of 
being temporarily funded by a single, current major disaster declaration. 
This will allow sustained continuity of disaster support within the 
Caribbean. Organization structure of these facilities are based on models 
used at other FEMA distribution centers.


https://www.dhs.gov/sites/default/files/2022-03/Federal%20Emergency%20Management%20Agency_Remediated.pdf


Reminder: The annual president's budget proposal is mostly imaginary. The 
traditional joke is Congress tosses it in the trash bin immediately after 
the press conference. Nevertheless, its good to see some of the problems 
after the 2017/2018 caribbean hurricanes mentioned.


In 2017/2018 took over a month to get the disaster response supply chain 
from CONUS to the island territories (PR/USVI) working in spite of many 
heroic efforts at the time.

Re: Let's Focus on Moving Forward Re: V6 still not supported

[Tom Beecher]

>
> A traceroute from my machine to 240.1.2.3 goes through six routers at my
> ISP before stopping (probably at the first default-route-free router).
>

My experience is the opposite. My home edge router (dd-wrt) will pass it,
but nothing in my ISP's network will. $DayJob networks aren't worth
checking, as I know I have 224/3 bogonized.

I'd be curious to see the data you guys have collected on what it has been
confirmed to work on if that's available somewhere. ( More for curiosity's
sake ; I still think that making 224/3 universally available isn't worth
the effort it would take to make it happen. )

On Sat, Mar 26, 2022 at 9:42 PM John Gilmore  wrote:

> Tom Beecher  wrote:
> > > */writing/* and */deploying/* the code that will allow the use of
> 240/4 the
> > > way you expect
> >
> > While Mr. Chen may have considered that, he has repeatedly hand waved
> that
> > it's 'not that big a deal.', so I don't think he adequately grasps the
> > scale of that challenge.
>
> From multiple years of patching and testing, the IPv4 Unicast Extensions
> Project knows that 240/4 ALREADY WORKS in a large fraction of the
> Internet.  Including all the Linux servers and desktops, all the Android
> phones and tablets, all the MacOS machines, all the iOS phones, many of
> the home wifi gateways.  All the Ethernet switches.  And some less
> popular stuff like routers from Cisco, Juniper, and OpenWRT.  Most of
> these started working A DECADE AGO.  If others grasp the scale of the
> challenge better than we do, I'm happy to learn from them.
>
> A traceroute from my machine to 240.1.2.3 goes through six routers at my
> ISP before stopping (probably at the first default-route-free router).
>
> Today Google is documenting to its cloud customers that they should use
> 240/4 for internal networks.  (Read draft-schoen-intarea-unicast-240 for
> the citation.)  We have received inquiries from two other huge Internet
> companies, which are investigating or already using 240/4 as private
> IPv4 address space.
>
> In short, we are actually making it work, and writing a spec for what
> already works.  Our detractors are arguing: not that it doesn't work,
> but that we should instead seek to accomplish somebody else's goals.
>
> John
>
> PS: Mr. Abraham Chen's effort is not related to ours.  Our drafts are
> agnostic about what 240/4 should be used for after we enable it as
> ordinary unicast.  His EzIP overlay network effort is one that I don't
> fully understand.  What I do understand is that since his effort uses
> 240/4 addresses as the outer addresses in IPv4 packets, it couldn't work
> without reaching our goal first: allowing any site on the Internet to
> send unicast packets to or from 240.0.0.1 and having them arrive.
>
>

RE: IPv6 "bloat" history

[Pascal Thubert (pthubert) via NANOG]

Hello Ohta-san

> An ARP table entry can be created when an IP address is assigned during
> registration process and destroyed if the registration is invalidated.
> 
> Or, do I misunderstand anything?

You're perfectly correct. This is exactly what the registration would be for. 
I'm concerned about its adoption that I do not see coming on Wi-Fi/ Ethernet, 
even for v6 (SLAAC) where the problem is a lot worse*.

The IoT world already adopted the registration model, though. Millions of nodes 
on 802.15.4 radios are deployed to prove it works. We even redistribute RFC 
8505 in routing protocols that carry host routes in IoT networks (RPL), which 
is how we avoid your lookup issue on a wide low power mesh network.

Extending the registration to IPv4 is easy, we could even use mapped addresses 
and be done. But what magic will trigger attention / adoption when adoption is 
not showing in IPv6?

Keep safe;

Pascal


* APs today snoop DHCP; DHCP is observable and stateful, with a lifetime that 
allows to clean up. So snooping it is mostly good enough there. The hassle is 
the SL in SLAAC which causes broadcasts and is not deterministically 
observable; this problem is specific to IPv6. We already have the registration 
to avoid snooping DHCP and SLAAC; yet we do not observe any adoption in 
mainline APs and STAs. 







> -Original Message-
> From: Masataka Ohta 
> Sent: mardi 29 mars 2022 10:57
> To: Pascal Thubert (pthubert) ; nanog@nanog.org
> Subject: Re: IPv6 "bloat" history
> 
> Pascal Thubert (pthubert) wrote:
> 
> > I tried exactly what you suggested for IPv6 with RFC 8505 and 8929.
> > But to few people in mainstream networks realize what you just said.
> 
> I found, theoretically by reading 802.11 specification, broadcast/multicast
> reliability problem and reported to
> IPv6 WG about 20 years ago. So, I'm pleased to know that some people
> recognize it as a real problem and worked on it. Thank you.
> 
> > It started long long ago with the idea to use inverse ARP for the
> > registration, I guess it is still doable but I am not optimistic about
> > adoption considering that v6 is a lot worse with more addresses and
> > DAD.
> 
> Aren't IP addresses are assigned from APs? Then, the APs can construct ARP
> table without actually running ARP or inverse ARP, I'm afraid.
> 
> > We are editing the piece on proxy ARP at this very moment at .11me.
> > APs are indeed supposed to proxy both v4 and v6. What is less clear is
> > how they form a deterministic state for that.
> 
> An ARP table entry can be created when an IP address is assigned during
> registration process and destroyed if the registration is invalidated.
> 
> Or, do I misunderstand anything?
> 
>   Masataka Ohta

Re: IPv6 "bloat" history

[Masataka Ohta]

Pascal Thubert (pthubert) wrote:


I tried exactly what you suggested for IPv6 with RFC 8505 and 8929.
But to few people in mainstream networks realize what you just said.


I found, theoretically by reading 802.11 specification,
broadcast/multicast reliability problem and reported to
IPv6 WG about 20 years ago. So, I'm pleased to know
that some people recognize it as a real problem and
worked on it. Thank you.


It started long long ago with the idea to use inverse ARP for the
registration, I guess it is still doable but I am not optimistic
about adoption considering that v6 is a lot worse with more addresses
and DAD.


Aren't IP addresses are assigned from APs? Then, the
APs can construct ARP table without actually running
ARP or inverse ARP, I'm afraid.


We are editing the piece on proxy ARP at this very moment at .11me.
APs are indeed supposed to proxy both v4 and v6. What is less clear
is how they form a deterministic state for that.


An ARP table entry can be created when an IP address is assigned
during registration process and destroyed if the registration is
invalidated.

Or, do I misunderstand anything?

Masataka Ohta