Re: routing issue for verizon dsl customers in western massachusetts
Congratulations on your nat444 connection. I suspect a autoblocklist of sorts. They somehow always end up blocking the hosts you are using. I vaguely remember my watchguard firebox 1000 doing so. It was red too. Regards and good luck, Seth typed on a tiny touchscreen, why exactly? Steve Bohrer schreef: >On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote: > >> On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold >> wrote: >>> Over the past week, we've discovered that there is an issue with >>> the way >>> some Verizon DSL customers are being routed in Western >>> Massachusetts that is >>> preventing them from reaching my employers public IPs. The problem >>> is only >>> limited to Verizon DSL customers, everyone else can reach these IP >>> addresses >>> just fine. After many hours on the phone with Verizon tech support, I >>> finally managed to get myself and one of my coworker's home dsl >>> connections >>> switched from a "redback router" to a "juniper router" which >>> resolved the >>> issue, but only for us. > >[...] >> >> If you buy verizon services at your day job you can probably make >> noise through your sales droids better than here (sadly)... verizon >> likes to jump when customers have problems, if the customer is a large >> corporation or other 'important' customer. > > >That is just the problem! The college does not buy any Verizon network >stuff directly, so we don't really have any access to their support. >(We have a few cell phones, but not enough to be "important".) > >Brian Gold (who first posted) happens to have their DSL to his house, >and he was one of five who have reported the problem, so that gave him >a slight in. But the only techs he could reach as an "end user" were >not high enough up to fix this problem in a general way. After >pressing them for literally hours, he was able to get transfered to >their NOC, and get the problem resolved for his one address. But, they >would not give him the NOC contact, and he had to repeat this multi- >hour process to get it fixed for an other user. Verizon's DSL support >suggested that we get our bandwith provider involved, and so they >tried to pitch in, but they don't have any Verizon NOC contact either, >especially since this issue is purely within a small corner of >Verizon's DSL network, not on any of Verizon's links to our provider. > >This issue hits only a few Verizon DSL users in NW Mass. It does not >really seem like a routing problem, because the affected users can >reach many of the servers in our AS, but not some addresses. >Unfortunately, the "blocked" addresses include our web server and our >mail server, so our staff who live out there noticed the issue pretty >quickly. Traceroutes from Brian's house show that for our blocked >hosts, the users don't get beyond Verizon's NAT. > >The Verizon tech's "fix" of re-patching Brian's DSL line in to a >different router feels to me like there is a config problem in the >other router, but the tech we got is not authorized to alter the >config. It would be nice if we could reach someone who could actually >edit the broken config and make it right. Anyone from Verzion's NOC >for Western Mass reading this? Or, does anyone else have useful >contact info for them? > >FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and >a good trace from Brian's house, to different servers on our campus : > >FAILS: >Tracing route to wilbur.simons-rock.edu [208.81.88.15] >over a maximum of 30 hops: > > 1<1 ms<1 ms<1 ms 192.168.10.1 > 2 1 ms 1 ms 1 ms 192.168.1.1 > 353 ms 104 ms 116 ms 10.14.1.1 > 4 *** Request timed out. > 5 *** Request timed out. > 6 *** Request timed out. > 7 *** Request timed out. > >WORKS: >Tracing route to dev.simons-rock.edu [208.81.88.25] >over a maximum of 30 hops: > >1<1 ms<1 ms<1 ms 192.168.10.1 >2 1 ms 1 ms 1 ms 192.168.1.1 >387 ms54 ms54 ms 10.14.1.1 >499 ms 109 ms 103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon- >gni.net [130.81.10.77] >516 ms18 ms16 ms so-7-3-1-0.NY5030-BB-RTR2.verizon- >gni.net [130.81.20.6] >619 ms17 ms17 ms 0.xe-3-1-0.BR3.NYC4.ALTER.NET >[152.63.2.81] >718 ms21 ms18 ms 204.255.168.194 >8 108 ms 188 ms 116 ms pos5-0-2488M.cr1.BOS1.gblx.net >[67.17.94.57] >924 ms28 ms23 ms pos0-0-0-155M.ar1.BOS1.gblx.net >[67.17.70.162] >10 121 ms 160 ms 127 ms 64.213.79.250 >1177 ms77 ms78 ms 208.81.88.25 > >Trace complete. > >Anyways, thanks for any suggestions you can offer. > >Steve Bohrer >Network Administrator >ITS, Bard College at Simon's Rock >413-528-7645 > > >
Re: routing issue for verizon dsl customers in western massachusetts
> On Thu, Sep 15, 2011 at 20:52 UTC, Christopher Morrow > wrote: >> On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer >> wrote: >>> Traceroutes from Brian's house >>> show that for our blocked hosts, the users don't get beyond Verizon's >>> NAT. >> >> I wasn't aware verizon implemented CGN already... way to be a 'first >> mover' in this field verizon! > > I am betting they have not. > >>> FAILS: >>> Tracing route to wilbur.simons-rock.edu [208.81.88.15] >>> over a maximum of 30 hops: >>> >>>  1   <1 ms   <1 ms   <1 ms  192.168.10.1 >>>  2   1 ms   1 ms   1 ms  192.168.1.1 >>>  3   53 ms  104 ms  116 ms  10.14.1.1 >>>  4   *     *     *   Request timed out. >>>  5   *     *     *   Request timed out. >>>  6   *     *     *   Request timed out. >>>  7   *     *     *   Request timed out. > > Here's a trace to the same destination from a Verizon residential DSL > on Maryland's Eastern Shore: > > Tracing route to wilbur.simons-rock.edu [208.81.88.15] > over a maximum of 30 hops: > > 1<1 ms<1 ms<1 ms 192.168.201.1 > 225 ms25 ms24 ms 10.31.8.1 > 338 ms99 ms78 ms > at-4-3-0-1712.sal-core-rtr1.verizon-gni.net [130.81.136.122] > 426 ms26 ms26 ms > so-0-0-0-0.sal-core-rtr2.verizon-gni.net [130.81.18.247] > 594 ms31 ms31 ms 130.81.20.238 > 632 ms32 ms32 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] > 732 ms33 ms31 ms te2-3.ar6.DCA3.gblx.net [64.215.195.113] > 833 ms33 ms32 ms xe6-2-0-10G.scr2.WDC2.gblx.net > [67.16.136.197] > 937 ms38 ms38 ms so2-2-0-10G.scr2.NYC1.gblx.net > [67.17.95.102] > 1043 ms44 ms44 ms pos9-0-2488M.cr2.BOS1.gblx.net > [67.17.94.157] > 11 244 ms 200 ms 204 ms pos1-0-0-155M.ar1.BOS1.gblx.net > [67.17.70.165] > 1250 ms51 ms50 ms 64.213.79.250 > 1349 ms50 ms48 ms wilbur.simons-rock.edu [208.81.88.15] > > 192.168.201.1 is the router behind the bridged ADSL CPE which > terminates the customer PPPoE. 10.31.8.1 is RFC 1918, but is not a > NAT. I know from various "test my crappy broadband" sites that the > only drain bramage on the provider side of the link is routine > consumer-class port blocking (SMB networking, SQL, and of course port > 80 so the mothe#@#$rs can charge extra for "business" with static IP > and unblocked http). At least https works. > > Looking at Brian's trace above, I can't help wondering if the client > is 444'd, but not due to CGN/LSN. Could both 192.168.10.1 and > 192.168.1.1 be on-premises, with 192.168.1.1 terminating PPPoE? The > latencies seem to confirm. It is possible it's only a single level of > NAT on .1.1, with more-respectable routing by .10.1... In my setup, 192.168.10.1 is my DD-WRT router and 192.168.1.1 is the DSL modem. When I ran a traceroute directly from the DSL modem's web interface, I got the following results: 157 ms98 ms 129 ms 10.14.1.1 3 *** Request timed out. 5 *** Request timed out. 5 *** Request timed out. 6 *** Request timed out. > > Cheers, > Dave Hart > >
Re: routing issue for verizon dsl customers in western massachusetts
On Thu, Sep 15, 2011 at 20:52 UTC, Christopher Morrow wrote: > On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer > wrote: >> Traceroutes from Brian's house >> show that for our blocked hosts, the users don't get beyond Verizon's NAT. > > I wasn't aware verizon implemented CGN already... way to be a 'first > mover' in this field verizon! I am betting they have not. >> FAILS: >> Tracing route to wilbur.simons-rock.edu [208.81.88.15] >> over a maximum of 30 hops: >> >> 1 <1 ms <1 ms <1 ms 192.168.10.1 >> 2 1 ms 1 ms 1 ms 192.168.1.1 >> 3 53 ms 104 ms 116 ms 10.14.1.1 >> 4 * * * Request timed out. >> 5 * * * Request timed out. >> 6 * * * Request timed out. >> 7 * * * Request timed out. Here's a trace to the same destination from a Verizon residential DSL on Maryland's Eastern Shore: Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops: 1<1 ms<1 ms<1 ms 192.168.201.1 225 ms25 ms24 ms 10.31.8.1 338 ms99 ms78 ms at-4-3-0-1712.sal-core-rtr1.verizon-gni.net [130.81.136.122] 426 ms26 ms26 ms so-0-0-0-0.sal-core-rtr2.verizon-gni.net [130.81.18.247] 594 ms31 ms31 ms 130.81.20.238 632 ms32 ms32 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 732 ms33 ms31 ms te2-3.ar6.DCA3.gblx.net [64.215.195.113] 833 ms33 ms32 ms xe6-2-0-10G.scr2.WDC2.gblx.net [67.16.136.197] 937 ms38 ms38 ms so2-2-0-10G.scr2.NYC1.gblx.net [67.17.95.102] 1043 ms44 ms44 ms pos9-0-2488M.cr2.BOS1.gblx.net [67.17.94.157] 11 244 ms 200 ms 204 ms pos1-0-0-155M.ar1.BOS1.gblx.net [67.17.70.165] 1250 ms51 ms50 ms 64.213.79.250 1349 ms50 ms48 ms wilbur.simons-rock.edu [208.81.88.15] 192.168.201.1 is the router behind the bridged ADSL CPE which terminates the customer PPPoE. 10.31.8.1 is RFC 1918, but is not a NAT. I know from various "test my crappy broadband" sites that the only drain bramage on the provider side of the link is routine consumer-class port blocking (SMB networking, SQL, and of course port 80 so the mothe#@#$rs can charge extra for "business" with static IP and unblocked http). At least https works. Looking at Brian's trace above, I can't help wondering if the client is 444'd, but not due to CGN/LSN. Could both 192.168.10.1 and 192.168.1.1 be on-premises, with 192.168.1.1 terminating PPPoE? The latencies seem to confirm. It is possible it's only a single level of NAT on .1.1, with more-respectable routing by .10.1... Cheers, Dave Hart
Re: routing issue for verizon dsl customers in western massachusetts
On Sep 15, 2011, at 4:52 PM, Christopher Morrow wrote: I wasn't aware verizon implemented CGN already... way to be a 'first mover' in this field verizon! Maybe they are tying it out here in the sticks, where the glitches only hit single-digit numbers of users? (Though, I'd think if it was actually new, then they might have a higher-order tech paying attention to the glitches.) Oh well. Close enough, mostly. Steve
Re: routing issue for verizon dsl customers in western massachusetts
On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer wrote: > On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote: > >> On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold wrote: >>> >>> Over the past week, we've discovered that there is an issue with the way >>> some Verizon DSL customers are being routed in Western Massachusetts that >>> is >>> preventing them from reaching my employers public IPs. The problem is >>> only >>> limited to Verizon DSL customers, everyone else can reach these IP >>> addresses >>> just fine. After many hours on the phone with Verizon tech support, I >>> finally managed to get myself and one of my coworker's home dsl >>> connections >>> switched from a "redback router" to a "juniper router" which resolved the >>> issue, but only for us. > > [...] >> >> If you buy verizon services at your day job you can probably make >> noise through your sales droids better than here (sadly)... verizon >> likes to jump when customers have problems, if the customer is a large >> corporation or other 'important' customer. > > > That is just the problem! The college does not buy any Verizon network stuff > directly, so we don't really have any access to their support. (We have a > few cell phones, but not enough to be "important".) > > Brian Gold (who first posted) happens to have their DSL to his house, and he > was one of five who have reported the problem, so that gave him a slight in. > But the only techs he could reach as an "end user" were not high enough up > to fix this problem in a general way. After pressing them for literally > hours, he was able to get transfered to their NOC, and get the problem > resolved for his one address. But, they would not give him the NOC contact, > and he had to repeat this multi-hour process to get it fixed for an other > user. Verizon's DSL support suggested that we get our bandwith provider > involved, and so they tried to pitch in, but they don't have any Verizon NOC > contact either, especially since this issue is purely within a small corner > of Verizon's DSL network, not on any of Verizon's links to our provider. > > This issue hits only a few Verizon DSL users in NW Mass. It does not really > seem like a routing problem, because the affected users can reach many of > the servers in our AS, but not some addresses. Unfortunately, the "blocked" > addresses include our web server and our mail server, so our staff who live > out there noticed the issue pretty quickly. Traceroutes from Brian's house > show that for our blocked hosts, the users don't get beyond Verizon's NAT. I wasn't aware verizon implemented CGN already... way to be a 'first mover' in this field verizon! > > The Verizon tech's "fix" of re-patching Brian's DSL line in to a different > router feels to me like there is a config problem in the other router, but > the tech we got is not authorized to alter the config. It would be nice if > we could reach someone who could actually edit the broken config and make it > right. Anyone from Verzion's NOC for Western Mass reading this? Or, does > anyone else have useful contact info for them? you probably want someone in the NOC which is (I think) stil in Reston, va... I don't think they have separate noc's per region. The first-line tech folks you chat with on the phone really arent' able (even to login really) to fix devices in the field :( anyways, this looks crappy :( but yeah for CGN being all it's cracked up to be! -chris > > FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and a good > trace from Brian's house, to different servers on our campus : > > FAILS: > Tracing route to wilbur.simons-rock.edu [208.81.88.15] > over a maximum of 30 hops: > > 1 <1 ms <1 ms <1 ms 192.168.10.1 > 2 1 ms 1 ms 1 ms 192.168.1.1 > 3 53 ms 104 ms 116 ms 10.14.1.1 > 4 * * * Request timed out. > 5 * * * Request timed out. > 6 * * * Request timed out. > 7 * * * Request timed out. > > WORKS: > Tracing route to dev.simons-rock.edu [208.81.88.25] > over a maximum of 30 hops: > > 1 <1 ms <1 ms <1 ms 192.168.10.1 > 2 1 ms 1 ms 1 ms 192.168.1.1 > 3 87 ms 54 ms 54 ms 10.14.1.1 > 4 99 ms 109 ms 103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon-gni.net > [130.81.10.77] > 5 16 ms 18 ms 16 ms so-7-3-1-0.NY5030-BB-RTR2.verizon-gni.net > [130.81.20.6] > 6 19 ms 17 ms 17 ms 0.xe-3-1-0.BR3.NYC4.ALTER.NET [152.63.2.81] > 7 18 ms 21 ms 18 ms 204.255.168.194 > 8 108 ms 188 ms 116 ms pos5-0-2488M.cr1.BOS1.gblx.net [67.17.94.57] > 9 24 ms 28 ms 23 ms pos0-0-0-155M.ar1.BOS1.gblx.net [67.17.70.162] > 10 121 ms 160 ms 127 ms 64.213.79.250 > 11 77 ms 77 ms 78 ms 208.81.88.25 > > Trace complete. > > Anyways, thanks for any suggestions you can offer. > > Steve Bohrer > Network Administrator > ITS, Bard College at Simon's Rock > 413-528-7645 > > > >
Re: routing issue for verizon dsl customers in western massachusetts
On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote: On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold wrote: Over the past week, we've discovered that there is an issue with the way some Verizon DSL customers are being routed in Western Massachusetts that is preventing them from reaching my employers public IPs. The problem is only limited to Verizon DSL customers, everyone else can reach these IP addresses just fine. After many hours on the phone with Verizon tech support, I finally managed to get myself and one of my coworker's home dsl connections switched from a "redback router" to a "juniper router" which resolved the issue, but only for us. [...] If you buy verizon services at your day job you can probably make noise through your sales droids better than here (sadly)... verizon likes to jump when customers have problems, if the customer is a large corporation or other 'important' customer. That is just the problem! The college does not buy any Verizon network stuff directly, so we don't really have any access to their support. (We have a few cell phones, but not enough to be "important".) Brian Gold (who first posted) happens to have their DSL to his house, and he was one of five who have reported the problem, so that gave him a slight in. But the only techs he could reach as an "end user" were not high enough up to fix this problem in a general way. After pressing them for literally hours, he was able to get transfered to their NOC, and get the problem resolved for his one address. But, they would not give him the NOC contact, and he had to repeat this multi- hour process to get it fixed for an other user. Verizon's DSL support suggested that we get our bandwith provider involved, and so they tried to pitch in, but they don't have any Verizon NOC contact either, especially since this issue is purely within a small corner of Verizon's DSL network, not on any of Verizon's links to our provider. This issue hits only a few Verizon DSL users in NW Mass. It does not really seem like a routing problem, because the affected users can reach many of the servers in our AS, but not some addresses. Unfortunately, the "blocked" addresses include our web server and our mail server, so our staff who live out there noticed the issue pretty quickly. Traceroutes from Brian's house show that for our blocked hosts, the users don't get beyond Verizon's NAT. The Verizon tech's "fix" of re-patching Brian's DSL line in to a different router feels to me like there is a config problem in the other router, but the tech we got is not authorized to alter the config. It would be nice if we could reach someone who could actually edit the broken config and make it right. Anyone from Verzion's NOC for Western Mass reading this? Or, does anyone else have useful contact info for them? FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and a good trace from Brian's house, to different servers on our campus : FAILS: Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops: 1<1 ms<1 ms<1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 353 ms 104 ms 116 ms 10.14.1.1 4 *** Request timed out. 5 *** Request timed out. 6 *** Request timed out. 7 *** Request timed out. WORKS: Tracing route to dev.simons-rock.edu [208.81.88.25] over a maximum of 30 hops: 1<1 ms<1 ms<1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 387 ms54 ms54 ms 10.14.1.1 499 ms 109 ms 103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon- gni.net [130.81.10.77] 516 ms18 ms16 ms so-7-3-1-0.NY5030-BB-RTR2.verizon- gni.net [130.81.20.6] 619 ms17 ms17 ms 0.xe-3-1-0.BR3.NYC4.ALTER.NET [152.63.2.81] 718 ms21 ms18 ms 204.255.168.194 8 108 ms 188 ms 116 ms pos5-0-2488M.cr1.BOS1.gblx.net [67.17.94.57] 924 ms28 ms23 ms pos0-0-0-155M.ar1.BOS1.gblx.net [67.17.70.162] 10 121 ms 160 ms 127 ms 64.213.79.250 1177 ms77 ms78 ms 208.81.88.25 Trace complete. Anyways, thanks for any suggestions you can offer. Steve Bohrer Network Administrator ITS, Bard College at Simon's Rock 413-528-7645
Re: routing issue for verizon dsl customers in western massachusetts
On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold wrote: > Hello all, I posted this to the tech@lopsa mailing list and was advised to > repost it here. If anyone can help, I would be very happy to avoid having to > deal with hours more of Verizon level 1 tech support. > > > > Over the past week, we've discovered that there is an issue with the way > some Verizon DSL customers are being routed in Western Massachusetts that is > preventing them from reaching my employers public IPs. The problem is only > limited to Verizon DSL customers, everyone else can reach these IP addresses > just fine. After many hours on the phone with Verizon tech support, I > finally managed to get myself and one of my coworker's home dsl connections > switched from a "redback router" to a "juniper router" which resolved the > issue, but only for us. I was told that everyone else in the area that is > being affected by the issue have to individually call Verizon tech support, > go through the same multi-hour troubleshooting steps, and if the technician > is bright enough to recognize what is going on, get their issue escalated up > to the central office where (in 2-4 business days) they will be switched > over to the juniper router. Obviously, this is not the ideal solution. I'd actually it's not a bad solution.. if verizon is looking to lose lots of money on tech support calls... :) > really like to make the higher ups at Verizon aware of this issue and come If you buy verizon services at your day job you can probably make noise through your sales droids better than here (sadly)... verizon likes to jump when customers have problems, if the customer is a large corporation or other 'important' customer. > up with a solution for all of the affected customers, but because I only > have a residential account and my employer doesn't use Verizon, I've been > stymied in all of my attempts so far. Does anyone here have any contacts at > Verizon that I could get in touch with? > >
routing issue for verizon dsl customers in western massachusetts
Hello all, I posted this to the tech@lopsa mailing list and was advised to repost it here. If anyone can help, I would be very happy to avoid having to deal with hours more of Verizon level 1 tech support. Over the past week, we've discovered that there is an issue with the way some Verizon DSL customers are being routed in Western Massachusetts that is preventing them from reaching my employers public IPs. The problem is only limited to Verizon DSL customers, everyone else can reach these IP addresses just fine. After many hours on the phone with Verizon tech support, I finally managed to get myself and one of my coworker's home dsl connections switched from a "redback router" to a "juniper router" which resolved the issue, but only for us. I was told that everyone else in the area that is being affected by the issue have to individually call Verizon tech support, go through the same multi-hour troubleshooting steps, and if the technician is bright enough to recognize what is going on, get their issue escalated up to the central office where (in 2-4 business days) they will be switched over to the juniper router. Obviously, this is not the ideal solution. I'd really like to make the higher ups at Verizon aware of this issue and come up with a solution for all of the affected customers, but because I only have a residential account and my employer doesn't use Verizon, I've been stymied in all of my attempts so far. Does anyone here have any contacts at Verizon that I could get in touch with?
