You may want to do traceroute using syn/ack packets to find the offending piece of equipment (may require modifying traceroute to set the syn and ack).
> On 15 Sep 2020, at 07:25, Andrey Khomyakov <khomyakov.and...@gmail.com> wrote: > > TL;DR I suspect there are middle boxes that don't like IPs ending in .255. > Anyone seen that? > > Folks, > We are troubleshooting a strange issue where some of our customers cannot > establish a successful connection with our HTTP front end. In addition to > checking the usual things like routing and interface errors and security > policy configurations, hopening support tickets with the load balancer vendor > so far all to no avail, we did packet captures. > Based on the packet captures we receive a SYN, we reply with SYN-ACK, but the > client never actually receives that SYN-ACK. In a different instance the > 3-way completes, followed by TLS client hello to us, we reply with TLS Server > Hello and that server hello never makes it to the client. > And again, this is only affecting a small subset of customers thus suggesting > it's not the load balancer or the edge routing configuration (in fact we can > traceroute fine to the customer's IP). > So far the only remaining theory that remains is that there are middle boxes > out there that do not like IPs ending in .255. The service that the clients > can't get to is hosted on two IPs ending in .255 > Let's just say they are x.x.121.255 and x.x.125.255. We even stood up a basic > "hello world" web server on x.x.124.255 with the same result. Standing up the > very same basic webserver on x.x.124.250 allows the client to succeed. > So far we have a friendly customer who has been working with us on > troubleshooting the issue and we have some pcaps from the client's side > somewhat confirming that it's not the customer's system either. > This friendly customer is in a small 5 people office with Spectrum business > internet (that's the SYN-ACK case). The same customer tried hopping on his > LTE hotspot which came up as Cellco Partnership DBA Verizon Wireless with the > same result (that's the TLS server hello case). That same customer with the > same workstation drives a town over and he can get to the application fine > (we are still waiting for the customer to let us know what that source IP is > when it does work). > Before you suggest that those .255 addresses are broadcasts on some VLAN, > they are not. They are injected as /32s using a routing protocol, while the > VLAN addressing is all RFC1918 addressing. > > --Andrey -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
