Hello Piba (and anyone else…)
Sorry for not having answered before…
To answer you questions, firstly, I’m not in a datacenter, only a client
offices with different ISP.
I agree with you double NAT is bad but you can’t alway get rid of it… and you
should know that on one of my wan connexion I was technically able to make a
bridge and I though the problem were the same with this connexion but in fact,
my fault, bad setting, so with this connexion everything is working !
So I stay with my third connexion witch is not working (double NAT) and only
with this one, I can see traffic but it’s not working, so I gave a try with the
flag you requested to try to give more information to understand what happens…
from outside to 2223 portwitch is where SSH deamon is listening on the pfsense
from OVH Connexion (double NAT) = not working
2.3.1-RELEASE][r...@pfsense.concorde-pereire.loc]/root: tcpdump -en -i re0 port
2223
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:42:56.509422 a4:b1:e9:f7:13:e8 > 00:0d:b9:33:7c:6c, ethertype IPv4 (0x0800),
length 66: 62.210.139.211.49236 > 192.168.101.254.2223: Flags [S], seq
2309097405, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
14:42:56.509584 00:0d:b9:33:7c:6c > 24:95:04:fb:ae:90, ethertype IPv4 (0x0800),
length 66: 192.168.101.254.2223 > 62.210.139.211.49236: Flags [S.], seq
3034279515, ack 2309097406, win 65228, options [mss 1460,nop,wscale
7,sackOK,eol], length 0
14:42:59.509726 00:0d:b9:33:7c:6c > 24:95:04:fb:ae:90, ethertype IPv4 (0x0800),
length 66: 192.168.101.254.2223 > 62.210.139.211.49236: Flags [S.], seq
3034279515, ack 2309097406, win 65228, options [mss 1460,nop,wscale
7,sackOK,eol], length 0
14:42:59.529210 a4:b1:e9:f7:13:e8 > 00:0d:b9:33:7c:6c, ethertype IPv4 (0x0800),
length 66: 62.210.139.211.49236 > 192.168.101.254.2223: Flags [S], seq
2309097405, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
from outside to 2223 port witch is where SSH deamon is listening on the pfsense
from SFR Connexion (double NAT) = working
[2.3.1-RELEASE][r...@pfsense.concorde-pereire.loc]/root: tcpdump -en -i re0
port 2223
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:43:47.280639 24:95:04:fb:ae:90 > 00:0d:b9:33:7c:6c, ethertype IPv4 (0x0800),
length 66: 62.210.139.211.49239 > 192.168.101.254.2223: Flags [S], seq
2327707324, win 9652, options [mss 1460,wscale 3,sackOK,eol], length 0
14:43:47.280797 00:0d:b9:33:7c:6c > 24:95:04:fb:ae:90, ethertype IPv4 (0x0800),
length 66: 192.168.101.254.2223 > 62.210.139.211.49239: Flags [S.], seq
3881093896, ack 2327707325, win 65228, options [mss 1460,nop,wscale
7,sackOK,eol], length 0
14:43:47.311955 24:95:04:fb:ae:90 > 00:0d:b9:33:7c:6c, ethertype IPv4 (0x0800),
length 60: 62.210.139.211.49239 > 192.168.101.254.2223: Flags [.], ack 1, win
32850, length 0
14:43:47.322754 24:95:04:fb:ae:90 > 00:0d:b9:33:7c:6c, ethertype IPv4 (0x0800),
length 82: 62.210.139.211.49239 > 192.168.101.254.2223: Flags [P.], seq 1:29,
ack 1, win 32850, length 28
14:43:47.322883 00:0d:b9:33:7c:6c > 24:95:04:fb:ae:90, ethertype IPv4 (0x0800),
length 54: 192.168.101.254.2223 > 62.210.139.211.49239: Flags [.], ack 29, win
513, length 0
14:43:47.343017 00:0d:b9:33:7c:6c > 24:95:04:fb:ae:90, ethertype IPv4 (0x0800),
length 75: 192.168.101.254.2223 > 62.210.139.211.49239: Flags [P.], seq 1:22,
ack 29, win 513, length 21
To the light of this new details, I can see that the pfsense is trying to
respond to the bad mac address (the working connexion one) ! and that is the
reason it’s not working ! So I had a look at the interface settings and I
noticed that the mac address it tries to reply is the one selected here in the
menu list, I have two since I have two gateway for one interface in the same
private network space…
First I want to tank you helping me clarifying what was going wrong (for the
second pfsense installation it’s a bad coincidence the problem is with the
modem configuration witch is defective)
So my question now is : How can I set both the gateway to have the same
priority or at least make the system answer to the address that initiate the
connexion ?
I don’t know if I’m clear with my configuration if someone has an idea or need
more clarifying, I would be more than happy to explain better my settings.
Thank you again,
Best regards,
Jean-Laurent Ivars
Responsable Technique | Technical Manager
22, rue Robert - 13007 Marseille
Tel: 09 84 56 64 30 - Mobile: 06.52.60.86.47
Linkedin <http://fr.linkedin.com/in/jlivars/> | Viadeo
<http://www.viadeo.com/fr/profile/jean-laurent.ivars> | www.ipgenius.fr
<https://www.ipgenius.fr/>
> Le 25 juin 2016 à 22:04, PiBa <pba_...@yahoo.com> a écrit :
>
> Hi Jean-Laurent,
>
> Op 25-6-2016 om 18:46 schreef Jean-Laurent Ivars:
>> You’re perfectly right i only gave extracts of the logs :) next time I’m
>> going to be more precise and make the capture from the beginning...
> Its not that extracts are bad, just knowing what part to extract to make it
> easier to compare on 1 screen.
> I cut your tcpdumps a bit shorter below after a few packets ;) should still
> have all relevant info for the current issue, if anyone else wants to go over
> them.
>> Anyway, as you suppose, I continue to see the same [S] paquet repeating
>> again and again…
> The client tries to connect, but never gets the [S.] reply packet. As such
> its does the right thing to try again a few times..
>> These captures have been done directly on a WAN interface (not on the LAN
>> one) the reason you see a private address is because there is a double NAT...
> Double nat is in general a bad thing. I assumed LAN because of private ip's..
> Anyway are all locations home/business connections where you physically
> manage the router that connects to the isp? Not like inside a OVH
> datacenter.? (i know OVH does some strange things in the DC..) Just for
> ruling some problem in them out you've tried restarting all those
> router/modem devices? Might not help at all, if it does whel that would
> indicate a issue there..
> Also i assume all the pfSense boxes can properly use the internet themselves
> without any issue?
>> As you ask, I am going to send you :
>>
>> - tcpdump on a client for the port 2223
>> - tcpdump on the pfsense on the port 2223 choosing the interface that is
>> directly connected to the modem-router
>>
>> 1- From the pfsense WAN interface point of view with a not working gateway
>> from the begining until the end (time out from the client side) :
>>
>> [2.3.1-RELEASE][r...@pfsense.concorde-pereire.loc
>> <mailto:r...@pfsense.concorde-pereire.loc>]/root: tcpdump -i re0 port 2223
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on re0, link-type EN10MB (Ethernet), capture size 65535 bytes
>> 18:01:51.206180 IP 196.222.21.93.rev.sfr.net <http://rev.sfr.net/>.51548 >
>> 192.168.101.254.2223: Flags [S], seq 3384870708, win 65535, options [mss
>> 1460,nop,wscale 5,nop,nop,TS val 333778453 ecr 0,sackOK,eol], length 0
>> 18:01:51.206358 IP 192.168.101.254.2223 > 196.222.21.93.rev.sfr.net
>> <http://rev.sfr.net/>.51548: Flags [S.], seq 2254053748, ack 3384870709, win
>> 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1269164786 ecr
>> 333778453], length 0
>> 18:01:52.214077 IP 196.222.21.93.rev.sfr.net <http://rev.sfr.net/>.51548 >
>> 192.168.101.254.2223: Flags [S], seq 3384870708, win 65535, options [mss
>> 1460,nop,wscale 5,nop,nop,TS val 333779453 ecr 0,sackOK,eol], length 0
>> 18:01:52.214177 IP 192.168.101.254.2223 > 196.222.21.93.rev.sfr.net
>> <http://rev.sfr.net/>.51548: Flags [S.], seq 2254053748, ack 3384870709, win
>> 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1269164786 ecr
>> 333779453], length 0
> The question is where/why does this reply [S.] packet get dropped between
> pfsense and client machine.
> Can you try above tcpdump 1. again with -en flags?
>
> *tcpdump -en -i re0 port 2223 *
>
> Only 4 packets should be enough.. Basically for checking if
> source/destination MAC addresses are the same..
>
>> 2- From the client point of view to the not working gateway from the
>> begining until the end (time out from the client side) :
>>
>> root@MacBook-de-Jean-Laurent:~$ tcpdump -i en0 port 2223
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
>> 18:01:51.196869 IP 192.168.10.50.51548 >
>> 20-98-190-109.dsl.ovh.fr.rockwell-csp3: Flags [S], seq 3384870708, win
>> 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 333778453 ecr
>> 0,sackOK,eol], length 0
>> 18:01:52.206194 IP 192.168.10.50.51548 >
>> 20-98-190-109.dsl.ovh.fr.rockwell-csp3: Flags [S], seq 3384870708, win
>> 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 333779453 ecr
>> 0,sackOK,eol], length 0
> Client never gets the response packets.. As such keeps trying again for a
> while.
>> 3- From the pfsense WAN interface point of view with a working gateway from
>> the begining until the end (connexion then a few commands and deconnexion) :
>>
>> [2.3.1-RELEASE][r...@pfsense.concorde-pereire.loc
>> <mailto:r...@pfsense.concorde-pereire.loc>]/root: tcpdump -i re0 port 2223
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on re0, link-type EN10MB (Ethernet), capture size 65535 bytes
>> 18:09:12.648018 IP 196.222.21.93.rev.sfr.net <http://rev.sfr.net/>.51615 >
>> 192.168.101.254.2223: Flags [S], seq 4029864830, win 65535, options [mss
>> 1460,wscale 6,sackOK,eol], length 0
>> 18:09:12.648159 IP 192.168.101.254.2223 > 196.222.21.93.rev.sfr.net
>> <http://rev.sfr.net/>.51615: Flags [S.], seq 2831647839, ack 4029864831, win
>> 65228, options [mss 1460,nop,wscale 7,sackOK,eol], length 0
>> 18:09:12.687456 IP 196.222.21.93.rev.sfr.net <http://rev.sfr.net/>.51615 >
>> 192.168.101.254.2223: Flags [.], ack 1, win 8192, length 0
>> 18:09:12.687607 IP 196.222.21.93.rev.sfr.net <http://rev.sfr.net/>.51615 >
>> 192.168.101.254.2223: Flags [P.], seq 1:22, ack 1, win 8192, length 21
>> 18:09:12.687653 IP 192.168.101.254.2223 > 196.222.21.93.rev.sfr.net
>> <http://rev.sfr.net/>.51615: Flags [.], ack 22, win 513, length 0
>> 18:09:12.737138 IP 192.168.101.254.2223 > 196.222.21.93.rev.sfr.net
>> <http://rev.sfr.net/>.51615: Flags [P.], seq 1:22, ack 22, win 513, length 21
>>
>> 4- From the client point of view to the working WAN access from the begining
>> until the end (connexion then a few commands and deconnexion) :
>>
>> root@MacBook-de-Jean-Laurent:~$ tcpdump -i en0 port 2223
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
>> 18:09:12.626154 IP 192.168.10.50.51615 >
>> 184.196.154.77.rev.sfr.net.rockwell-csp3: Flags [S], seq 4029864830, win
>> 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 334217159 ecr
>> 0,sackOK,eol], length 0
>> 18:09:12.666015 IP 184.196.154.77.rev.sfr.net.rockwell-csp3 >
>> 192.168.10.50.51615: Flags [S.], seq 2831647839, ack 4029864831, win 65535,
>> options [mss 1460,wscale 8,sackOK,eol], length 0
>> 18:09:12.666099 IP 192.168.10.50.51615 >
>> 184.196.154.77.rev.sfr.net.rockwell-csp3: Flags [.], ack 1, win 8192, length >> 0
>> 18:09:12.666537 IP 192.168.10.50.51615 >
>> 184.196.154.77.rev.sfr.net.rockwell-csp3: Flags [P.], seq 1:22, ack 1, win
>> 8192, length 21
>> 18:09:12.705914 IP 184.196.154.77.rev.sfr.net.rockwell-csp3 >
>> 192.168.10.50.51615: Flags [.], ack 22, win 513, length 0
>> 18:09:12.754171 IP 184.196.154.77.rev.sfr.net.rockwell-csp3 >
>> 192.168.10.50.51615: Flags [P.], seq 1:22, ack 22, win 513, length 21
> Both sides from captures 3 and 4 show each packet arriving as such it all
> works fine ;)
>> I hope these complete captures can tell you more what’s happening ?
>>
>> I have no special packages installed, had only iperf and nmap for testing
>> purposes… and just uninstalled but it changed nothing :(
>>
>> I have absolutely not any exotic or special configuration like
>> limiters/shapers and absolutely no VPN at the moment, no IPSEC, OPENVPN nor
>> anything else… on the other pfsense box I have openvpn installed but the
>> symptoms aren’t different so I don’t think it’s related.
>>
>> I really don’t understand what’s happening here but the fact that on both my
>> pfsenses I have some WAN redirections or DMZ that are working and that are
>> not the only factor that changes here is the internet provider and maybe
>> it’s not hardware related but I would think that it’s some kind of network
>> setting somewhere that could help me. And we should not forget that before
>> the upgrade on the version 2.3 it was working on one of the box for sure !
>> (the other wasn’t installed at the moment)
>>
>> In my opinion it’s something silly like mtu for exemple but making my life a
>> hell !
> Mtu can be troublesome, but i think the syn-ack would make it through even if
> the mtu is a bit off.. Anyway MTU seems to be a proper 1500 at the ovh side.
>> I know it’s a long long mail with a lot of informations but if someone can
>> help me with that I would be so much grateful !!!
>>
>> Thank you for those who are reading :)
>> Best regards,
>>
>>
>> Jean-Laurent Ivars
>> Responsable Technique | Technical Manager
>> 22, rue Robert - 13007 Marseille
>> Tel: 09 84 56 64 30 - Mobile: 06.52.60.86.47
>> Linkedin <http://fr.linkedin.com/in/jlivars/
>> <http://fr.linkedin.com/in/jlivars/>> | Viadeo
>> <http://www.viadeo.com/fr/profile/jean-laurent.ivars
>> <http://www.viadeo.com/fr/profile/jean-laurent.ivars>> | www.ipgenius.fr
>> <http://www.ipgenius.fr/><https://www.ipgenius.fr/
>> <https://www.ipgenius.fr/>>
> Not really sure what the issue is just yet.
> Hope some of my comments help get you closer ;)
>
> You might also try calling ovh and try to see if maybe they are blocking
> something.?.
>
> Regards,
> PiBa-NL
>>> Le 25 juin 2016 à 16:47, PiBa <pba_...@yahoo.com> a écrit :
>>>
>>> Hi Jean-Laurent,
>>>
>>> Op 25-6-2016 om 10:37 schreef Jean-Laurent Ivars:
>>>> This is logs generated by tcpdump from the same client machine when I try
>>>> to access the firewall thru working internet access provider :
>>>>
>>>> port 2223
>>>> 16:55:04.501509 IP 46.105.230.225.39304 > 192.168.101.254.2223: Flags
>>>> [P.], seq 29:701, ack 22, win 32844, length 672
>>>> 16:55:04.501652 IP 192.168.101.254.2223 > 46.105.230.225.39304: Flags
>>>> [P.], seq 22:910, ack 701, win 508, length 888
>>>> port 10000
>>>> 16:58:51.821691 IP 192.168.101.254.10000 > 46.105.230.225.5829: Flags
>>>> [P.], seq 209411:210119, ack 2393, win 513, length 708
>>>> 16:58:52.058014 IP 46.105.230.225.5829 > 192.168.101.254.10000: Flags [.],
>>>> ack 210119, win 32673, length 0
>>> Both these captures are in the 'middle' of a already working connection,
>>> but does show it 'works' at that time.. To compare it to the capture below
>>> better start capturing before initiation the connection :) .
>>>> And there the same command output when I try to access from one that is
>>>> not working :
>>>>
>>>> Port 2223
>>>> 16:53:13.240166 IP 46.105.230.225.19480 > 192.168.101.254.2223: Flags [S],
>>>> seq 3864438539, win 8192, options [mss 1460,nop,nop,sackOK], length 0
>>>> 16:53:13.240306 IP 192.168.101.254.2223 > 46.105.230.225.19480: Flags
>>>> [S.], seq 2492220538, ack 3864438540, win 65228, options [mss
>>>> 1460,nop,wscale 7,sackOK,eol], length 0
>>>> Port 10000
>>>> 16:56:39.864021 IP 46.105.230.225.41932 > 192.168.101.254.10000: Flags
>>>> [S], seq 2837326484, win 8192, options [mss 1460,nop,wscale
>>>> 2,nop,nop,sackOK], length 0
>>>> 16:56:39.864169 IP 192.168.101.254.10000 > 46.105.230.225.41932: Flags
>>>> [S.], seq 1993261464, ack 2837326485, win 65228, options [mss
>>>> 1460,nop,wscale 7,sackOK,eol], length 0
>>> These SYN [S] and SYN-ACK [S.] packets are the normal start of a tcp
>>> connection. Sofar nothing strange during connection initiation, there
>>> should follow a 3rd ACK [.] packet after which there would follow some
>>> useful data exchange like in the first working captures.. But i guess you
>>> keep seeing these same two [S] and [S.] packets repeating.?
>>> You have performed these captures on the LAN interface, could you repeat
>>> them on the WAN interface to confirm that the [S.] is properly send back on
>>> that same interface the client send the request on? Also if its possible to
>>> capture on the client device it would be interesting to see if the [S.]
>>> arrives there properly.
>>>> I use pcengine APU system, the model is AMD G-T40E Processor with 3 NIC (
>>>> I believe It could be something related to a NIC setting somewhere but
>>>> really don’t know)
>>>>
>>>> Is someone encounter the same issue than me ? maybe it’s just a setting in
>>>> the NIC driver ?
>>> I don't expect it to be hardware/driver related at this moment..
>>>
>>> Do you have any packages installed? Snort or Suricata can sometimes
>>> unexpectedly block traffic you do want.. Or other configurations like
>>> limiters/shapers or openvpn/ipsec networks can possibly interfere..
>>>
>>> Regards,
>>> PiBa-NL
>>> _______________________________________________
>>> pfSense mailing list
>>> https://lists.pfsense.org/mailman/listinfo/list
>>> Support the project with Gold! https://pfsense.org/gold
>> _______________________________________________
>> pfSense mailing list
>> https://lists.pfsense.org/mailman/listinfo/list
>> <https://lists.pfsense.org/mailman/listinfo/list>
>> Support the project with Gold! https://pfsense.org/gold
>> <https://pfsense.org/gold>
>
>
> _______________________________________________
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> <https://lists.pfsense.org/mailman/listinfo/list>
> Support the project with Gold! https://pfsense.org/gold
> <https://pfsense.org/gold>
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold
