Good afternoon,
We are using pfSense 2.1.4 and OpenBGPD package 0.9.2. We are trying to
implement Amazon AWS direct connect. I believe I have the bgpd.conf
correct as I am seeing the following in routing.log, also bgpd status
shows messages being exchanged.
=== snip routing.log ===
Aug 7 17:16:00 4slgbmernfw01 bgpd[5653]: startup
Aug 7 17:16:00 4slgbmernfw01 bgpd[5653]: rereading config
Aug 7 17:16:00 4slgbmernfw01 bgpd[5783]: route decision engine ready
Aug 7 17:16:00 4slgbmernfw01 bgpd[5790]: session engine ready
Aug 7 17:16:00 4slgbmernfw01 bgpd[5783]: RDE reconfigured
Aug 7 17:16:00 4slgbmernfw01 bgpd[5790]: listening on 192.168.55.1
Aug 7 17:16:00 4slgbmernfw01 bgpd[5790]: SE reconfigured
Aug 7 17:16:00 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC
MER Peer): state change None -> Idle, reason: None
Aug 7 17:16:00 4slgbmernfw01 bgpd[5653]: nexthop 192.168.55.3 now valid:
directly connected
Aug 7 17:16:01 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC
MER Peer): state change Idle -> Connect, reason: Start
Aug 7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC
MER Peer): state change Connect -> OpenSent, reason: Connection opened
Aug 7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC
MER Peer): state change OpenSent -> OpenConfirm, reason: OPEN message
received
Aug 7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC
MER Peer): state change OpenConfirm -> Established, reason: KEEPALIVE
message received
Aug 7 17:16:30 4slgbmernfw01 bgpd[5783]: Rib Loc-RIB: neighbor
192.168.55.5 (AWS-DC MER Peer) AS9059: update 172.16.24.0/21 via
192.168.55.5
Aug 7 17:16:30 4slgbmernfw01 bgpd[5653]: nexthop 192.168.55.5 now valid:
via 192.168.55.1
=========
However when a server on a local subnet in our AS tries to ping a server
in the remote AS the traffic gets routed to the WAN interface and not over
the BGP nexthop.
Here is our BGPD config :
=== snip ===
# This file was created by the package manager. Do not edit!
########
## Our AS
########
AS 65458
fib-update yes
listen on 192.168.55.1
log updates
network 192.168.48.0/25 set nexthop 192.168.55.3
network 192.168.48.128/25 set nexthop 192.168.55.3
network 192.168.49.0/25 set nexthop 192.168.55.3
########
## Peer Groups
########
group "AWSDC" {
remote-as 9059
neighbor 192.168.55.5 {
descr "AWS-DC MER Peer"
tcp md5sig password
8e484c715b2be0e50d576bc0bb0c29d4
announce all
local-address 192.168.55.3
}
}
deny from any
deny to any
allow from 192.168.55.5
allow to 192.168.55.5
=========
..here is the BGPD Status
Summary:
Neighbor AS MsgRcvd MsgSent OutQ Up/Down
State/PrfRcvd
AWS-DC MER Peer 9059 13 12 0 00:04:09 1
Interfaces:
Interface Nexthop state Flags Link state
opt6_vip249 ok UP CARP, master
igb2_vlan300 ok UP active, 1000 MBit/s
ovpns1 ok UP active
wan_vip250 ok UP CARP, master
opt4_vip251 ok UP CARP, master
opt3_vip252 ok UP CARP, master
opt2_vip253 ok UP CARP, master
opt1_vip254 ok UP CARP, master
wan_vip255 ok UP CARP, master
lagg0_vlan50 ok UP active, 10 MBit/s
lagg0_vlan30 ok UP active, 10 MBit/s
lagg0_vlan20 ok UP active, 10 MBit/s
lagg0_vlan10 ok UP active, 10 MBit/s
lagg0 ok UP Ethernet, active, 1000 MBit/s
pflog0 invalid invalid
lo0 ok UP invalid
pfsync0 ok UP invalid
enc0 ok UP invalid
igb7 ok UP active, 1000 MBit/s
igb6 ok UP Ethernet, active, 1000 MBit/s
igb5 ok UP active, 1000 MBit/s
igb4 invalid Ethernet, invalid, 10 MBit/s
igb3 ok UP active, 1000 MBit/s
igb2 ok UP Ethernet, active, 1000 MBit/s
igb1 ok UP active, 1000 MBit/s
igb0 ok UP Ethernet, active, 1000 MBit/s
Routing:
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete
flags destination gateway lpref med aspath origin
*> 172.16.24.0/21 192.168.55.5 100 0 9059 i
AI*> 192.168.48.0/25 192.168.55.3 100 0 i
AI*> 192.168.48.128/25 192.168.55.3 100 0 i
AI*> 192.168.49.0/25 192.168.55.3 100 0 i
Forwarding:
flags: * = valid, B = BGP, C = Connected, S = Static
N = BGP Nexthop reachable via this route
r = reject route, b = blackhole route
flags prio destination gateway
*S 48 0.0.0.0/0 81.27.95.81
*S 48 10.101.1.0/25 192.168.48.1
*S 48 10.101.1.128/25 192.168.48.129
*S 48 10.101.2.0/25 192.168.49.1
*S 48 10.101.5.0/25 192.168.48.1
*S 48 10.101.5.128/25 192.168.48.129
*S 48 10.101.6.0/25 192.168.49.1
* 48 81.27.95.80/28 81.27.95.84
*C 48 81.27.95.84/32 link#11
*C 48 81.27.95.93/32 link#23
*C 48 81.27.95.94/32 link#18
*C 48 84.20.199.91/32 link#1
*C 0 127.0.0.0/8 link#0
*C 48 127.0.0.1/32 link#11
*B 48 172.16.24.0/21 192.168.55.1
*S 48 192.168.44.0/23 192.168.48.1
*S 48 192.168.46.0/24 192.168.48.1
*C 48 192.168.48.0/25 link#14
*C 48 192.168.48.118/32 link#11
*C 48 192.168.48.126/32 link#19
*C 48 192.168.48.128/25 link#15
*C 48 192.168.48.246/32 link#11
*C 48 192.168.48.254/32 link#20
*C 48 192.168.49.0/25 link#16
*C 48 192.168.49.118/32 link#11
*C 48 192.168.49.126/32 link#21
*C 48 192.168.49.128/25 link#17
*C 48 192.168.49.246/32 link#11
*C 48 192.168.49.254/32 link#22
*S 48 192.168.50.0/24 192.168.48.1
* N 48 192.168.55.0/29 192.168.55.1
*C 48 192.168.55.1/32 link#11
*CN 48 192.168.55.3/32 link#26
*S 48 192.168.90.0/24 192.168.48.1
*S 48 192.168.200.0/24 192.168.200.2
*C 48 192.168.200.1/32 link#11
*C 48 192.168.200.2/32 link#24
*C 48 192.168.226.0/27 link#7
*C 48 192.168.226.2/32 link#11
*C 0 ::1/128 link#0
*C 48 ::1/128 link#11
*C 48 fe80:1::/64 link#1
*C 48 fe80:1::225:90ff:feea:3074/128 link#11
*C 48 fe80:2::/64 link#2
*C 48 fe80:2::225:90ff:feea:3075/128 link#11
*C 48 fe80:3::/64 link#3
*C 48 fe80:3::225:90ff:feea:3076/128 link#11
*C 48 fe80:4::/64 link#4
*C 48 fe80:4::225:90ff:feea:3077/128 link#11
*C 48 fe80:6::/64 link#6
*C 48 fe80:6::225:90ff:fef3:8fc7/128 link#11
*C 48 fe80:7::/64 link#7
*C 48 fe80:7::225:90ff:fef3:8fc8/128 link#11
*C 48 fe80:8::/64 link#8
*C 48 fe80:8::225:90ff:fef3:8fc9/128 link#11
*C 48 fe80:b::/64 link#11
*C 48 fe80:b::1/128 link#11
*C 48 fe80:d::/64 link#13
*C 48 fe80:d::225:90ff:feea:3075/128 link#11
*C 48 fe80:e::/64 link#14
*C 48 fe80:e::225:90ff:feea:3074/128 link#11
*C 48 fe80:f::/64 link#15
*C 48 fe80:f::225:90ff:feea:3074/128 link#11
*C 48 fe80:10::/64 link#16
*C 48 fe80:10::225:90ff:feea:3074/128 link#11
*C 48 fe80:11::/64 link#17
*C 48 fe80:11::225:90ff:feea:3074/128 link#11
*C 48 fe80:18::225:90ff:feea:3074/128 link#11
*C 48 fe80:19::/64 link#25
*C 48 fe80:19::225:90ff:feea:3074/128 link#11
* 48 ff01:1::/32 fe80:1::225:90ff:feea:3074
* 48 ff01:2::/32 fe80:2::225:90ff:feea:3075
* 48 ff01:3::/32 fe80:3::225:90ff:feea:3076
* 48 ff01:4::/32 fe80:4::225:90ff:feea:3077
* 48 ff01:6::/32 fe80:6::225:90ff:fef3:8fc7
* 48 ff01:7::/32 fe80:7::225:90ff:fef3:8fc8
* 48 ff01:8::/32 fe80:8::225:90ff:fef3:8fc9
* 48 ff01:b::/32 ::1
* 48 ff01:d::/32 fe80:d::225:90ff:feea:3075
* 48 ff01:e::/32 fe80:e::225:90ff:feea:3074
* 48 ff01:f::/32 fe80:f::225:90ff:feea:3074
* 48 ff01:10::/32 fe80:10::225:90ff:feea:3074
* 48 ff01:11::/32 fe80:11::225:90ff:feea:3074
* 48 ff01:18::/32 fe80:18::225:90ff:feea:3074
* 48 ff01:19::/32 fe80:19::225:90ff:feea:3074
* 48 ff02:1::/32 fe80:1::225:90ff:feea:3074
* 48 ff02:2::/32 fe80:2::225:90ff:feea:3075
* 48 ff02:3::/32 fe80:3::225:90ff:feea:3076
* 48 ff02:4::/32 fe80:4::225:90ff:feea:3077
* 48 ff02:6::/32 fe80:6::225:90ff:fef3:8fc7
* 48 ff02:7::/32 fe80:7::225:90ff:fef3:8fc8
* 48 ff02:8::/32 fe80:8::225:90ff:fef3:8fc9
* 48 ff02:b::/32 ::1
* 48 ff02:d::/32 fe80:d::225:90ff:feea:3075
* 48 ff02:e::/32 fe80:e::225:90ff:feea:3074
* 48 ff02:f::/32 fe80:f::225:90ff:feea:3074
* 48 ff02:10::/32 fe80:10::225:90ff:feea:3074
* 48 ff02:11::/32 fe80:11::225:90ff:feea:3074
* 48 ff02:18::/32 fe80:18::225:90ff:feea:3074
* 48 ff02:19::/32 fe80:19::225:90ff:feea:3074
Network:
flags: S = Static
flags destination
*S 0 192.168.48.0/25 192.168.55.3
*S 0 192.168.48.128/25 192.168.55.3
*S 0 192.168.49.0/25 192.168.55.3
Nexthops:
Flags: * = nexthop valid
Nexthop Route Prio Gateway Iface
* 192.168.55.3 192.168.55.3/32 48 connected opt6_vip249 (UP,
master)
* 192.168.55.5 192.168.55.0/29 48 192.168.55.1 igb2_vlan300
(UP, 1000 Mbps)
IP:
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete
flags destination gateway lpref med aspath origin
*> 172.16.24.0/21 192.168.55.5 100 0 9059 i
AI*> 192.168.48.0/25 192.168.55.3 100 0 i
AI*> 192.168.48.128/25 192.168.55.3 100 0 i
AI*> 192.168.49.0/25 192.168.55.3 100 0 i
Neighbors:
BGP neighbor is 192.168.55.5, remote AS 9059
Description: AWS-DC MER Peer
BGP version 4, remote router-id 192.168.55.5
BGP state = Established, up for 00:04:09
Last read 00:00:23, holdtime 90s, keepalive interval 30s
Neighbor capabilities:
Multiprotocol extensions: IPv4 unicast
Route Refresh
Graceful Restart
4-byte AS numbers
Message statistics:
Sent Received
Opens 1 1
Notifications 0 0
Updates 2 2
Keepalives 9 10
Route Refresh 0 0
Total 12 13
Update statistics:
Sent Received
Updates 12 1
Withdraws 0 0
End-of-Rib 1 1
Local host: 192.168.55.1, Local port: 179
Remote host: 192.168.55.5, Remote port: 59288
... and lastly here is the traceroute from the client server:
tracert 172.16.24.7
Tracing route to 172.16.24.7 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.48.118
2 5 ms 2 ms 1 ms 81.27.95.83
3 1 ms 1 ms 1 ms 109.104.114.134
4 1 ms 1 ms 1 ms betelgeuse-hardy.c4l.co.uk
[109.104.114.105]
5 1 ms 2 ms 70 ms hardy-wolverine.c4l.co.uk [109.104.114.6]
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
Mark Relf
Principal Consultant
4sl Group, 4 Snow Hill, London EC1A 2DJ
t: +44 (0) 203 307 1053
m: +44 (0) 7868 842548
w: www.4sl.com <http://www.4sl.com/>
e: mark.r...@4sl.com
Planned away dates: None
Legal Disclaimer: The information in this email and any attachment is
confidential and may also be privileged. If you have received this message in
error please notify the sender and delete the message and attachments from your
system immediately. You are not entitled to retain, copy or use this email for
any purpose, nor disclose all or any part of its content to any other person.
_______________________________________________
List mailing list
List@lists.pfsense.org
https://lists.pfsense.org/mailman/listinfo/list
