I believe we may have discovered a bug in ScreenOS 6.3 for ISG-1000 platforms running in Transparent/Layer-2 mode.
The symptom is TCP traffic not being delivered to single IP Addresses behind the firewall. This happens on a frequency of about once a month, one IP address behind the firewall will simply stop receiving inbound TCP traffic. ICMP and UDP packets are unaffected and get delivered without issue. The device is under support and I am working with JTAC. Our next step is to wait until the condition happens again, and collect a packet traces of both the Untrust and Trust interface of the ISG. However, the bug is difficult to reproduce so I am reaching out to the JNSP community to see if anyone else has experienced this issue. The only way to stop the error condition and restore service to the affected IP is to reset the ISG. I tried clearing the session table, ARP table, MAC-learn table, and I tried disconnect/reconnecting the affected node’s Ethernet cable from the network to no avail. I also tried disabling TCP Syn Check, TCP Seq Check, and the HTTP ALG. HTTP session cache is also disabled. Only a reset of the ISG restores service, which wipes any useful debugging info about the problem. Packet traces collected from within the Trust zone confirm that the packets are not being delivered by the firewall. There is one curious output from debug flow basic. ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- When the traffic works correctly, it looks like this: ---------------------------------------------------------------------------- isg-1000-fw-> set ff src-ip 10.0.2.44 dst-ip 10.0.10.36 isg-1000-fw-> set ff src-ip 10.0.10.36 dst-ip 10.0.2.44 isg-1000-fw-> debug flow basic isg-1000-fw-> get db str **st: <V1-Untrust|ethernet1/1|Root|4f1> 483dd40: 35b4:10.0.2.44/e7b4->10.0.10.36/50,6,60 ****** 00338.0: <V1-Untrust/ethernet1/1> packet received [60]****** ipid = 13748(35b4), @0483dd40 packet passed sanity check. packet with vlan 1, vlan-group vlan1, vsd 0 v1-untrust:10.0.2.44/59316->10.0.10.36/80,6<Root> found mac 00e0deadbeef on ethernet1/4 flow_decap_vector IPv4 process no session found flow_first_sanity_check: in <v1-untrust>, out <v1-trust> policy search from zone 11-> zone 12 policy_flow_search policy search nat_crt from zone 11-> zone 12 RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 10.0.10.36, port 80, proto 6) No SW RPC rule match, search HW rule swrs_search_ip: policy matched id/idx/action = 104/1/0x1 Permitted by policy 104 No src xlate choose interface v1-trust as outgoing phy if session application type 6, name HTTP, nas_id 0, timeout 1800sec service lookup identified service 0. flow_first_final_check: in <v1-untrust>, out <v1-trust> existing vector list 2-4b0f7ac. Session (id:129990) created for first pak flow_first_install_session======> xpt: cache src mac in session xpt: cache dst mac in session Success installing work and forward sessions flow got session. flow session id 129990 skip ttl adjust for packet. transfer packet to hardware. ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- When the traffic fails, it looks like this: ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- **st: <V1-Untrust|ethernet1/1|Root|4f1> 4811b40: df23:10.0.2.44/b471->10.0.10.36/50,6,60 ****** 3026650.0: <V1-Untrust/ethernet1/1> packet received [60]****** ipid = 57123(df23), @04811b40 packet passed sanity check. packet with vlan 1, vlan-group vlan1, vsd 0 v1-untrust:10.0.2.44/46193->10.0.10.36/80,6<Root> found mac 00e0deadbeef on ethernet1/4 flow_decap_vector IPv4 process no session found flow_first_sanity_check: in <v1-untrust>, out <v1-trust> policy search from zone 11-> zone 12 policy_flow_search policy search nat_crt from zone 11-> zone 12 RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 10.0.10.36, port 80, proto 6) No SW RPC rule match, search HW rule swrs_search_ip: policy matched id/idx/action = 104/0/0x1 Permitted by policy 104 No src xlate choose interface v1-trust as outgoing phy if session application type 6, name HTTP, nas_id 0, timeout 1800sec service lookup identified service 0. flow_first_final_check: in <v1-untrust>, out <v1-trust> existing vector list 12-1aa2b174. Session (id:127841) created for first pak flow_first_install_session======> xpt: cache src mac in session xpt: cache dst mac in session Success installing work and forward sessions flow got session. flow session id 127841 skip ttl adjust for packet. tcp proxy processing... ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- (The IP and MAC addresses in this output have been sanitized) Notice the only difference is the last line, packets that fail are "tcp proxy processing" and packets that succeed are transferred to hardware. TCP Syn Proxy is disabled, and 'get counter flow' shows 0 hits for 'tcp proxy'. Has anyone else out there with an ISG running 6.3 in transparent mode experienced this issue?? -Nicholas _______________________________________________ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp