It depends on the time space between packets. It seems that some NIC drivers are caching packets and sending out in chunks -- I believe that was the issue when we discovered it.

Anyways, long story short, if timing between packets is wider this isn't an issue, as timing narrows packets seem to come out of order. We built our own version that ensures packets are received on the far end before the next packet is sent, with this in place we can still blast traffic at a good rate and never get out of order packets, but it is unique to our environment.

James.

On 09/25/2012 04:07 PM, Richard Blalock wrote:
I suppose I misspoke. I'm pretty certain that tcpreplay is sending the packets in the correct order, and I'm aware that tcpreplay doesn't "get" packets as if it were a live host. But my pcap file is opening four concurrent sockets like this.

- client sends syn
- server sends syn/ack
- client sends syn (socket #2)
- server sends syn/ack
- client sends syn (socket #3)
- server sends syn/ack
- client sends syn (socket #4)
- server sends syn/ack
- client finally sends ack for first socket

but when I replay the pcap file, the packet capture on the server side looks like this

- client sends syn
- server sends syn/ack
- server sends syn/ack (for the second socket. hasn't received syn for this socket yet.) - Things go haywire from here. The correct packets are being sent and received, but as wireshark sniffs them out they are in the wrong order.

I figured tcpreplay was just replaying packets faster than the firewall could forward them, and the second syn packet just didn't get through the firewall before the second syn/ack was played. I also figured that the solution was just to slow down the replay and the traffic would look normal and in the correct sequence. I was right, but I didn't realize that the speed would have to be as slow as 1/100th in order for this to work.

I guess my question is this. Is it common for a firewall to cause packets to slow down this much? It's especially peculiar when you consider the setup is the same as when the pcap file was originally created. Thanks for your patience.

On Tue, Sep 25, 2012 at 2:35 PM, Aaron Turner <[email protected] <mailto:[email protected]>> wrote:

    On Tue, Sep 25, 2012 at 7:20 PM, Richard Blalock
    <[email protected] <mailto:[email protected]>> wrote:
    > Tcpreplay is getting packets out of sequence if I replay them at
    anything

    Not sure what you mean by "Tcpreplay getting packets out of sequence"?
     Tcpreplay doesn't "get" packets, it only sends them.

    > even close to original speed. I have to slow them down to around
    1/100th of
    > the original speed in order for them to get back and forth in
    correct
    > sequence. The packets are being replayed through the exact same
    DUT, so the
    > latency should be identical. I've manually edited the ARP
    tables, so there
    > shouldn't be any ARP requests to slow it down. Any idea why this is
    > happening?

    Are you saying that the DUT is seeing the packets out of order?

    Tcpreplay only sends the packets in order of the pcap- it's not
    multi-threaded and there's no mechanism to skip around a pcap file.
    If you're seeing a problem with out of order packets being sent, then
    it's something to do with your kernel, or NIC/NIC driver.  If you're
    running linux on your tcpreplay box, try:

    tcpdump -i any -s0 -w test.pcap

    and see if you're seeing the problem there.

    That or something with how you are checking for packet order is the
    cause of the problem or some intermediary device causing the problem.
    For example if you're going through a modern switch which has a store
    & forward architecture, that can change the order of packets.

    Sorry I don't have a good answer, but I really have no idea what your
    test bed looks like and have no way of reproducing it.

    --
    Aaron Turner
    http://synfin.net/         Twitter: @synfinatic
    http://tcpreplay.synfin.net/ - Pcap editing and replay tools for
    Unix & Windows
    Those who would give up essential Liberty, to purchase a little
    temporary
    Safety, deserve neither Liberty nor Safety.
        -- Benjamin Franklin
    "carpe diem quam minimum credula postero"

    
------------------------------------------------------------------------------
    Live Security Virtual Conference
    Exclusive live event will cover all the ways today's security and
    threat landscape has changed and how IT managers can respond.
    Discussions
    will include endpoint security, mobile security and the latest in
    malware
    threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
    _______________________________________________
    Tcpreplay-users mailing list
    [email protected]
    <mailto:[email protected]>
    https://lists.sourceforge.net/lists/listinfo/tcpreplay-users
    Support Information: http://tcpreplay.synfin.net/trac/wiki/Support




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/


_______________________________________________
Tcpreplay-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tcpreplay-users
Support Information: http://tcpreplay.synfin.net/trac/wiki/Support

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Tcpreplay-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tcpreplay-users
Support Information: http://tcpreplay.synfin.net/trac/wiki/Support

Reply via email to