I can do that. I guess I should do the capture from tun1 as that is the place that the tcp-traffic is visible? My non-virtual nic is only seeing OpenVPN encapsulated UDP-traffic.
On 25 January 2017 at 23:48, Neal Cardwell <ncardw...@google.com> wrote: > On Wed, Jan 25, 2017 at 5:38 PM, Hans-Kristian Bakke <hkba...@gmail.com> > wrote: > >> Actually.. the 1-4 mbit/s results with fq sporadically appears again as I >> keep testing but it is most likely caused by all the unknowns between me an >> my testserver. But still, changing to pfifo_qdisc seems to normalize the >> throughput again with BBR, could this be one of those times where BBR and >> pacing actually is getting hurt for playing nice in some very variable >> bottleneck on the way? >> > > Possibly. Would you be able to take a tcpdump trace of each trial (headers > only would be ideal), and post on a web site somewhere a pcap trace for one > of the slow trials? > > For example: > > tcpdump -n -w /tmp/out.pcap -s 120 -i eth0 -c 1000000 & > > thanks, > neal > > > >> >> On 25 January 2017 at 23:01, Neal Cardwell <ncardw...@google.com> wrote: >> >>> On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke <hkba...@gmail.com> >>> wrote: >>> >>>> Hi >>>> >>>> Kernel 4.9 finally landed in Debian testing so I could finally test BBR >>>> in a real life environment that I have struggled with getting any kind of >>>> performance out of. >>>> >>>> The challenge at hand is UDP based OpenVPN through europe at around 35 >>>> ms rtt to my VPN-provider with plenty of available bandwith available in >>>> both ends and everything completely unknown in between. After tuning the >>>> UDP-buffers up to make room for my 500 mbit/s symmetrical bandwith at 35 ms >>>> the download part seemed to work nicely at an unreliable 150 to 300 mbit/s, >>>> while the upload was stuck at 30 to 60 mbit/s. >>>> >>>> Just by activating BBR the bandwith instantly shot up to around 150 >>>> mbit/s using a fat tcp test to a public iperf3 server located near my VPN >>>> exit point in the Netherlands. Replace BBR with qubic again and the >>>> performance is once again all over the place ranging from very bad to bad, >>>> but never better than 1/3 of BBRs "steady state". In other words "instant >>>> WIN!" >>>> >>> >>> Glad to hear it. Thanks for the test report! >>> >>> >>>> However, seeing the requirement of fq and pacing for BBR and noticing >>>> that I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE >>>> host with fq_codel on all physical interfaces, I was surprised to see that >>>> it worked so well. >>>> I then replaced pfifo_fast with fq and the performance went right down >>>> to only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained >>>> the performance at once. >>>> >>>> I have got some questions to you guys that know a lot more than me >>>> about these things: >>>> >>> 1. Do fq (and fq_codel) even work reliably in a VM? What is the best >>>> choice for default qdisc to use in a VM in general? >>>> >>> >>> Eric covered this one. We are not aware of specific issues with fq in VM >>> environments. And we have tested that fq works sufficiently well on Google >>> Cloud VMs. >>> >>> >>>> 2. Why do BBR immediately "fix" all my issues with upload through that >>>> "unreliable" big BDP link with pfifo_fast when fq pacing is a requirement? >>>> >>> >>> For BBR, pacing is part of the design in order to make BBR more "gentle" >>> in terms of the rate at which it sends, in order to put less pressure on >>> buffers and keep packet loss lower. This is particularly important when a >>> BBR flow is restarting from idle. In this case BBR starts with a full cwnd, >>> and it counts on pacing to pace out the packets at the estimated bandwidth, >>> so that the queue can stay relatively short and yet the pipe can be filled >>> immediately. >>> >>> Running BBR without pacing makes BBR more aggressive, particularly in >>> restarting from idle, but also in the steady state, where BBR tries to use >>> pacing to keep the queue short. >>> >>> For bulk transfer tests with one flow, running BBR without pacing will >>> likely cause higher queues and loss rates at the bottleneck, which may >>> negatively impact other traffic sharing that bottleneck. >>> >>> >>>> 3. Could fq_codel on the physical host be the reason that it still >>>> works? >>>> >>> >>> Nope, fq_codel does not implement pacing. >>> >>> >>>> 4. Do BBR _only_ work with fq pacing or could fq_codel be used as a >>>> replacement? >>>> >>> >>> Nope, BBR needs pacing to work correctly, and currently fq is the only >>> Linux qdisc that implements pacing. >>> >>> >>>> 5. Is BBR perhaps modified to do the right thing without having to >>>> change the qdisc in the current kernel 4.9? >>>> >>> >>> Nope. Linux 4.9 contains the initial public release of BBR from >>> September 2016. And there have been no code changes since then (just >>> expanded comments). >>> >>> Thanks for the test report! >>> >>> neal >>> >>> >> >
_______________________________________________ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat