Re: [Bloat] T-Mobile LTE buffer bloat

[Stefan Alfredsson]

Hi, Stephen, the list,

We're running a 3G/4G measurement site at Karlstad university, with 
mobile broadband subscriptions from the four major telecom operators in 
Sweden. I was curious to see how measurements in our network would 
compare to yours, so I did a measurement run with netalyzer.


To summarize, the measured buffering is well below one second for all 
tests, although the results are not directly comparable since we are 
using directly attached USB modems (Huawei E392 with Ubuntu 12.04) 
instead of a separate router box. However, the numbers might be 
interesting in a more general perspective.


Tele2 LTE: Upload 12 Mbit/s, 240 ms buffering, Download > 20  Mbit/s, 
buffering not measurable

http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6593-7f837e62-5321-488e-b480

Tele2 HSPA+: Upload 1.1 Mbit/s, 630 ms buffering, Download > 20 Mbit/s, 
509 ms buffering

http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-23052-a0e80cbb-9a7c-44e0-ba88

Telenor LTE: Upload 16  Mbit/s, 190  ms buffering, Download >20  Mbit/s, 
470 ms buffering

http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31617-45bca5e3-53ef-450e-a9bf

Telenor HSPA+: Upload 3.4 Mbit/s,  200 ms buffering, Download > 14 
Mbit/s, 490 ms buffering

http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31612-c6d0d98d-8bcf-4a03-832d

Telia LTE: Upload 18 Mbit/s, 140 ms buffering, Download > 20 Mbit/s, 
buffering not measurable

http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31599-1ebc684e-c164-4786-832a

Telia HSPA+: Upload 1.1 Mbit/s, 630  ms buffering, Download > 20 Mbit/s, 
640 ms buffering

http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-22984-40c59ae2-0c36-4e67-9039
(note similarity with Tele2 HSPA+ - IIUC Tele2 and Telia share the 3G 
network infrastructure)


Tre LTE: Upload 19 Mbit/s, 150 ms buffering, Download > 20 Mbit/s, 98 ms 
buffering

http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31648-86a273a5-82ae-4929-a634

Tre HSPA+: Upload 3.5 Mbit/s, 190 ms buffering, Download > 8.6 Mbit/s, 
780 ms buffering

http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6609-3316da33-9ea6-41ca-a8ac


For reference, I also made a measurement from the same host, but using a 
wired connection over the Swedish university network: Upload was 
measured to > 20 Mbit/s and download to 15 Mbit/s, with no measurable 
buffering.
http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6563-078896ce-dc0a-4333-9119 



Regards,
  Stefan Alfredsson



On 10/31/13 00:25 , Stephen Hemminger wrote:

I got one of these Samsung LTE hotspot.
Not surprisingly it has huge bloat and a stupid http proxy
that netalyzer claims rewrites images.
  
Bandwidth: Up 1.6 Mbit/sec Down 4.3Mbit/sec

Latency: 140ms 0% loss
Buffering: Uplink 5100ms Down 1800ms

How can the uplink side be so bad! 5 seconds???

Might even return it as defective. You can't even add a review on their
website. Probably they would end taking down like Apple.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat



___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] mobile broadband buffer bloat

[Stefan Alfredsson]

On 10/31/13 03:32 , Dave Taht wrote:

On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:

Hi, Stephen, the list,

We're running a 3G/4G measurement site at Karlstad university, with
mobile broadband subscriptions from the four major telecom operators
in Sweden. I was curious to see how measurements in our network
would compare to yours, so I did a measurement run with netalyzer.

Groovy!

Can I ask that you try something that can more likely stress
out that bus than netanlyzer? The netperf-wrappers tools
(notably the tcp_bidirectional and rrul test), use C, rather than java.


Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4 
during July to September, roughly 3-4 times per day, in total over 3000 
individual tests.


I think there's a lot of interesting information in there, but I have 
not done aggregate analysis yet (eg. looking at time-of-day effects, 
long-term trends, or the 10.000-students-arriving-in-august effect :-) )


I packed the rrul-dumps for one day (1 sept 2013) and put it here: 
http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz .


Here's an overview of this set, showing the per-run mean RTT, download 
and upload throughput:


[alfs@sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do 
test=$(echo $f | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep 
-A 4 'avg:'| awk -v test=$test '/Mean:/ { if (NR==3) { rtt=$2 } if 
(NR==10) {dl=$2} if (NR==16) {ul=$2; printf "%s:\tRTT: %.1f ms DL: %.1f 
Mbit/s UL: %.1f Mbit/s\n", test, rtt, dl, ul}}'; done

Tele2/35G:  RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s
Tele2/35G:  RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s
Tele2/35G:  RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s
Tele2/35G:  RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s
Tele2/4G:   RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s
Tele2/4G:   RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s
Tele2/4G:   RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s
Tele2/4G:   RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s
Telenor/35G:RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s
Telenor/35G:RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s
Telenor/35G:RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s
Telenor/35G:RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s
Telenor/4G: RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s
Telenor/4G: RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s
Telenor/4G: RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s
Telenor/4G: RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s
Telia/35G:  RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s
Telia/35G:  RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s
Telia/35G:  RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s
Telia/35G:  RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s
Telia/4G:   RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s
Telia/4G:   RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s
Telia/4G:   RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s
Tre/35G:RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s
Tre/35G:RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s
Tre/35G:RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
Tre/35G:RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s


To summarize, the measured buffering is well below one second for
all tests, although the results are not directly comparable since we
are using directly attached USB modems (Huawei E392 with Ubuntu
12.04)

ubuntu has backported fq_codel as far as 12.04. What kernel are you using?

On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP

The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with 
web10g patches, since we were interested in extracting TCP variable data 
not available from packet traces.



instead of a separate router box. However, the numbers might
be interesting in a more general perspective.

Yes, they are. These are repeatable?




For the netalyzer I've only done singular tests, but it can easily be 
repeated if needed (probably not).
The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, 
and for 3.5G it's 573 ms.
This indicates that results are repeatable, and the set from 2013-09-01 
is representative.


"below one second" was in relation to the >5 second RTT's, ideally it 
should be much lower of course. As a side note, we also do measurements 
in unloaded networks, and for 4G we get around 40-60 ms RTT (ICMP and 
TCP packets).


Also, you might be interested in a paper we published in June, that 
reports on measurements of bloat for short flows in relation to 
different congestion control algorithms, in 3G, 3.5G and 4G (but for a 
single operator).

See http://urn.kb.se/resolve?urn=urn:nbn:se:kau:diva-27779
with fulltext at 

Re: [Bloat] AQM and PPP on Linux

[Stefan Alfredsson]

* Quoting Dave Taht  [28 Jul-15 16:44]:
> On Tue, Jul 28, 2015 at 3:09 PM, Juliusz Chroboczek
>  wrote:
> > I'm currently away from home, and using a 3G modem for Internet access.
> > I've found out that both NetworkManager and wvdial/pppd setup the
> > interface to use pfifo_fast (with a qlen of a mere 3 packets!).  Setting
> > fq_codel manually appears to work fine, but needs to be redone every time
> > the modem has a hiccup.
> 
> However, the netusb-related drivers, and the 3g devices themselves,
> contain oft-huge buffers that reduce the effectiveness of any aqm or
> fq system, sometimes to immeasurability.
> 
> I would be very interested in flent benchmarks of your 3g device with
> the 3 packet txqueue and with fq_codel, for the tcp_upload, rrul, and
> rrul_be tests.


I have a station with mobile broadband USB modems (4xHuawei E392),
directly connected to a Linux box running kernel 3.16.3, with
a (non-public) netperf server (same kernel) hosted on the well-connected
Swedish university network.

I also have a 3 packet txqueue, and ran the flent tcp_upload, rrul, rrul_be
for 3.5G (HSPA+) and 4G (LTE) networks for one of the operators/modems.
For comparison, I ran both pfifo_fast and fq_codel.

The execution script, output and flent result files are available at
https://www.dropbox.com/s/n5tc1dhtu9jdplz/bloatlist-ppp-150729.zip

>From flent --gui inspections, I'd say there is about a factor 10 increase
from base to load latency (from around 40-50 ms to around 400-500 ms),
for both pfifo_fast and fq_codel, for both link types,
with some transients above 1000 ms.


Juliusz: Regarding data usage, the 4G measurements consumed 1.4 Gbyte
download and 357 Mbyte upload - but it all depends on the network
throughput as the experiments are time bound (by default 60 seconds).
For 3.5G 100 Mbyte data was sent, and guesstimately 500 Mbyte received
(sorry, didn't capture the amount; but it can probably be found in
the trace files if necessary. However it all depends on your link 
throughput which will be different from mine anyway).


Cheers,
 Stefan

-- 
Stefan Alfredsson, PhD Tel: +46 (0) 54 700 1668
Datavetenskap  Kontor: 21E-414 (Hus Vanern)
Karlstads universitet  PGP 0xB19B4B16
SE-651 88 Karlstad http://www.cs.kau.se/stefalfr/
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] bufferbloat at high edge rates

[Stefan Alfredsson]

I had the same problem, getting no bloat report. I tested just now 
running as root, and got bloat measurements in 
http://www.dslreports.com/speedtest/6166098 and 
http://www.dslreports.com/speedtest/6166295


From Firefox, I get somewhat higer latency (~100 ms) versus ~20-60 ms 
via the command line client:


http://www.dslreports.com/speedtest/6130740
http://www.dslreports.com/speedtest/6130727
http://www.dslreports.com/speedtest/6130708
http://www.dslreports.com/speedtest/6130690


Two things to note:

- Firefox tests were run yesterday at around lunchtime  (~12.00 CET), 
and CLI tests just now (~06.40 CET). So time-of-day effect may be a 
reason for less bloat now. I'll do a better comparison when I get to my 
desktop.


- CLI tests were run in a docker container, for security purposes. I 
used host networking so it should not have affected measurements much, 
but still. This was how it was executed:



// downloaded the dslreports cli tool to my host /tmp directory, mapping 
/tmp to /host in the debian container


$ docker run --rm -t -i -v /tmp:/host:ro debian

root@db0ea060caa7:/# apt-get update ; apt-get install ca-certificates
[... snip ...]
root@db0ea060caa7:/# /host/dslrcli-linux-amd64
Selecting nearest servers
Download Testing.
Upload Testing.
Uploading results...
Download : 883.56 Megabit/sec Upload : 895.32 Megabit/sec
http://www.dslreports.com/speedtest/6166098



A better option would be using CAP_NET_RAW, I'll see if this works 
instead of running with full root privs.


/Stefan



On 16/11/16 05:09, jb wrote:
It has to run as root / Admin in order to do ICMP in order to test 
buffer bloat.


If you run it under a non privileged user account it cannot get 
permission for ICMP, so although it locates the nearest servers using 
http ping, it isn't doing any buffer bloat testing.


I'm not sure that is the issue but that's the first thing that comes 
to mind..



On Wed, Nov 16, 2016 at 11:34 AM, David Lang > wrote:


On Tue, 15 Nov 2016, jb wrote:

The command line tool is available to anyone now (Windows, OSX
and linux),
it does buffer bloat probing, using ICMP if run as root, and
is immune to
any browser issues. It can be downloaded here from the sticky:
http://www.dslreports.com/forum/speedtestbinary



This does not seem to be reporting any bloat info (I've run it a
couple times)

http://www.dslreports.com/speedtest/6156013


David Lang
___
Bloat mailing list
Bloat@lists.bufferbloat.net 
https://lists.bufferbloat.net/listinfo/bloat





___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat




___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] bufferbloat at high edge rates

[Stefan Alfredsson]

* Quoting Stefan Alfredsson  [16 Nov-16 07:00]:
> - Firefox tests were run yesterday at around lunchtime  (~12.00 CET), and
> CLI tests just now (~06.40 CET). So time-of-day effect may be a reason for
> less bloat now. I'll do a better comparison when I get to my desktop.

I've done tests with CLI, Firefox and Chrome; repeated 3 times in
a round-robin fashion (cli, ff, chrome, cli, ff, chrome, cli, ff, chrome).

In summary:

- There is 5-10 ms additional latency during all tests.
- CLI and Chrome have the same latency figures, 23 ms base latency 
  Firefox has a base latency of 100 ms RTT, most likely due to US
  responders being used by Firefox and EU responders being used by CLI and 
Chrome.  
- Chrome achives 30-40 Mbps higher throughput than CLI.


Results from a standard Ubuntu Xenial system running kernel 4.4.0-45-generic 
#66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016

Dslrcli version 0.1 - 15-Nov-2016
test 1 http://www.dslreports.com/speedtest/6169686
test 4 http://www.dslreports.com/speedtest/6169764
test 7 http://www.dslreports.com/speedtest/6169798


Firefox 49.0.2:
test 2 http://www.dslreports.com/speedtest/6169706
test 5 http://www.dslreports.com/speedtest/6169776
test 8 http://www.dslreports.com/speedtest/6169819


Google Chrome 54.0.2840.71
test 3 http://www.dslreports.com/speedtest/6169747
test 6 http://www.dslreports.com/speedtest/6169788
test 9 http://www.dslreports.com/speedtest/6169833



> A better option would be using CAP_NET_RAW, I'll see if this works instead
> of running with full root privs.

Yes, copying the binary to /usr/bin and setting cap_net_raw made it report 
bloat.
(sudo setcap cap_net_raw+ep /usr/bin/dslrcli-linux-amd64)

/Stefan

-- 
Stefan Alfredsson, PhD Tel: +46 (0) 54 700 1668
Datavetenskap  Kontor: 21E-414 (Hus Vanern)
Karlstads universitet  PGP 0xB19B4B16
SE-651 88 Karlstad http://www.cs.kau.se/stefalfr/
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] different speeds on different ports? (benchmarking fun)

[Stefan Alfredsson]

On 2017-08-30 07:15, Aaron Wood wrote:



The current hypothesis that we have is that this is due to either 
traffic class, or the ports that traffic are running on.  I've ruled 
out the ping streams, as a parallel set of netperf tcp_maerts 
downloads has the same 120Mbps roof.


Also think of server differentiation (maybe full capacity to known 
measurement servers, shaped for other destinations). Does dslreports 
have flent/netperf responders on their servers as well?




It would be interesting if we could run some netperf tests using port 
80/443 for the listening socket for the data connection (although if 
doing deep-packet inspection, we might need to use an actual HTTP 
transfer).


UDP 443 (QUIC) would also be interesting for comparison.

/Stefan
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] vyatta in AT&T 5G gear

[Stefan Alfredsson]

On 2018-10-16 11:31, Mikael Abrahamsson wrote:


On Mon, 15 Oct 2018, Dave Taht wrote:


Vyos (the open source fork of vyatta) was one of the first to add
fq_codel support... I wonder

http://linuxgizmos.com/att-releases-white-box-spec-for-its-linux-based-5g-routers/ 



Isn't Vyos just running the Linux kernel for forwarding? So they 
received fq_codel for free when the Linux kernel got support for it? 
They just had to make it configurable?




Yes, according to this blog post, 
http://www.five-ten-sg.com/mapper/blog/Bufferbloat%20solved%20with%20Vyos


"Now that Vyos "helium" is available with a Linux 3.13 kernel, the 
fq_codel queueing discipline can be used to solve many bufferbloat 
issues. The nightly "lithium" builds contain my patches that allow 
fq_codel to be used via the native Vyos configuration system."


Anyway it's nice to see the Vyatta heritage living on in it's various 
forms (the AT&T "production hardened" Vyatta, to the Ubiquity EdgeMax 
and some UniFi devices, to the VyOS open version and now the future 
plans with dNOS -> DANOS.


/Stefan



___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat