Re: [Make-wifi-fast] [PATCH RFC v5 3/4] mac80211: Add airtime accounting and scheduling to TXQs
On Fri, Oct 12, 2018 at 12:38 AM Rajkumar Manoharan wrote: > > On 2018-10-11 03:38, Toke Høiland-Jørgensen wrote: > > Rajkumar Manoharan writes: > > > >> Hmm... mine is bit different. txqs are refilled only once for all > >> txqs. > >> It will give more opportunity for non-served txqs. drv_wake_tx_queue > >> won't be > >> called from may_tx as the driver anyway will not push packets in > >> pull-mode. > > > > So, as far as I can tell, this requires the hardware to "keep trying"? > > I.e., if it just stops scheduling a TXQ after may_transmit() returns > > false, there is no guarantee that that TXQ will ever get re-awoken > > unless a new packet arrives for it? > > > That is true and even now ath10k operates the same way in pull mode. Not > just packet arrival, even napi poll routine tries to pushes the packets. > One more thing, fetch indication may pull ~4ms/8ms of packets from each > tid. > This makes deficit too low and so refilling txqs by just airtime_weight > becomes > cumbersome. In may_transmit, the deficit are incremented by 20 * > airtime_weight. > In future this will be also replaced by station specific quantum. we can > revisit > this once BQL in place. Performance issue is resolved by this approach. > Do you foresee any issues? I'll have some time in the coming weeks to be able to test this stuff. I'm mostly interested in algorithmic correctness more than the API changes... Is there a version of these patches that is stable enough on ath9 or ath10k? Do I foresee any issues? Jeeze, no, we *never* have any issues with wifi. "fetch indication may pull ~4ms/8ms of packets from each tid" made me really twitchy. > > #define IEEE80211_TXQ_MAY_TX_QUANTUM 20 > bool ieee80211_txq_may_transmit(struct ieee80211_hw *hw, > struct ieee80211_txq *txq) > { > struct ieee80211_local *local = hw_to_local(hw); > struct txq_info *txqi = to_txq_info(txq); > struct sta_info *sta; > u8 ac = txq->ac; > > lockdep_assert_held(>active_txq_lock[ac]); > > if (!txqi->txq.sta) > goto out; > > sta = container_of(txqi->txq.sta, struct sta_info, sta); > if (sta->airtime[ac].deficit >= 0) > goto out; > > list_for_each_entry(txqi, >active_txqs[ac], > schedule_order) { > if (!txqi->txq.sta) > continue; > sta = container_of(txqi->txq.sta, struct sta_info, sta); > sta->airtime[ac].deficit += > (IEEE80211_TXQ_MAY_TX_QUANTUM * > sta->airtime_weight); > } > > return false; > > out: > list_del_init(>schedule_order); > return true; > } > > -Rajkumar > ___ > Make-wifi-fast mailing list > make-wifi-f...@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740
Re: Tool to debug wifi pkt sniffs?
On Wed, Oct 10, 2018 at 1:44 PM Ben Greear wrote: > > On 10/10/2018 12:13 PM, Dave Taht wrote: > > On Wed, Oct 10, 2018 at 10:10 AM Ben Greear wrote: > >> > >> On 10/03/2018 01:29 PM, Dave Taht wrote: > >>> On Wed, Oct 3, 2018 at 1:16 PM Toke Høiland-Jørgensen > >>> wrote: > >>>> > >>>> Ben Greear writes: > >>>> > >>>>> Hello, > >>>>> > >>>>> I often find myself wanting to figure out what equipment is to blame > >>>>> (and why) > >>>>> in a wifi environment. > >>>>> > >>>>> I am thinking writing a tool that would parse a pcap file and look at > >>>>> frames > >>>>> in enough detail to flag block-ack bugs, rate-ctrl bugs, guess at the > >>>>> sniffer's > >>>>> capture ability, etc. > >>>>> > >>>>> Does anyone have anything already written that they would like to > >>>>> share, or know > >>>>> of projects that might already do some of this? > >>>> > >>>> Not sure if this fits your criteria, but Sven's tool to create airtime > >>>> charts from packet sniffing data immediately came to mind: > >>>> > >>>> https://github.com/cloudtrax/airtime-pie-chart > >>> > >>> I have used that. Oy, it's a PITA. Some of kathie's code over here > >>> (example: https://github.com/pollere/pping ) uses the slightly less > >>> painful http://libtins.github.io/ library for parsing packets. > >> > >> I couldn't find anything that did what I wanted, so I wrote my own. > >> > >> The (perl) code is in the wifi-diag directory of this public repo: > >> > >> https://github.com/greearb/lanforge-scripts > >> > >> The rest of the scripts in that repo are not related to the wifi-diag > >> script, so just ignore those. > >> > >> Here is example output for what I have so far: > >> > >> https://www.candelatech.com/oss/wifi-diag/netgear-up-5s/index.html > > > > I *miss* writing in perl. :) I did take a quick look at the perl. It's been too long > > > > My guess from looking at that output that that was a udp flood test. > > Do I win the internets? > > Yes, UDP upload test with 20 emulated stations, sending ~500 byte UDP frames. > One thing we notice in the case we are debugging, is > that the average time from transmitter station device receiving BA from the AP > to the transmitter station device putting the next AMPDU frame on air > is 0.728ms for the problem AP, and 0.448ms for the good AP. I'm not big on averages. A cdf plot would show you if the delay was consistent across the range or had a knee in it. > > I checked that the wmm config in the beacons for the two APs is the same. > > I am at a loss as to what could cause this delay, other than possibly the > problem > AP has a funky transmitter than puts a bit of extra noise on the air after it > is done transmitting a frame? A possible explanation would be garbage at one or more tried mcs rate (not successfully captured). minstrel at least tries multiple mcs rates. > > The problem AP also has 5% retransmits vs about 2% for the good AP, and > problem AP > is typically using MCS8 instead of MCS9, but even so, I do not see how that > would explain > the extra BA to AMPDU delay. It's highly probable I'm misunderstanding you and would need to look directly at the cap. "typically using" says to me "more often trying the wrong rate" > > Thanks, > Ben > > > -- > Ben Greear > Candela Technologies Inc http://www.candelatech.com > -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740
Re: Tool to debug wifi pkt sniffs?
On Wed, Oct 10, 2018 at 10:10 AM Ben Greear wrote: > > On 10/03/2018 01:29 PM, Dave Taht wrote: > > On Wed, Oct 3, 2018 at 1:16 PM Toke Høiland-Jørgensen wrote: > >> > >> Ben Greear writes: > >> > >>> Hello, > >>> > >>> I often find myself wanting to figure out what equipment is to blame (and > >>> why) > >>> in a wifi environment. > >>> > >>> I am thinking writing a tool that would parse a pcap file and look at > >>> frames > >>> in enough detail to flag block-ack bugs, rate-ctrl bugs, guess at the > >>> sniffer's > >>> capture ability, etc. > >>> > >>> Does anyone have anything already written that they would like to share, > >>> or know > >>> of projects that might already do some of this? > >> > >> Not sure if this fits your criteria, but Sven's tool to create airtime > >> charts from packet sniffing data immediately came to mind: > >> > >> https://github.com/cloudtrax/airtime-pie-chart > > > > I have used that. Oy, it's a PITA. Some of kathie's code over here > > (example: https://github.com/pollere/pping ) uses the slightly less > > painful http://libtins.github.io/ library for parsing packets. > > I couldn't find anything that did what I wanted, so I wrote my own. > > The (perl) code is in the wifi-diag directory of this public repo: > > https://github.com/greearb/lanforge-scripts > > The rest of the scripts in that repo are not related to the wifi-diag script, > so just ignore those. > > Here is example output for what I have so far: > > https://www.candelatech.com/oss/wifi-diag/netgear-up-5s/index.html I *miss* writing in perl. :) My guess from looking at that output that that was a udp flood test. Do I win the internets? > > The general idea is to get a performance test going, and then use tshark or > similar > to grab a short sample (my script is slow, it can process only about 400 > packets per second > on my desktop, so a 5 sec capture at full speed takes around 5 minutes to > process), > and then pipe that decoded pcap into my script. > > It tries to pay attention to latencies between block-ack and next AMPDU frame, > MCS distributions, packet-type distributions, retries, and other > such things. I'm guessing tweaking wmm settings (or changing QoS in the > generated traffic) > would be visible in this kind of metric, for instance. > > The goal is to be able to answer the question of why one AP is faster or > slower than another > when running the same test case. > > Feedback (and even patches) is welcome...what other things can I report that > would > be helpful? > > > Thanks, > Ben > > -- > Ben Greear > Candela Technologies Inc http://www.candelatech.com > -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740
Re: [PATCH 9/9] mac80211: rc80211_minstrel: remove variance / stddev calculation
On Sat, Oct 6, 2018 at 11:18 AM Felix Fietkau wrote: > > On 2018-10-06 19:59, Dave Taht wrote: > > On Sat, Oct 6, 2018 at 10:37 AM Felix Fietkau wrote: > >> > >> When there are few packets (e.g. for sampling attempts), the exponentially > >> weighted variance is usually vastly overestimated, making the resulting > >> data > >> essentially useless. As far as I know, there has not been any practical use > >> for this, so let's not waste any cycles on it. > >> > >> Signed-off-by: Felix Fietkau > >> --- > >> net/mac80211/rc80211_minstrel.c| 6 - > >> net/mac80211/rc80211_minstrel.h| 26 +- > >> net/mac80211/rc80211_minstrel_debugfs.c| 14 > >> net/mac80211/rc80211_minstrel_ht_debugfs.c | 14 > >> 4 files changed, 9 insertions(+), 51 deletions(-) > >> > >> diff --git a/net/mac80211/rc80211_minstrel.c > >> b/net/mac80211/rc80211_minstrel.c > >> index dead57ba9eac..a34e9c2ca626 100644 > >> --- a/net/mac80211/rc80211_minstrel.c > >> +++ b/net/mac80211/rc80211_minstrel.c > >> @@ -167,12 +167,6 @@ minstrel_calc_rate_stats(struct minstrel_rate_stats > >> *mrs) > >> if (unlikely(!mrs->att_hist)) { > >> mrs->prob_ewma = cur_prob; > >> } else { > >> - /* update exponential weighted moving variance */ > >> - mrs->prob_ewmv = minstrel_ewmv(mrs->prob_ewmv, > >> - cur_prob, > >> - mrs->prob_ewma, > >> - EWMA_LEVEL); > >> - > >> /*update exponential weighted moving avarage */ > >> mrs->prob_ewma = minstrel_ewma(mrs->prob_ewma, > >>cur_prob, > >> diff --git a/net/mac80211/rc80211_minstrel.h > >> b/net/mac80211/rc80211_minstrel.h > >> index 54b2b2c3e10a..23ec953e3a24 100644 > >> --- a/net/mac80211/rc80211_minstrel.h > >> +++ b/net/mac80211/rc80211_minstrel.h > >> @@ -35,19 +35,6 @@ minstrel_ewma(int old, int new, int weight) > >> return old + incr; > >> } > >> > >> -/* > >> - * Perform EWMV (Exponentially Weighted Moving Variance) calculation > >> - */ > > > > I worry about this one. where are you getting your proof from? > I've done quite a few measurements myself to see if this can be usable > for further rate control improvements or for the upcoming TPC work. > The data this generates simply fluctuates wildly and incoherently based > on the sampling behavior, making it completely useless. > Together with Thomas (who introduced this code), I tried a few times to > fix this, but couldn't find any way to make it coherent and usable. > > Thomas and I both agreed that it's better to just remove it until > somebody has a better idea what to do. > > Also, this was only used for debugfs statistics, not for any actual rate > control behavior. OK, thanks. I'm totally delighted to see this patchset otherwise. > - Felix -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740
Re: [PATCH 9/9] mac80211: rc80211_minstrel: remove variance / stddev calculation
On Sat, Oct 6, 2018 at 10:37 AM Felix Fietkau wrote: > > When there are few packets (e.g. for sampling attempts), the exponentially > weighted variance is usually vastly overestimated, making the resulting data > essentially useless. As far as I know, there has not been any practical use > for this, so let's not waste any cycles on it. > > Signed-off-by: Felix Fietkau > --- > net/mac80211/rc80211_minstrel.c| 6 - > net/mac80211/rc80211_minstrel.h| 26 +- > net/mac80211/rc80211_minstrel_debugfs.c| 14 > net/mac80211/rc80211_minstrel_ht_debugfs.c | 14 > 4 files changed, 9 insertions(+), 51 deletions(-) > > diff --git a/net/mac80211/rc80211_minstrel.c b/net/mac80211/rc80211_minstrel.c > index dead57ba9eac..a34e9c2ca626 100644 > --- a/net/mac80211/rc80211_minstrel.c > +++ b/net/mac80211/rc80211_minstrel.c > @@ -167,12 +167,6 @@ minstrel_calc_rate_stats(struct minstrel_rate_stats *mrs) > if (unlikely(!mrs->att_hist)) { > mrs->prob_ewma = cur_prob; > } else { > - /* update exponential weighted moving variance */ > - mrs->prob_ewmv = minstrel_ewmv(mrs->prob_ewmv, > - cur_prob, > - mrs->prob_ewma, > - EWMA_LEVEL); > - > /*update exponential weighted moving avarage */ > mrs->prob_ewma = minstrel_ewma(mrs->prob_ewma, >cur_prob, > diff --git a/net/mac80211/rc80211_minstrel.h b/net/mac80211/rc80211_minstrel.h > index 54b2b2c3e10a..23ec953e3a24 100644 > --- a/net/mac80211/rc80211_minstrel.h > +++ b/net/mac80211/rc80211_minstrel.h > @@ -35,19 +35,6 @@ minstrel_ewma(int old, int new, int weight) > return old + incr; > } > > -/* > - * Perform EWMV (Exponentially Weighted Moving Variance) calculation > - */ I worry about this one. where are you getting your proof from? > -static inline int > -minstrel_ewmv(int old_ewmv, int cur_prob, int prob_ewma, int weight) > -{ > - int diff, incr; > - > - diff = cur_prob - prob_ewma; > - incr = (EWMA_DIV - weight) * diff / EWMA_DIV; > - return weight * (old_ewmv + MINSTREL_TRUNC(diff * incr)) / EWMA_DIV; > -} > - > struct minstrel_rate_stats { > /* current / last sampling period attempts/success counters */ > u16 attempts, last_attempts; > @@ -56,11 +43,8 @@ struct minstrel_rate_stats { > /* total attempts/success counters */ > u32 att_hist, succ_hist; > > - /* statistis of packet delivery probability > -* prob_ewma - exponential weighted moving average of prob > -* prob_ewmsd - exp. weighted moving standard deviation of prob */ > + /* prob_ewma - exponential weighted moving average of prob */ > u16 prob_ewma; > - u16 prob_ewmv; > > /* maximum retry counts */ > u8 retry_count; > @@ -140,14 +124,6 @@ struct minstrel_debugfs_info { > char buf[]; > }; > > -/* Get EWMSD (Exponentially Weighted Moving Standard Deviation) * 10 */ > -static inline int > -minstrel_get_ewmsd10(struct minstrel_rate_stats *mrs) > -{ > - unsigned int ewmv = mrs->prob_ewmv; > - return int_sqrt(MINSTREL_TRUNC(ewmv * 1000 * 1000)); > -} > - > extern const struct rate_control_ops mac80211_minstrel; > void minstrel_add_sta_debugfs(void *priv, void *priv_sta, struct dentry > *dir); > > diff --git a/net/mac80211/rc80211_minstrel_debugfs.c > b/net/mac80211/rc80211_minstrel_debugfs.c > index 698a668b5316..c8afd85b51a0 100644 > --- a/net/mac80211/rc80211_minstrel_debugfs.c > +++ b/net/mac80211/rc80211_minstrel_debugfs.c > @@ -70,14 +70,13 @@ minstrel_stats_open(struct inode *inode, struct file > *file) > p = ms->buf; > p += sprintf(p, "\n"); > p += sprintf(p, > -"best __rate_ > statisticslast___sum-of\n"); > +"best __rate_statistics___ > last___sum-of\n"); > p += sprintf(p, > -"rate [name idx airtime max_tp] [avg(tp) avg(prob) > sd(prob)] [retry|suc|att] [#success | #attempts]\n"); > +"rate [name idx airtime max_tp] [avg(tp) avg(prob)] > [retry|suc|att] [#success | #attempts]\n"); > > for (i = 0; i < mi->n_rates; i++) { > struct minstrel_rate *mr = >r[i]; > struct minstrel_rate_stats *mrs = >r[i].stats; > - unsigned int prob_ewmsd; > > *(p++) = (i == mi->max_tp_rate[0]) ? 'A' : ' '; > *(p++) = (i == mi->max_tp_rate[1]) ? 'B' : ' '; > @@ -93,15 +92,13 @@ minstrel_stats_open(struct inode *inode, struct file >
Re: Tool to debug wifi pkt sniffs?
On Wed, Oct 3, 2018 at 1:16 PM Toke Høiland-Jørgensen wrote: > > Ben Greear writes: > > > Hello, > > > > I often find myself wanting to figure out what equipment is to blame (and > > why) > > in a wifi environment. > > > > I am thinking writing a tool that would parse a pcap file and look at frames > > in enough detail to flag block-ack bugs, rate-ctrl bugs, guess at the > > sniffer's > > capture ability, etc. > > > > Does anyone have anything already written that they would like to share, or > > know > > of projects that might already do some of this? > > Not sure if this fits your criteria, but Sven's tool to create airtime > charts from packet sniffing data immediately came to mind: > > https://github.com/cloudtrax/airtime-pie-chart I have used that. Oy, it's a PITA. Some of kathie's code over here (example: https://github.com/pollere/pping ) uses the slightly less painful http://libtins.github.io/ library for parsing packets. > > -Toke -- Dave Täht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619
Re: [RFC] mac80211: budget outstanding airtime for transmission
As a side note (good work!) - I would dearly like to visibly account for management frames somewhere that can be seen from userspace. ?
Re: [Make-wifi-fast] [PATCH] ath10k: Re-enable TXQs for all devices
On Thu, Nov 9, 2017 at 4:10 PM, Toke Høiland-Jørgensenwrote: > Rajkumar Manoharan writes: > >>> Commit 4ca1807815aa6801aaced7fdefa9edacc2521767 disables the use of the >>> mac80211 TXQs for some devices because of a theoretical throughput >>> regression. We have not seen this regression for a while now, so it should >>> be >>> safe to re-enable TXQs. >>> >>> Signed-off-by: Toke Høiland-Jørgensen >>> --- >>> This has been in LEDE trunk for a couple of months now with good results. >>> >> Toke, >> >> Good to know that the performance drop is not seen with the chips that does >> not >> have push-pull support. The issue was originally reported with ap152 + >> qca988x >> by community [1]. Hope this combination is also considered in LEDE. > > Ah, was that the original bug report? Thank you, I have not been able to > find that anywhere! > > The issue that seems to point to has been fixed a while ago; I'll send > and updated patch with a better commit message (also forgot to cc the > ath10k list, I see). > > -Toke Hmm. I remember that thread. I thought we'd basically resolved that issue (45% of the time spent in fq_codel_drop under udp flood), back then, with eric adding the batch drop fix to fq_codel itself: See commit: https://osdn.net/projects/android-x86/scm/git/kernel/commits/9d18562a227874289fda8ca5d117d8f503f1dcca which fixed up the problem beautifully: https://lists.bufferbloat.net/pipermail/make-wifi-fast/2016-May/000590.html So if we've been carrying this darn patch for the ath10k vs something that we'd actually fixed elsewhere in the stack, for over a year, sigh. -- Dave Täht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619
Re: [Make-wifi-fast] [PATCH v3] mac80211: Dynamically set CoDel parameters per station
On Thu, Apr 6, 2017 at 8:58 AM, Toke Høiland-Jørgensenwrote: > Eric Dumazet writes: > >> On Thu, 2017-04-06 at 11:38 +0200, Toke Høiland-Jørgensen wrote: >> >>> + >>> +if (thr && thr < STA_SLOW_THRESHOLD * sta->local->num_sta) { >>> +sta->cparams.target = MS2TIME(50); >>> +sta->cparams.interval = MS2TIME(300); >>> +sta->cparams.ecn = false; >>> +} else { >>> +sta->cparams.target = MS2TIME(20); >>> +sta->cparams.interval = MS2TIME(100); >>> +sta->cparams.ecn = true; >>> +} >>> +} >> >> Why ECN is flipped on/off like that ? > > The reasoning is that at really low bandwidths we're better off dropping > the packet and getting potentially latency-sensitive data queued behind > it through (see Dave's various rants with the topic "Packets have > mass"). My general take on wifi is that if you are running at - particularly, stuck at - a low rate (sub 6mbits in the case of this code) - you have so many other problems like retransmits, interference, etc, in the first place, that the presence or absence of codel here is just a small contributor to that noise. We could leave ecn at whatever it is set to here and not flip it on or off. It does seem sane to twiddle the parameters enough to make sure codel doesn't trigger at less than a MTU vs the achieved rate. >> ECN really should be an admin choice. > > Well, the trouble is that the mac80211 queues don't really have an admin > interface currently. So it'll always use ECN (before this change). Should we add a sysfs api to this? >> Also, this change in parameters looks suspect to me, adding a bimodal >> behavior. I would consult Kathleen and Van on this possibility. It's sort of trimodal, actually. I think a more effective approach would be codel's default were the normal 5% of 100ms, bumping it up (as per the above) when we're having bad connectivity and we tried to tackle excessive retransmits harder, and addressed the side impacts of multicast, instead, as much bigger parts of the problem. > Yeah, I agree that it's somewhat of a hack from a theoretical point of > view. I've also been experimenting with Kathy's recommended way of > dealing with bursty MACs (turning off CoDel when there's less than an > MTU's worth of data left), but have not had a lot of success with it. I'm not in a position to resume trying myself. > Guess I can go back and try some variants on that, unless someone else > has better ideas? Just as stuck as you are! > > -Toke > ___ > Make-wifi-fast mailing list > make-wifi-f...@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
Re: Packet throughput (and those iperf data rate) with mac80211/ath9k is 20% worse than net80211/madwifi
On Mon, Jan 30, 2017 at 8:17 AM, Toke Høiland-Jørgensenwrote: > Klaus Kinski writes: > >> Hello all, >> >> this is a blast from the past, but something that still bothers me. >> I have two systems with Atheros/QCA cards: >> >> System A: >> OS and driver: Linux 3.18.36 with last Madwifi/sample code from trunk >> >> WLAN card: AR5413 (Senao EMP-8602 PLUS-S) >> >> System B: >> OS and driver: Linux 3.18.36 with mac80211/minstrel and ath9k from >> backports-4.2 >> >> WLAN card: AR9280 (Compex WLE200NX) >> >> While doing the performance measurements both systems are connected to a >> reference system >> >> with a HF cable, so there should be no outside influences. >> >> Both systems are running in 802.11a mode on channel 40. >> The following table shows 802.11 data packets sent from system A and B >> generated by >> >> iperf in UDP mode over a 2s interval: > > What version of iperf, and configured to which rate? Some versions of > iperf will send its traffic in very large bursts (see > http://burntchrome.blogspot.se/2016/09/iperf3-and-microbursts.html?m=1) > which could cause the queue inside ath9k to overflow (it is only 123 > packets pre-4.10). > > Did you try the latest mac80211/ath9k from 4.10? The queueing structure > changed dramatically, which would impact this, at least if it's a queue > overflow problem... Packet captures would be helpful. Aircaps, if possible, also. > -Toke -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
Re: [PATCH] mac80211: prevent skb/txq mismatch
Yay! This sounds like a potential fix for this? https://bugs.lede-project.org/index.php?do=details_id=368 Are all the ath10k chipsets excluded by commit: 4ca1807815aa6801aaced7fdefa9edacc2521767 Still needed to be excluded?
Re: [PATCH net-next] bridge: multicast to unicast
On Tue, Jan 10, 2017 at 9:23 AM, Felix Fietkau <n...@nbd.name> wrote: > On 2017-01-10 18:17, Dave Taht wrote: >> In the case of wifi I have 3 issues with this line of thought. >> >> multicast in wifi has generally supposed to be unreliable. This makes >> it reliable. reliability comes at a cost - >> >> multicast is typically set at a fixed low rate today. unicast is >> retried at different rates until it succeeds - for every station >> listening. If one station is already at the lowest rate, the total >> cost of the transmit increases, rather than decreases. >> >> unicast gets block acks until it succeeds. Again, more delay. >> >> I think there is something like 31 soft-retries in the ath9k driver > If I remember correctly, hardware retries are counted here as well. I chopped this to something more reasonable but never got around to quantifying it, so never pushed the patch. I figured I'd measure ATF in a noisy environment (which I'd be doing now if it weren't for https://bugs.lede-project.org/index.php?do=details_id=368 ) first. >> what happens to diffserv markings here? for unicast CS1 goes into the >> BE queue, CS6, the VO queue. Do we go from one flat queue for all of >> multicast to punching it through one of the hardware queues based on >> the diffserv mark now with this patch? I meant CS1=BK here. Tracing the path through the bridge code made my head hurt, I can go look at some aircaps to see if the mcast->unicast conversion respects those markings or not (my vote is *not*). >> I would like it if there was a way to preserve the unreliability >> (which multiple mesh protocols depend on), send stuff with QoSNoack, >> etc - or dynamically choose (based on the rates of the stations) >> between conventional multicast and unicast. >> >> Or - better, IMHO, keep sending multicast as is but pick the best of >> the rates available to all the listening stations for it. > The advantage of the multicast-to-unicast conversion goes beyond simply > selecting a better rate - aggregation matters a lot as well, and that is > simply incompatible with normal multicast. Except for the VO queue which cannot aggregate. And for that matter, using any other hardware queue than BE tends to eat a txop that would otherwise possibly be combined with an aggregate. (and the VI queue has always misbehaved, long on my todo list) > Some multicast streams use lots of small-ish packets, the airtime impact > of those is vastly reduced, even if the transmission has to be > duplicated for a few stations. The question was basically how far up does it scale. Arguably, for a very few, well connected stations, this patch would help. For a network with more - and more badly connected stations, I think it would hurt. What sorts of multicast traffic are being observed that flood the network sufficiently to be worth optimizing out? arp? nd? upnp? mdns? uftp? tv? (my questions above are related to basically trying to setup a sane a/b test, I've been building up a new testbed in noisy environment to match the one I have in a quiet one, and don't have any "good" mcast tests defined. Has anyone done an a/b test of this code with some repeatable test already?) (In my observations... The only truly heavy creator of a multicast "burp" has tended to be upnp and mdns on smaller networks. Things like nd and arp get more problematic as the number of stations go up also. I can try things like abusing vlc or uftp to see what happens?) I certainly agree multicast is a "problem" (I've seen 20-80% or more of a given wifi network eaten by multicast) but I'm not convinced that making it reliable, aggregatable unicast scales much past basement-level testing of a few "good" stations, and don't know which protocols are making it worse, the worst, in typical environments. Certainly apple gear puts out a lot of multicast. ... As best as I recall a recommendation in the 802.11-2012 standard was that multicast packets be rate-limited so that you'd have a fixed amount of crap after each beacon sufficient to keep the rest of the unicast traffic flowing rapidly, instead of dumping everything into a given beacon transmit. That, combined with (maybe) picking the "best" union of known rates per station, was essentially the strategy I'd intended[1] to pursue for tackling the currently infinite wifi multicast queue - fq the entries, have a fairly short queue (codel is not the best choice here) drop from head, and limit the number of packets transmitted per beacon to spread them out. That would solve the issue for sparse multicast (dhcp etc), and smooth out the burps from bigger chunks while impacting conventional unicast minimally. There's also the pursuit of less multicast overall at least in some protocols https://tools.ietf.org/html/draft-ietf-dnssd-hybrid-05 > > - Felix [1] but make-wifi-fast has been out of funding since august -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
Re: [PATCH net-next] bridge: multicast to unicast
In the case of wifi I have 3 issues with this line of thought. multicast in wifi has generally supposed to be unreliable. This makes it reliable. reliability comes at a cost - multicast is typically set at a fixed low rate today. unicast is retried at different rates until it succeeds - for every station listening. If one station is already at the lowest rate, the total cost of the transmit increases, rather than decreases. unicast gets block acks until it succeeds. Again, more delay. I think there is something like 31 soft-retries in the ath9k driver what happens to diffserv markings here? for unicast CS1 goes into the BE queue, CS6, the VO queue. Do we go from one flat queue for all of multicast to punching it through one of the hardware queues based on the diffserv mark now with this patch? I would like it if there was a way to preserve the unreliability (which multiple mesh protocols depend on), send stuff with QoSNoack, etc - or dynamically choose (based on the rates of the stations) between conventional multicast and unicast. Or - better, IMHO, keep sending multicast as is but pick the best of the rates available to all the listening stations for it. Has anyone actually looked at the effects of this with, say, 5-10 stations at middlin to poor quality (longer distance)? using something to measure the real effect of the multicast conversion? (uftp, mdns?)
Re: scheduled scan interval
On Mon, Nov 21, 2016 at 7:08 AM, Luca Coelhowrote: > Hi Arend, > > On Mon, 2016-11-21 at 13:03 +0100, Arend Van Spriel wrote: >> On 21-11-2016 12:30, Arend Van Spriel wrote: >> > On 21-11-2016 12:19, Arend Van Spriel wrote: >> > > Hi Johannes, Luca, >> > > >> > > The gscan work made me look at scheduled scan and the implementation of >> > > it in brcmfmac. The driver ignored the interval parameter from >> > > user-space. Now I am fixing that. One thing is that our firmware has a >> > > minimum interval which can not be indicated in struct wiphy. The other >> > > issue is how the maximum interval is used in the nl80211.c. >> > > >> > > In nl80211_parse_sched_scan_plans() it is used against value passed in >> > > NL80211_ATTR_SCHED_SCAN_INTERVAL and NL80211_SCHED_SCAN_PLAN_INTERVAL. >> > > For the first one it caps the value to the maximum, but for the second >> > > one it returns -EINVAL. I suspect this is done because maximum interval >> > > was introduced with schedule scan plans, but it feels inconsistent. >> > >> > It also maybe simply wrong to cap. At least brcmfmac does not set the >> > maximum so it will always get interval being zero. Maybe better to do: >> > >> > if (wiphy->max_sched_scan_plan_interval && >> > request->scan_plans[0].interval > >> > wiphy->max_sched_scan_plan_interval) >> > return -EINVAL; >> > >> > > Thoughts? >> >> Digging deeper. Looking at v4.3 before introduction of sched_scan_plans. >> struct sched_scan_request::interval was specified in milliseconds! Below >> the drivers that I see having scheduled scan support: >> >> iwlmvm: cap interval, convert to seconds. >> ath6kl: cap to 1sec minimum, no max check, convert to seconds. >> wl12xx: no checking in driver, fw need milliseconds. >> wl18xx: no checking in driver, fw need milliseconds. >> >> The milliseconds conversion seems to be taken care of by multiplying >> with MSEC_PER_SEC in wl{12,18}xx drivers. >> >> It seems in 4.8 only iwlmvm set wiphy->max_sched_scan_plan_interval so >> other drivers will get interval of zero which only ath6kl handles. > > With the introduction of scheduled scan plans, we sort of deprecated > the "generic" scheduled scan interval. It doesn't make sense to have > both passed at the same time, so nl80211 forbids > NL80211_ATTR_SCHED_SCAN_INTERVAL if we pass > NL80211_ATTR_SCHED_SCAN_PLANS. > > The original NL80211_ATTR_SCHED_SCAN_INTERVAL was specified in msecs, > which is silly because we can never get millisecond accuracy in this. > Thus, in the plans API, we decided to use seconds instead (because it > makes much more sense). Additionally, the interval is considered > "advisory", because the FW may not be able guarantee the exact > intervals (for instance, the iwlwifi driver actually starts the > interval timer after scan completion, so if you specify 10 seconds > intervals, in practice they'll be 13-14 seconds). > > I'm not sure I'm answering your question, because I'm also not sure I > understood the question. :) I'm not sure if I understand the discussion and hooks myself, but recently fixes for doing channel scans saner from userspace ended up here, after some discussion. https://bugzilla.gnome.org/show_bug.cgi?id=766482 Anything that can reduce the impact of this behavior, I'm for! http://www.taht.net/~d/channel_scans_destroying_latency_under_load_for_10s.png > > -- > Cheers, > Luca. -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
make-wifi-fast linuxplumbers talk summarized on lwn.net
and available here: https://lwn.net/SubscriberLink/705884/1bdb9c4aa048b0d5/ After the talk I discussed with several folk about applying the same debloating techniques to other chipsets. I don't remember, unfortunately, who all those folk were, nor the candidate chipsets! -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
Re: [RFC] mac80211: set wifi_acked[_valid] bits for transmitted SKBs
On Thu, Oct 27, 2016 at 7:21 AM, Johannes Bergwrote: > From: Johannes Berg > > There may be situations in which the in-kernel originator of an > SKB cares about its wifi transmission status. To have that, set > the wifi_acked[_valid] bits before freeing/orphaning the SKB if > the destructor is set. The originator can then use it in there. > > Signed-off-by: Johannes Berg > --- > net/mac80211/status.c | 5 + > 1 file changed, 5 insertions(+) > > diff --git a/net/mac80211/status.c b/net/mac80211/status.c > index ddf71c648cab..dc3132d0effe 100644 > --- a/net/mac80211/status.c > +++ b/net/mac80211/status.c > @@ -541,6 +541,11 @@ static void ieee80211_report_used_skb(struct > ieee80211_local *local, > } else if (info->ack_frame_id) { > ieee80211_report_ack_skb(local, info, acked, dropped); > } > + > + if (!dropped && skb->destructor) { > + skb->wifi_acked_valid = 1; > + skb->wifi_acked = acked; > + } > } One of the things I've been curious about one day trying to take advantage of is the pacing available from sch_fq, in a world where we were trying to take advantage of the chocolatey goodness of the new TCP BBR congestion control algorithm. (sch_fq is apparently required for BBR to work right) By moving the fq_codel algo into the softmac layer as we are doing, we currently expose the "noqueue" interface to the qdisc layer, there, which works great for routers, but for dual use (acting as a NAS host and routing) seems less than ideal. Now it turns out that you can indeed slap the fq qdisc on top of the new wifi intermediate queues code... dave@nemesis:~/slashdot$ tc -s qdisc show dev wlp3s0 qdisc fq 8001: root refcnt 5 limit 1p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms Sent 30828141202 bytes 20530733 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 1127 flows (1127 inactive, 0 throttled) 0 gc, 117 highprio, 714646 throttled but as 1127 inactive flows have been there for a day now, and don't show up in netstat, I guess that somewhere in here we aren't "retiring" a skb in a way the tcp stack understands. root@nemesis:~/slashdot# tc qdisc del dev wlp3s0 root root@nemesis:~/slashdot# tc -s qdisc show dev wlp3s0 qdisc noqueue 0: root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 > > /* > -- > 2.9.3 > -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
Re: Bayesian rate control
On Sun, Oct 23, 2016 at 6:57 AM, Björn Smedmanwrote: > Hi all, > > I've been thinking about rate control a bit lately. I've written up > some of my thoughts in a blog post > (http://www.openias.org/bayesian-wifi-rate-control), but very briefly It is nice to see some newer thinking here. > put I'd like to build a rate control algorithm based on Bayesian > statistical inference, possibly by modeling the rate control problem > as a "multi-armed bandit" problem and/or using Thompson sampling. The paper on minstrel's design was never widely published. I linked to it here: http://blog.cerowrt.org/post/minstrel/ Looking harder at rate control has long been on my todo list, but at the top of my list to finish first has been the fair queuing (fq_codel) and airtime fairness work. https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html#results http://blog.cerowrt.org/post/real_results Once you are statistically hitting more stations, more often, on a more regular basis, with smaller txops, I felt that many things that were perceived as rate control problems would go away, and other things become easier. A basic "fix" to minstrel is to opportunistically sample (which so far as I know, minstrel-blues does), rather than at a fixed rate. btw: I called my early (unpublished) attempt at a "minstrel-2", "bard". :) The now-enormous search space is a big problem in present-day minstrel, followed by excessive retries/latency when sampling, and hidden stations are becoming more and more of a problem as densities go up. (long list of minstrel issues on that first link I posted above). > A couple of questions for the list: > > 1. Is there anybody else out there thinking along similar lines? Yes and no. At the moment I am thinking about the insights from the TCP "BBR" work google just published: (paywalled but at: http://queue.acm.org/app/ ) where they also point to max-plus algebra as being helpful for solving the problems it had. > I'd very much like to find collaborators interested in working on > this. It coruld serve as a pretty nice masters thesis problem, for > example. Please join us over on the make-wifi-fast list. There are more than a few good papers to be had out of it. > > 2. What would be the best hardware/software stack to base this work on? Presently ath9k is the only game in town, and developing/debugging on x86 is the easiest. > I'm thinking the best driver for rate control experimentation would be > ath9k, right? If so then a TP-Link TL-WA901ND router (apparently based > on Qualcomm QCA956x SOC) with OpenWrt, and a TP-Link TL-WDN4800 PCIe > card (apparently based on Atheros AR9380 with PCI ID 168c:0030) for my > desktop sounds like a good combo, no? But would I have to run a custom > kernel on my desktop then (or can I somehow get by with an Ubuntu > standard kernel)? These days I am using a pcengines apu2 as my primary x86 testbed, with ath9k and ath10k cards in it (and one day mt72). The new turris omnia looks like a good platform also. I've been trying to use stuff newer than AR92xx there. Another box I really like is the ubnt uap-lite. Prior to all those, it was the wndr3800, archer c7v2, and nanostation m5s for outdoor work. > > Any other thoughts or pointers are also more than welcome. > > Many thanks, > > Björn Smedman -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
Re: [PATCH v3] ath10k: implement NAPI support
On Fri, Aug 26, 2016 at 4:12 AM, Johannes Berg <johan...@sipsolutions.net> wrote: > On Fri, 2016-08-26 at 03:48 -0700, Dave Taht wrote: >> I'm always rather big on people testing latency under load, and napi >> tends to add some. > > That's a completely useless comment. > > Obviously, everybody uses NAPI; it's necessary for system load and thus > performance, and lets drivers take advantage of TCP merging to reduce > ACKs, which is tremendously helpful (over wifi in particular.) > > Please stop making such drive-by comments that focus only on the single > thing you find important above all; not all people can care only about > that single thing, and unconstructively reiterating it over and over > doesn't help. Well, I apologize for being testy. It is I spent a lot of time testing michal's patchset for the ath10k back in may, and I *will* go and retest ath10k, when these patches land. My principal concern with using napi is at lower rates than the maxes typically reported in a patchset. But it would be nice if people always did test for latency under load when making improvements, before getting to me, and despite having helped make a very comprehensive test suite available (flent) that tests all sorts of things for wifi, getting people to actually use it to see real problems, (in addition to latency under load!) while their fingers are still hot in the codebase, and track/plot their results, remains an ongoing issue across the entire industry. http://blog.cerowrt.org/post/fq_codel_on_ath10k/ There are many other problems in wifi, of course, that could use engineering mental internalization, like airtime fairness, and the mis-behavior of the hardware queues, http://blog.cerowrt.org/post/cs5_lockout/ wifi channel scans http://blog.cerowrt.org/post/disabling_channel_scans/ and so on. I have a ton more datasets and blog entries left to write up from the ath9k work thus far which point to some other issues (minstrel, aggregation, retries) > Thanks, > johannes -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
Re: [PATCH v3] ath10k: implement NAPI support
I'm always rather big on people testing latency under load, and napi tends to add some.
Re: [Make-wifi-fast] [PATCH v2] mac80211: Move crypto IV generation to after TXQ dequeue.
On Wed, Aug 17, 2016 at 9:49 PM, Johannes Bergwrote: > Hi, > > You need to work on coding style, a lot of your indentation is > completely messed up. > >> + switch (sdata->vif.type) { >> + case NL80211_IFTYPE_STATION: >> + if (sdata->u.mgd.use_4addr) { >> + pn_offs = 30; >> + break; >> + } >> + pn_offs = 24; >> + break; >> + case NL80211_IFTYPE_AP_VLAN: >> + if (sdata->wdev.use_4addr) { >> + pn_offs = 30; >> + break; >> + } >> + /* fall through */ >> + case NL80211_IFTYPE_ADHOC: >> + case NL80211_IFTYPE_AP: >> + pn_offs = 24; >> + break; >> + default: >> + return; >> + } >> + >> + if (sta->sta.wme) { >> + pn_offs += 2; >> + } > > I think you just reinvented ieee80211_hdrlen(). No? > >> - if (fast_tx->pn_offs) { >> - u64 pn; >> - u8 *crypto_hdr = skb->data + fast_tx->pn_offs; > > No need to undo the pn_offs optimisation for the !txq case, you can > pass it in to the new function that will fill it. > > However, you're still doing it wrong - now you haven't fixed anything > for TKIP, which won't hit the fastpath. well, we're getting there. the results of both patch attempts were really nice, and brought encrypted performance with fq back into line with unencrypted. Still running crypted tests as I write... So fixing TKIP would be next, forcing the AP to use that? What other scenarios do we have to worry about? WDS? > johannes > ___ > Make-wifi-fast mailing list > make-wifi-f...@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org
Re: [PATCH] ath10k: disable wake_tx_queue for older devices
On Thu, Aug 4, 2016 at 12:07 PM, Roman Yeryomin <leroi.li...@gmail.com> wrote: > On 1 August 2016 at 12:04, Dave Taht <dave.t...@gmail.com> wrote: >> On Mon, Aug 1, 2016 at 1:35 AM, Roman Yeryomin <leroi.li...@gmail.com> wrote: >>> On 7 July 2016 at 19:30, Valo, Kalle <kv...@qca.qualcomm.com> wrote: >>>> Michal Kazior <michal.kaz...@tieto.com> writes: >>>> >>>>> Ideally wake_tx_queue should be used regardless as >>>>> it is a requirement for reducing bufferbloat and >>>>> implementing airtime fairness in the future. >>>>> >>>>> However some setups (typically low-end platforms >>>>> hosting QCA988X) suffer performance regressions >>>>> with the current wake_tx_queue implementation. >>>>> Therefore disable it unless it is really >>>>> beneficial with current codebase (which is when >>>>> firmware supports smart pull-push tx scheduling). >>>>> >>>>> Signed-off-by: Michal Kazior <michal.kaz...@tieto.com> >>>> >>>> I think it's too late to send this to 4.7 anymore (and this due to my >>>> vacation). So I'm planning to queue this to 4.8, but if the feedback is >>>> positive we can always send this to a 4.7 stable release. >>>> >>> >>> Sorry guys, drowned. >>> So, yes, applying this patch does the job. That is gets me to the >>> results similar to >>> https://lists.openwrt.org/pipermail/openwrt-devel/2016-May/041448.html >>> >>> Going to try latest code on same system... >> >> Can you try increasing the quantum to 1514, and reducing the codel >> target to 5ms? (without this patch?) >> > > So it was 1514 already... based on some testing of 20, codel target should be 5ms and isn't. https://github.com/torvalds/linux/commit/5caa328e3811b7cfa33fd02c93280ffa622deb0e > Regards, > Roman > > ___ > ath10k mailing list > ath...@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/ath10k -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] ath10k: disable wake_tx_queue for older devices
On Mon, Aug 1, 2016 at 1:35 AM, Roman Yeryominwrote: > On 7 July 2016 at 19:30, Valo, Kalle wrote: >> Michal Kazior writes: >> >>> Ideally wake_tx_queue should be used regardless as >>> it is a requirement for reducing bufferbloat and >>> implementing airtime fairness in the future. >>> >>> However some setups (typically low-end platforms >>> hosting QCA988X) suffer performance regressions >>> with the current wake_tx_queue implementation. >>> Therefore disable it unless it is really >>> beneficial with current codebase (which is when >>> firmware supports smart pull-push tx scheduling). >>> >>> Signed-off-by: Michal Kazior >> >> I think it's too late to send this to 4.7 anymore (and this due to my >> vacation). So I'm planning to queue this to 4.8, but if the feedback is >> positive we can always send this to a 4.7 stable release. >> > > Sorry guys, drowned. > So, yes, applying this patch does the job. That is gets me to the > results similar to > https://lists.openwrt.org/pipermail/openwrt-devel/2016-May/041448.html > > Going to try latest code on same system... Can you try increasing the quantum to 1514, and reducing the codel target to 5ms? (without this patch?) > > Regards, > Roman > > ___ > ath10k mailing list > ath...@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/ath10k -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP performance regression in mac80211 triggered by the fq code
Just to add another datapoint, the "rack" optimization for tcp entered the kernel recently. It has some "interesting" timing/batching sensitive behaviors. While the TSO case is described, the packet aggregation case seems similar, and is not. https://www.ietf.org/proceedings/96/slides/slides-96-tcpm-3.pdf 10 Jan 2016 https://kernelnewbies.org/Linux_4.4#head-2583c31a65e6592bef9af426a78940078df7f630 The draft was significantly updated this month. https://tools.ietf.org/html/draft-cheng-tcpm-rack-01 -- Andrew Shewmaker On Mon, Jul 18, 2016 at 2:49 PM, Toke Høiland-Jørgensenwrote: > Toke Høiland-Jørgensen writes: > >> Felix Fietkau writes: >> >>> Hi, >>> >>> With Toke's ath9k txq patch I've noticed a pretty nasty performance >>> regression when running local iperf on an AP (running the txq stuff) to >>> a wireless client. >>> >>> Here's some things that I found: >>> - when I use only one TCP stream I get around 90-110 Mbit/s >>> - when running multiple TCP streams, I get only 35-40 Mbit/s total >>> - fairness between TCP streams looks completely fine >>> - there's no big queue buildup, the code never actually drops any packets >>> - if I put a hack in the fq code to force the hash to a constant value >>> (effectively disabling fq without disabling codel), the problem >>> disappears and even multiple streams get proper performance. >>> >>> Please let me know if you have any ideas. >> >> Hmm, I see two TCP streams get about the same aggregate throughput as >> one, both when started from the AP and when started one hop away. > > So while I have still not been able to reproduce the issue you > described, I have seen something else that is at least puzzling, and may > or may not be related: > > When monitoring the output of /sys/kernel/debug/ieee80211/phy0/aqm I see > that all stations have their queues empty all the way to zero several > times per second. This is a bit puzzling; the queue should be kept under > control, but really shouldn't empty completely. I figure this might also > be the reason why you're seeing degraded performance... > > Since the stats output doesn't include a counter for drops, I haven't > gotten any further with figuring out if it's CoDel that's being too > aggressive, or what is happening. But will probably add that in and take > another look. > > -Toke > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP performance regression in mac80211 triggered by the fq code
On Wed, Jul 13, 2016 at 10:53 AM, Felix Fietkauwrote: >> To me this implies a contending lock issue, too much work in the irq >> handler or too delayed work in the softirq handler >> >> I thought you were very brave to try and backport this. > I don't think this has anything to do with contending locks, CPU > utilization, etc. The code does something to the packets that TCP really > doesn't like. With your 70% idle figure, I am inclined to agree... could you get an aircap of the two different tests? - as well as a regular packetcap taken at the client or server? And put somewhere I can get at them? What version of OSX are you running? I will setup an ath9k box shortly... -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP performance regression in mac80211 triggered by the fq code
On Tue, Jul 12, 2016 at 4:02 PM, Dave Taht <dave.t...@gmail.com> wrote: > On Tue, Jul 12, 2016 at 3:21 PM, Felix Fietkau <n...@nbd.name> wrote: >> On 2016-07-12 14:13, Dave Taht wrote: >>> On Tue, Jul 12, 2016 at 12:09 PM, Felix Fietkau <n...@nbd.name> wrote: >>>> Hi, >>>> >>>> With Toke's ath9k txq patch I've noticed a pretty nasty performance >>>> regression when running local iperf on an AP (running the txq stuff) to >>>> a wireless client. >>> >>> Your kernel? cpu architecture? >> QCA9558, 720 MHz, running Linux 4.4.14 So this is a single core at the near-bottom end of the range. I guess we also should find a MIPS 24c derivative that runs at 400Mhz or so. What HZ? (I no longer know how much higher HZ settings make any difference, but I'm usually at NOHZ and 250, rather than 100.) And all the testing to date was on much higher end multi-cores. >>> What happens when going through the AP to a server from the wireless client? >> Will test that next. And? >> >>> Which direction? >> AP->STA, iperf running on the AP. Client is a regular MacBook Pro >> (Broadcom). > > There are always 2 wifi chips in play. Like the Sith. > >>>> Here's some things that I found: >>>> - when I use only one TCP stream I get around 90-110 Mbit/s >>> >>> with how much cpu left over? >> ~20% >> >>>> - when running multiple TCP streams, I get only 35-40 Mbit/s total >>> with how much cpu left over? >> ~30% To me this implies a contending lock issue, too much work in the irq handler or too delayed work in the softirq handler I thought you were very brave to try and backport this. > > Hmm. > > Care to try netperf? > >> >>> context switch difference between the two tests? >> What's the easiest way to track that? > > if you have gnu "time" time -v the_process > > or: > > perf record -e context-switches -ag > > or: process /proc/$PID/status for cntx > >>> tcp_limit_output_bytes is? >> 262144 > > I keep hoping to be able to reduce this to something saner like 4096 > one day. It got bumped to 64k based on bad wifi performance once, and > then to it's current size to make the Xen folk happier. > > The other param I'd like to see fiddled with is tcp_notsent_lowat. > > In both cases reductions will increase your context switches but > reduce memory pressure and lead to a more reactive tcp. > > And in neither case I think this is the real cause of this problem. > > >>> got perf? >> Need to make a new build for that. >> >>>> - fairness between TCP streams looks completely fine >>> >>> A codel will get to long term fairness pretty fast. Packet captures >>> from a fq will show much more regular interleaving of packets, >>> regardless. >>> >>>> - there's no big queue buildup, the code never actually drops any packets >>> >>> A "trick" I have been using to observe codel behavior has been to >>> enable ecn on server and client, then checking in wireshark for ect(3) >>> marked packets. >> I verified this with printk. The same issue already appears if I have >> just the fq patch (with the codel patch reverted). > > OK. A four flow test "should" trigger codel > > Running out of cpu (or hitting some other bottleneck), without > loss/marking "should" result in a tcptrace -G and xplot.org of the > packet capture showing the window continuing to increase > > >>>> - if I put a hack in the fq code to force the hash to a constant value >>> >>> You could also set "flows" to 1 to keep the hash being generated, but >>> not actually use it. >>> >>>> (effectively disabling fq without disabling codel), the problem >>>> disappears and even multiple streams get proper performance. >>> >>> Meaning you get 90-110Mbits ? >> Right. >> >>> Do you have a "before toke" figure for this platform? >> It's quite similar. >> >>>> Please let me know if you have any ideas. >>> >>> I am in berlin, packing hardware... >> Nice! >> >> - Felix >> > > > > -- > Dave Täht > Let's go make home routers and wifi faster! With better software! > http://blog.cerowrt.org -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP performance regression in mac80211 triggered by the fq code
On Tue, Jul 12, 2016 at 3:21 PM, Felix Fietkau <n...@nbd.name> wrote: > On 2016-07-12 14:13, Dave Taht wrote: >> On Tue, Jul 12, 2016 at 12:09 PM, Felix Fietkau <n...@nbd.name> wrote: >>> Hi, >>> >>> With Toke's ath9k txq patch I've noticed a pretty nasty performance >>> regression when running local iperf on an AP (running the txq stuff) to >>> a wireless client. >> >> Your kernel? cpu architecture? > QCA9558, 720 MHz, running Linux 4.4.14 > >> What happens when going through the AP to a server from the wireless client? > Will test that next. > >> Which direction? > AP->STA, iperf running on the AP. Client is a regular MacBook Pro > (Broadcom). There are always 2 wifi chips in play. Like the Sith. >>> Here's some things that I found: >>> - when I use only one TCP stream I get around 90-110 Mbit/s >> >> with how much cpu left over? > ~20% > >>> - when running multiple TCP streams, I get only 35-40 Mbit/s total >> with how much cpu left over? > ~30% Hmm. Care to try netperf? > >> context switch difference between the two tests? > What's the easiest way to track that? if you have gnu "time" time -v the_process or: perf record -e context-switches -ag or: process /proc/$PID/status for cntx >> tcp_limit_output_bytes is? > 262144 I keep hoping to be able to reduce this to something saner like 4096 one day. It got bumped to 64k based on bad wifi performance once, and then to it's current size to make the Xen folk happier. The other param I'd like to see fiddled with is tcp_notsent_lowat. In both cases reductions will increase your context switches but reduce memory pressure and lead to a more reactive tcp. And in neither case I think this is the real cause of this problem. >> got perf? > Need to make a new build for that. > >>> - fairness between TCP streams looks completely fine >> >> A codel will get to long term fairness pretty fast. Packet captures >> from a fq will show much more regular interleaving of packets, >> regardless. >> >>> - there's no big queue buildup, the code never actually drops any packets >> >> A "trick" I have been using to observe codel behavior has been to >> enable ecn on server and client, then checking in wireshark for ect(3) >> marked packets. > I verified this with printk. The same issue already appears if I have > just the fq patch (with the codel patch reverted). OK. A four flow test "should" trigger codel Running out of cpu (or hitting some other bottleneck), without loss/marking "should" result in a tcptrace -G and xplot.org of the packet capture showing the window continuing to increase >>> - if I put a hack in the fq code to force the hash to a constant value >> >> You could also set "flows" to 1 to keep the hash being generated, but >> not actually use it. >> >>> (effectively disabling fq without disabling codel), the problem >>> disappears and even multiple streams get proper performance. >> >> Meaning you get 90-110Mbits ? > Right. > >> Do you have a "before toke" figure for this platform? > It's quite similar. > >>> Please let me know if you have any ideas. >> >> I am in berlin, packing hardware... > Nice! > > - Felix > -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP performance regression in mac80211 triggered by the fq code
On Tue, Jul 12, 2016 at 2:57 PM, Toke Høiland-Jørgensen <t...@toke.dk> wrote: > Dave Taht <dave.t...@gmail.com> writes: > >>> As for why this would happen... There could be a bug in the dequeue code >>> somewhere, but since you get better performance from sticking everything >>> into one queue, my best guess would be that the client is choking on the >>> interleaved packets? I.e. expending more CPU when it can't stick >>> subsequent packets into the same TCP flow? >> >> I share this concern. >> >> The quantum is? I am not opposed to a larger quantum (2 full size >> packets = 3028 in this case?). > > The quantum is hard-coded to 300 bytes in the current implementation > (see net/fq_impl.h). don't do that. :) A single full size packet is preferable, and saves going around the main dequeue loop 5-6 times per flow on this workload. My tests on the prior patch set were mostly at the larger quantum. > -Toke -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP performance regression in mac80211 triggered by the fq code
On Tue, Jul 12, 2016 at 2:28 PM, Toke Høiland-Jørgensenwrote: > Felix Fietkau writes: > >> Hi, >> >> With Toke's ath9k txq patch I've noticed a pretty nasty performance >> regression when running local iperf on an AP (running the txq stuff) to >> a wireless client. >> >> Here's some things that I found: >> - when I use only one TCP stream I get around 90-110 Mbit/s >> - when running multiple TCP streams, I get only 35-40 Mbit/s total >> - fairness between TCP streams looks completely fine >> - there's no big queue buildup, the code never actually drops any packets >> - if I put a hack in the fq code to force the hash to a constant value >> (effectively disabling fq without disabling codel), the problem >> disappears and even multiple streams get proper performance. >> >> Please let me know if you have any ideas. > > Hmm, I see two TCP streams get about the same aggregate throughput as > one, both when started from the AP and when started one hop away. > However, do see TCP flows take a while to ramp up when started from the > AP - a short test gets ~70Mbps when run from one hop away and ~50Mbps > when run from the AP. how long are you running the tests for? > > (I seem to recall the ramp-up issue to be there pre-patch as well, > though). The original ath10k code had a "swag" at hooking in an estimator from rate control. With minstrel in play that can be done better in the ath9k. > As for why this would happen... There could be a bug in the dequeue code > somewhere, but since you get better performance from sticking everything > into one queue, my best guess would be that the client is choking on the > interleaved packets? I.e. expending more CPU when it can't stick > subsequent packets into the same TCP flow? I share this concern. The quantum is? I am not opposed to a larger quantum (2 full size packets = 3028 in this case?). > -Toke > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: TCP performance regression in mac80211 triggered by the fq code
On Tue, Jul 12, 2016 at 12:09 PM, Felix Fietkauwrote: > Hi, > > With Toke's ath9k txq patch I've noticed a pretty nasty performance > regression when running local iperf on an AP (running the txq stuff) to > a wireless client. Your kernel? cpu architecture? What happens when going through the AP to a server from the wireless client? Which direction? > Here's some things that I found: > - when I use only one TCP stream I get around 90-110 Mbit/s with how much cpu left over? > - when running multiple TCP streams, I get only 35-40 Mbit/s total with how much cpu left over? context switch difference between the two tests? tcp_limit_output_bytes is? got perf? > - fairness between TCP streams looks completely fine A codel will get to long term fairness pretty fast. Packet captures from a fq will show much more regular interleaving of packets, regardless. > - there's no big queue buildup, the code never actually drops any packets A "trick" I have been using to observe codel behavior has been to enable ecn on server and client, then checking in wireshark for ect(3) marked packets. > - if I put a hack in the fq code to force the hash to a constant value You could also set "flows" to 1 to keep the hash being generated, but not actually use it. > (effectively disabling fq without disabling codel), the problem > disappears and even multiple streams get proper performance. Meaning you get 90-110Mbits ? Do you have a "before toke" figure for this platform? > Please let me know if you have any ideas. I am in berlin, packing hardware... > > - Felix > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: A question about MAC80211 (3.9)
What are you trying to accomplish? I look forward to seeing any rate control algorithm that can address the issues in minstrel! ( http://blog.cerowrt.org/post/minstrel/ ) It has generally been my hope to implement some form of better service sharing between the VO, VI, and BE queues than what currently exists in linux mainline... after the swq, fq_codel and airtime fairness stuff finishes landing. ( https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html ) VO could only offer service once per 10ms per destination, VI, cap the backlog at no more than 50ms, and in neither case starve BE, BK could often be folded into BE, and so on. We've got bits of an architecture for managing these queues discussed out (more discussion at ietf next week), folding flows into the most appropriate queue rather than treating them as totally separate. (so as to take better advantage of aggregation) Some of the other things in the make-wifi-fast backlog are in the appendix here: https://docs.google.com/document/d/1Se36svYE1Uzpppe1HWnEyat_sAGghB3kE285LElJBW4/edit#heading=h.3ankl68j6jjo On Mon, Jul 11, 2016 at 2:24 PM, Joan Josep Aleixendriwrote: > Hello everybody! > > > I'm a student working on a project about mac80211. I'm trying to modify the > behaviour of the AC's queues but I could really need some help about this > subject. > > > We are implementing a rate control algorithm. To do this we enqueue packets > on ieee80211_tx_frags() function, just after we are sure the packet is going > to be submitted to the driver under the standard code. You can see the code > below > > > The problem is that once we send a packet that was going to the driver to > the pending queue (because our custom logic), the tasklet takes over and > tries to resubmit the packet to tx_frags(). The result is that we start > losing connection and queuing the same packet over and over. Is there a way > to stop the tasklet for some defined period of time? Or maybe another advice > to implement rate limiting at mac80211? > > > Thanks for your time! And i really appreciate your help! > > > Joan Josep Aleixendri Cruelles > > > Code: > > ieee80211_tx_frags(): > > spin_lock_irqsave(>queue_stop_reason_lock, flags); > if (local->queue_stop_reasons[q] || > (!txpending && !skb_queue_empty(>pending[q]))) { > if (unlikely(info->flags & > IEEE80211_TX_INTFL_OFFCHAN_TX_OK)) { > if (local->queue_stop_reasons[q] & > ~BIT(IEEE80211_QUEUE_STOP_REASON_OFFCHANNEL)) { > /* > * Drop off-channel frames if queues > * are stopped for any reason other > * than off-channel operation. Never > * queue them. > */ > spin_unlock_irqrestore( > >queue_stop_reason_lock, > flags); > ieee80211_purge_tx_queue(>hw, > skbs); > return true; > } > } else { > /* > * Since queue is stopped, queue up frames for > * later transmission from the tx-pending > * tasklet when the queue is woken again. > */ > if (txpending) > skb_queue_splice_init(skbs, > >pending[q]); > else > skb_queue_splice_tail_init(skbs, > >pending[q]); > > spin_unlock_irqrestore(>queue_stop_reason_lock, > flags); > return false; > } > } > ++ // CUSTOM CODE > ++ if(packet_meets_requirements_to_be_xmitted()){ > ++ goto xmit; > ++ } > ++ else{ > ++ //Send the packet back to pending > ++ if (txpending) > ++ skb_queue_splice_init(skbs, > ++ >pending[q]); > ++ else > ++ skb_queue_splice_tail_init(skbs, > ++ >pending[q]); > ++ spin_unlock_irqrestore(>queue_stop_reason_lock, > ++ flags); > ++ return false; > ++ } > ++ xmit: > spin_unlock_irqrestore(>queue_stop_reason_lock, flags); > > info->control.vif = vif; > control.sta = sta; > > __skb_unlink(skb, skbs); > drv_tx(local, , skb); > } > > return true; > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To
Re: [ath9k-devel] [PATCH] ath9k: Support 4.9Ghz channels on AR9580 adapter.
On Tue, Jun 21, 2016 at 2:41 AM, Jouni Malinenwrote: > On Tue, Jun 21, 2016 at 11:02:20AM +1000, Julian Calaby wrote: >> I've only done this work as I hate to see people's efforts go to >> waste and I feel that there's enough roadblocks in the way of >> actually using this functionality that casual idiots won't be able >> to. > > Are these really ready to go to the upstream kernel in this state and > without the other changes that would be needed to operate correctly? > What is the use case for these and how have these been tested? So far as I know the use case for these is to make it possible to build open source wifi systems that enable emergency services. This strikes me as a worthy goal. -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [ath9k-devel] [PATCH 1/2] ath9k: use mac80211 intermediate software queues
>> struct ath_atx_tid { >> struct list_head list; >> + struct sk_buff_head i_q; > Do we really need a third queue here? Instead of adding yet another > layer of queueing here, I think we should even get rid of buf_q. Less queues, more filling! > > Channel context based queue handling can be dealt with by > stopping/starting relevant queues on channel context changes. what can be done to reduce the impact of channel scans? http://blog.cerowrt.org/post/disabling_channel_scans/ > buf_q becomes unnecessary when you remove all code in the drv_tx > codepath that moves frames to the intermediate queue. > > Any frame that was pulled from the intermediate queue and prepared for > tx, but which can't be sent right now can simply be queued to retry_q. > > This will also help with getting the diffstat insertion/deletion ratio > under control ;) The ideas here can apply elsewhere, also. Are you still actively working with the mt76? Anything else "out there" besides that and the ath5k worth looking at? Am I seeing patches and firmware changes for better statistic keeping on the ath10k that look promising for airtime fairness... or am I delusional? > elsewhere powersave was mentioned How big can a powersave queue get? -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to use fractional center frequencies (4942.5, for instance)?
On Mon, Jun 6, 2016 at 8:04 AM, Ben Greearwrote: > > It appears that some cisco equipment, at least, uses fractional center > frequencies for 5Mhz channels. I was not aware that anything supported 5Mhz other than ath9k and ath5k. Thx. Can you identify what cisco gear supports this? I have long advocated that meshy networks in increasingly dense areas use narrower channels. https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final142.pdf was one of the better papers that went into it back in the day. > Has anyone attempted to support this with ath9k? > > Thanks, > Ben > > -- > Ben Greear > Candela Technologies Inc http://www.candelatech.com > > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/RFT 2/5] ath9k: use mac80211 intermediate software queues
For the record, michal's lastest patchset for the ath10k is here: https://github.com/kazikcz/linux/tree/fqmac-v5 which includes the reworked codel.h support (which also landed in net-next as of april 22) (no, haven't tried it yet, I'm only a day back from vacation) ... but it would pay to leverage rate control more, for the ath9k, and I'd like folk to agree on a standardized set of statistics in a std location that can be polled for all implementations (ath9k, ath10k, mt76) I am also reviewing this: http://info.iet.unipi.it/~luigi/papers/20160511-mysched-preprint.pdf as we have a chance to innovate and use less locking with all this stuff happening at the mac80211 layer. -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/RFT 2/5] ath9k: use mac80211 intermediate software queues
On Sun, Jun 5, 2016 at 10:50 PM, Tim Shepardwrote: > > >> > Thanks. I've gotten no other feedback that suggests anyone else has >> > read the code. So I very much appreciate it. >> >> I not only read it but tested it extensively against the ath9k stock, >> ath10k stock, ath10k -michal's fqmac 3.5 patches, and your ath9k >> patches... > > My patch to ath9k shouldn't have any effect on any of the ath10k > stuff. It only cuts ath9k over to use the new intermediate queues. Like the Sith, there are always at least two wifi chips on a link. One behaving significantly differently (like flow queuing and codeling and striving for airtime fairness) does affect the behavior of the other. While I did do some ath9k to ath9k testing, I was mostly testing the "whole enchalada". There are many, many moving parts left to swap out. And the patchset I was testing included your fq_codel port for ath9k but it was based on codel5.h. Michal's latest stuff reworked mainline codel to be generically usable, and it *is* a different variant of the algorithm. The airtime-fair stuff does not as yet include fq_codel on top. It's a MPU, and still puzzling as to why it did not get closer to perfect fairness. I also fiddled with the idea of dynamically altering the beacon's txop sizes. A really short best effort txop (94) was "Interesting". I need to take apart how more beacons are constructed from non-linux vendors. The whole enchalada is tasting pretty good thus far. > I should point out again that Avery's observation that michal's > mac80211 flow queueing patches and mac80211 codel stuff aren't needed > to the improvement between competing client stations. To *2* competing client stations, this is somewhat true at present. There are at least 5 other (pending) factors to factor in. * Toke's preliminary airtime-fair patches already showed a net gain in bandwidth for the higher rate competing station. The "performance anomaly", identified way back in 2003, is still with us without also striving for airtime fairness. * In order to hold latencies low with > 2 stations active, I advocate gradually using shorter txops, which will improve behavior for stations doing interactive tasks, and offering service sooner to infrequently seen stations. * In order to do gang scheduling for mu-mimo, we need to have several 2-3ms sized queues outstanding for the devices that can be mu-mimoed. * Getting minstrel-blues in there to sample more dynamically would be nice * Reducing retries and retransmits would be nice when already congested. I'd also like to try QosNoack. > All that's > needed is to use the new mac80211 per-station per-tid intermediate > queues and also set the IFF_NO_QUEUE bit. It's a heckofastart. > > For ath9k, Felix's mac80211 interemediate queues patch (already in > mainline Linux over a year ago), my patch to ath9k, and just > Michal's first patch alone "[PATCHv3 1/5] mac80211: skip netdev > queue control with software queuing" should (and seems to in > testing I've done so far) get all the latency improvement there is > to be had when the competing traffic is to a different client > station. I think it can be shaved from the presently observed 7-12ms minimum at 160mbit by another 2-3x. Also the codel implementation is not as yet as tightly controlling queue size as I'd like - I haven't pushed it as hard at sub 20mbit performance as I'd like (coping with being enraged at networkmanager's over-use of channel scans was what I was at, last) but I'm showing queue depth of well over 25ms at that rate right now on the ath9k patch. > > > >> After losing my temper at the impact of channel scans... >> >> ( https://plus.google.com/u/0/107942175615993706558/posts/WA915Pt4SRN >> ), I got told how to get rid of them for testing, and started redoing >> the work when I got back from vacation. > > Heh... I've never seen that problem. But I'm not running > network-manager on any machine in my testbed. I tend to think it is important to measure what happens in the real world, to clearly identify what the real world problems actually are. I let everybody else, with attenuators, and emulators, do whatever they want. Me, I'm perpetually setting up real-world labs like the yurtlab and sflab and the isclab to try to see what happens in practice. I now plan to put some work in on making channel scans less nasty and to also look into what it takes to implement 802.11k, 802.11r and 802.11v. or at the very least, nag people to get it more right. https://bugzilla.gnome.org/show_bug.cgi?id=766482 NetworkManager's author suggests here that https://blogs.gnome.org/dcbw/2016/05/16/networkmanager-and-wifi-scans/ "You can also advocate that your favorite WiFi interface add support for NetworkManager’s RequestScan()" > >> > I looked for a way to ask mac80211 if there are any packets left in >> > the intermediate queue without dequeueing a packet and I failed to >> > find such an interface. >> >> qdisc->peek like function?
Re: iwlwifi: mvm: add reorder buffer per queue
I can't even describe how much I hate the concept of the reorder buffer in general. Ordering is the endpoints problem. Someday, after we get fq_codeled, short queues again, I'll be able to show why. On Mon, May 16, 2016 at 4:41 AM, Luca Coelhowrote: > On Fri, 2016-05-13 at 11:54 +0300, Dan Carpenter wrote: >> Hello Sara Sharon, >> >> The patch b915c10174fb: "iwlwifi: mvm: add reorder buffer per queue" >> from Mar 23, 2016, leads to the following static checker warnings: >> >> drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c:912 >> iwl_mvm_rx_mpdu_mq() >> error: potential NULL dereference 'sta'. >> >> drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c:912 >> iwl_mvm_rx_mpdu_mq() >> error: we previously assumed 'sta' could be null (see line 796) > > Thanks for the analysis and report, Dan! > > I have queued a fix for this through our internal tree. > > -- > Cheers, > Luca. > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Limit rate of BCM34348 on Raspberry PI-3 via brcmfmac
Ugh. A somewhat dumb question is how would I disable bluetooth entirely on the rpi3? I had done some initial tests on the rpi3 for the fq_codel on wifi work and gave up due to dismal results on the flent tests. I hadn't got around to writing them up here, http://blog.cerowrt.org/tags/wifi/ but perhaps disabling bluetooth would help. On Mon, May 9, 2016 at 9:59 AM, Barry Reinholdwrote: > Arend (and Hante), > > I appreciate the feedback on the core problem - that brings the issue into a > lot sharper focus. Based on your response I assume we should be able to see > successful coexistence with Bluetooth and 802.11 STA mode, as well as STA and > AP (with Bluetooth turned off). However, BT with AP mode should fail, and BT > with AP and STA will fail. > > I will rerun some of our crude tests to see if I can corroborate your > understanding by testing the different "should coexist" and "should not > coexist" cases. > > You have a very interesting lead in statement in your response (bottom of > message) but the sentence just ends with "because of...". Would you be > willing to complete that thought, I would like to further understand the > nature of the problem. > > Is there a known reason why the brcmfmac does not support the > set_bitrate_mask() callback (such as the associated family of chips do not > support rate limiting) or is this something that nobody has cared about to do > date, ie, is it something that could be done if there was interest and > resources? > > > From: Arend Van Spriel > Sent: Monday, May 9, 2016 8:23 > To: Barry Reinhold; linux-wireless@vger.kernel.org > Cc: Tom Harada; brcm80211-dev-list > Subject: Re: Limit rate of BCM34348 on Raspberry PI-3 via brcmfmac > > On 7-5-2016 21:24, Barry Reinhold wrote: >> I have observed erratic behavior with http connectivity over the WiFi >> interface of the Raspberry PI 3. This appears to be consistent with issues >> that a number of other people have reported. I fear, but can not provide >> definitive evidence, that these failures could be an RF design/layout issue >> with the RP-3 itself. >> >> The purpose of this post is to see if this possible issue can be confirmed >> by others, and to seek a possible work around by re configuring the BCM43438 >> chip via the brcmfmac driver; or the other associated wifi modules. >> >> How the issue is being seen: >> >> Note: The testing I have done is limited and has the potential to be >> misleading, so any input on improving the test process would be appreciated. >> >> There are two metrics we are using to define/see failure: (1) Loss/delay in >> ICMP Echo requests/replys (pings), and (2) The output of messages in >> journalctl from the wpa_supplicant or hostapd (sudo journalctl -u >> wpa_supplicant -u hostapd -f) indicating a disconnect event with associated >> reason - typically 0 - (wlan0: CTRL-EVENT-DISCONNECTED >> bssid=60:02:92:cd:c9:30 reason=0). >> Ping times vary from 1 to several hundred ms, to outright loss. >> There are also failures to reassociate (wlan0: CTRL-EVENT-ASSOC-REJECT >> status_code=16). >> >> The test Environment is composed of: >> An official Raspberry PI-3 model B with an official Raspberry PI-3 power >> supply. >> Raspbian release: Jesse (March 18) >> Kernel 4.4.6 >> wpa_supplicant 2.3 >> brcmfmac 7.45.41.23 (as reported by ethool) >> BCM43438 firmware: 01-cc44eda9c >> BlueZ 5.23 >> >> We are running both wpa_supplicant and hostapd, (disabling hostapd does not >> impact the results of the tests). >> We have an application that is monitoring for BTLE/Bluetooth connections so >> it is scanning on a regular basis, as well as sending out Bluetooth INQUIRE >> messages. >> >> WiFi Access Points: >> 1. Cisco DPC3939B (supports n) >> 2. Cisco Linksys E1200 (supports n) >> 3. Netgear WNDR3400 (supports n) >> 4. Linksys WAP54G v3 (does not support n) >> >> >> >> Test Process >> >> >> While the application is running (thus generating Bluetooth activity) >> 1. Connect a PC to the RPi3's software access point and ping the RPi3 >> continuously. >> 2. Connect the RPi3 to an AP from the set above. >> 3. Let the system run for 10 minutes while counting wpa_supplicant >> disconnects and lost pings. >> >> Observations: >> In our testing we noticed that either we essentially got no errors, or we >> got 12+ errors. Some error rates high enough that we couldn't count them as >> they just scrolled off our screen. Hence we considered thing to work (less >> then 2 errors) or failed (greater than 10 errors). >> >> The results table for the different APs is as follows: >> DPC3939B - Failed >> E1200 - Failed >> WNDR3400 - Failed >> WAP54G - Passed >> >> Since only the WAP54G passed (no n support), we modified the data rate on >> the Netgear WND3400 and limited its data rate to 54 mbs, at this point the >> WNDR3400 passed. >> >> We then tried changing channels. this
Re: [PATCHv4 5/5] mac80211: add debug knobs for codel
On Thu, May 5, 2016 at 11:33 PM, Michal Kazior <michal.kaz...@tieto.com> wrote: > On 6 May 2016 at 07:51, Dave Taht <dave.t...@gmail.com> wrote: >> On Thu, May 5, 2016 at 10:27 PM, Michal Kazior <michal.kaz...@tieto.com> >> wrote: >>> On 5 May 2016 at 17:21, Dave Taht <dave.t...@gmail.com> wrote: >>>> On Thu, May 5, 2016 at 4:00 AM, Michal Kazior <michal.kaz...@tieto.com> >>>> wrote: >>>>> This adds a few debugfs entries to make it easier >>>>> to test, debug and experiment. >>>> >>>> I might argue in favor of moving all these (inc the fq ones) into >>>> their own dir, maybe "aqm" or "sqm". >>>> >>>> The mixture of read only stats and configuration vars is a bit confusing. >>>> >>>> Also in my testing of the previous patch, actually seeing the stats >>>> get updated seemed to be highly async or inaccurate. For example, it >>>> was obvious from the captures themselves that codel_ce_mark-ing was >>>> happening, but the actual numbers out of wack with the mark seen or >>>> fq_backlog seen. (I can go back to revisit this) >>> >>> That's kind of expected since all of these bits are exposed as >>> separate debugfs entries/files. To avoid that it'd be necessary to >>> provide a single debugfs entry/file whose contents are generated on >>> open() while holding local->fq.lock. But then you could argue it >>> should contain all per-sta-tid info as well (backlog, flows, drops) as >>> well instead of having them in netdev*/stations/*/txqs. >>> Hmm.. >> >> I have not had time to write up todays results to any full extent, but >> they were pretty spectacular. >> >> I have a comparison of the baseline ath10k driver vs your 3.5 patchset >> here on the second plot: >> >> http://blog.cerowrt.org/post/predictive_codeling/ >> >> The raw data is here: >> https://github.com/dtaht/blog-cerowrt/tree/master/content/flent/qca-10.2-fqmac35-codel-5 > > It's probably good to explicitly mention that you test(ed) ath10k with > my RFC DQL patch applied. Without it the fqcodel benefits are a lot > less significant. Yes. I am trying to establish a baseline before and after, starting at the max rate my ath9k (2x2) can take the ath10k (2x2) at a distance of about 12 feet. Without moving anything. https://github.com/dtaht/blog-cerowrt/tree/master/content/flent/stock-4.4.1-22 has the baseline stats from that ubuntu 16.04 kernel... but the comparison plots I'd generated there were against the ct-10.1 firmware and before I'd realized you'd used the smaller quantum. Life is *even* better with using the bigger quantum in the qca-10.2-fqmac35-codel-5 patchset. > > (oh, and the "3.5" is pre-PATCHv4 before fq/codel split work: > https://github.com/kazikcz/linux/tree/fqmac-v3.5 ) I have insufficient time in life to track any but the most advanced patchset, and I am catching up as fast as I can. First up was finding the max ath9k performance, (5x reduction in latency, no reduction in throughput at about 110mbit). Then I'll try locking the bitrate at say 24mbit for another run. You already showed the latency reduction at 6mbit at about 100x to 1, so I don't plan to repeat that. then I'll get another ath10k 3x3 up and wash, rinse, repeat. I would not mind if your patch 4.1 had good stats generation (maybe put all the relevant stats in a single file?) and defaulted to quantum 1514, since it seems likely I'll not get done this first test run before monday. Additional test suggestions wanted? I plan to add the tcp_square_wave tests to the next run to show how much better the congestion control is, and I'll add iperf3 floods too. I am not sure how avery is planning to test each individual piece. > >> >> ... >> >> a note: quantum of the mtu (typically 1514) is a saner default than 300, >> >> (the older patch I had, set it to 300, dunno what your default is now). > > I still use 300. > > >> and quantum 1514, codel target 5ms rather than 20ms for this test >> series was *just fine* (but more testing of the lower target is >> needed) > > I would keep 20ms for now until we get more test data. I'm mostly > concerned about MU performance on ath10k which requires significant > amount of buffering. ok. > >> However: >> >> quantum "300" only makes sense for very, very low bandwidths (say < >> 6mbits), in other scenarios it just eats extra cpu (5 passes through >> the loop to send a big packet) and disables >> the "new/old" queue feature which helps "push" new flows to flo
Re: [PATCHv4 5/5] mac80211: add debug knobs for codel
On Thu, May 5, 2016 at 10:27 PM, Michal Kazior <michal.kaz...@tieto.com> wrote: > On 5 May 2016 at 17:21, Dave Taht <dave.t...@gmail.com> wrote: >> On Thu, May 5, 2016 at 4:00 AM, Michal Kazior <michal.kaz...@tieto.com> >> wrote: >>> This adds a few debugfs entries to make it easier >>> to test, debug and experiment. >> >> I might argue in favor of moving all these (inc the fq ones) into >> their own dir, maybe "aqm" or "sqm". >> >> The mixture of read only stats and configuration vars is a bit confusing. >> >> Also in my testing of the previous patch, actually seeing the stats >> get updated seemed to be highly async or inaccurate. For example, it >> was obvious from the captures themselves that codel_ce_mark-ing was >> happening, but the actual numbers out of wack with the mark seen or >> fq_backlog seen. (I can go back to revisit this) > > That's kind of expected since all of these bits are exposed as > separate debugfs entries/files. To avoid that it'd be necessary to > provide a single debugfs entry/file whose contents are generated on > open() while holding local->fq.lock. But then you could argue it > should contain all per-sta-tid info as well (backlog, flows, drops) as > well instead of having them in netdev*/stations/*/txqs. > Hmm.. I have not had time to write up todays results to any full extent, but they were pretty spectacular. I have a comparison of the baseline ath10k driver vs your 3.5 patchset here on the second plot: http://blog.cerowrt.org/post/predictive_codeling/ The raw data is here: https://github.com/dtaht/blog-cerowrt/tree/master/content/flent/qca-10.2-fqmac35-codel-5 ... a note: quantum of the mtu (typically 1514) is a saner default than 300, (the older patch I had, set it to 300, dunno what your default is now). and quantum 1514, codel target 5ms rather than 20ms for this test series was *just fine* (but more testing of the lower target is needed) However: quantum "300" only makes sense for very, very low bandwidths (say < 6mbits), in other scenarios it just eats extra cpu (5 passes through the loop to send a big packet) and disables the "new/old" queue feature which helps "push" new flows to flow balance. I'd default it to the larger value. ... In other news, spacex just landed on the barge a few minutes ago. The webcast is still going on https://www.youtube.com/watch?v=L0bMeDj76ig and you can reverse it to the landing. :awesome: > > > Michał -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv4 4/5] mac80211: implement codel on fair queuing flows
On Thu, May 5, 2016 at 4:00 AM, Michal Kaziorwrote: > There is no other limit other than a global > packet count limit when using software queuing. > This means a single flow queue can grow insanely > long. This is particularly bad for TCP congestion > algorithms which requires a little more > sophisticated frame dropping scheme than a mere > headdrop on limit overflow. > > Hence apply (a slighly modified, to fit the knobs) > CoDel5 on flow queues. This improves TCP > convergence and stability when combined with > wireless driver which keeps its own tx queue/fifo > at a minimum fill level for given link conditions. > > Signed-off-by: Michal Kazior > --- > > Notes: > v4: > * removed internal codel.h and re-used in-kernel one > > include/net/mac80211.h | 14 +- > net/mac80211/ieee80211_i.h | 5 +++ > net/mac80211/tx.c | 109 > - > 3 files changed, 126 insertions(+), 2 deletions(-) > > diff --git a/include/net/mac80211.h b/include/net/mac80211.h > index ffb90dfe0d70..cc534f1b0f8e 100644 > --- a/include/net/mac80211.h > +++ b/include/net/mac80211.h > @@ -21,6 +21,7 @@ > #include > #include > #include > +#include > #include > > /** > @@ -895,7 +896,18 @@ struct ieee80211_tx_info { > unsigned long jiffies; > }; > /* NB: vif can be NULL for injected frames */ > - struct ieee80211_vif *vif; > + union { > + /* NB: vif can be NULL for injected frames */ > + struct ieee80211_vif *vif; > + > + /* When packets are enqueued on txq it's easy > +* to re-construct the vif pointer. There's no > +* more space in tx_info so it can be used to > +* store the necessary enqueue time for packet > +* sojourn time computation. > +*/ > + codel_time_t enqueue_time; > + }; Can't the skb->timestamp be used instead? (or does that still stomp on tcp) (my longstanding dream of course has been to always timestamp coming off the rx ring, and to not have to do it on entrance to the codel enqueue routine here. It adds measuring total system processing time to the queue measurement, allows for offloaded timestamping, etc, but did involve changing all of linux to use it) > struct ieee80211_key_conf *hw_key; > u32 flags; > /* 4 bytes free */ > diff --git a/net/mac80211/ieee80211_i.h b/net/mac80211/ieee80211_i.h > index 6f8375f1df88..54edfb6fc1d1 100644 > --- a/net/mac80211/ieee80211_i.h > +++ b/net/mac80211/ieee80211_i.h > @@ -812,10 +812,12 @@ enum txq_info_flags { > * @tin: contains packets split into multiple flows > * @def_flow: used as a fallback flow when a packet destined to @tin hashes > to > * a fq_flow which is already owned by a different tin > + * @def_cvars: codel vars for @def_flow > */ > struct txq_info { > struct fq_tin tin; > struct fq_flow def_flow; > + struct codel_vars def_cvars; > unsigned long flags; > > /* keep last! */ > @@ -1108,6 +1110,9 @@ struct ieee80211_local { > struct ieee80211_hw hw; > > struct fq fq; > + struct codel_vars *cvars; > + struct codel_params cparams; > + struct codel_stats cstats; > > const struct ieee80211_ops *ops; > > diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c > index 47936b939591..013b382f6888 100644 > --- a/net/mac80211/tx.c > +++ b/net/mac80211/tx.c > @@ -25,6 +25,8 @@ > #include > #include > #include > +#include > +#include > #include > #include > > @@ -1269,11 +1271,92 @@ static struct txq_info *ieee80211_get_txq(struct > ieee80211_local *local, > return NULL; > } > > +static void ieee80211_set_skb_enqueue_time(struct sk_buff *skb) > +{ > + IEEE80211_SKB_CB(skb)->control.enqueue_time = codel_get_time(); > +} > + > +static void ieee80211_set_skb_vif(struct sk_buff *skb, struct txq_info *txqi) > +{ > + IEEE80211_SKB_CB(skb)->control.vif = txqi->txq.vif; > +} > + > +static u32 codel_skb_len_func(const struct sk_buff *skb) > +{ > + return skb->len; > +} > + > +static codel_time_t codel_skb_time_func(const struct sk_buff *skb) > +{ > + const struct ieee80211_tx_info *info; > + > + info = (const struct ieee80211_tx_info *)skb->cb; > + return info->control.enqueue_time; > +} > + > +static struct sk_buff *codel_dequeue_func(struct codel_vars *cvars, > + void *ctx) > +{ > + struct ieee80211_local *local; > + struct txq_info *txqi; > + struct fq *fq; > +
Re: [PATCHv4 5/5] mac80211: add debug knobs for codel
On Thu, May 5, 2016 at 4:00 AM, Michal Kaziorwrote: > This adds a few debugfs entries to make it easier > to test, debug and experiment. I might argue in favor of moving all these (inc the fq ones) into their own dir, maybe "aqm" or "sqm". The mixture of read only stats and configuration vars is a bit confusing. Also in my testing of the previous patch, actually seeing the stats get updated seemed to be highly async or inaccurate. For example, it was obvious from the captures themselves that codel_ce_mark-ing was happening, but the actual numbers out of wack with the mark seen or fq_backlog seen. (I can go back to revisit this) > > Signed-off-by: Michal Kazior > --- > > Notes: > v4: > * stats adjustments (in-kernel codel has more of them) > > net/mac80211/debugfs.c | 40 > 1 file changed, 40 insertions(+) > > diff --git a/net/mac80211/debugfs.c b/net/mac80211/debugfs.c > index 43592b6f79f0..c7cfedc61fc4 100644 > --- a/net/mac80211/debugfs.c > +++ b/net/mac80211/debugfs.c > @@ -124,6 +124,15 @@ static const struct file_operations name## _ops = { > \ > res;\ > }) > > +#define DEBUGFS_RW_BOOL(arg) \ > +({ \ > + int res;\ > + int val;\ > + res = mac80211_parse_buffer(userbuf, count, ppos, "%d", ); \ > + arg = !!(val); \ > + res;\ > +}) > + > DEBUGFS_READONLY_FILE(fq_flows_cnt, "%u", > local->fq.flows_cnt); > DEBUGFS_READONLY_FILE(fq_backlog, "%u", > @@ -132,6 +141,16 @@ DEBUGFS_READONLY_FILE(fq_overlimit, "%u", > local->fq.overlimit); > DEBUGFS_READONLY_FILE(fq_collisions, "%u", > local->fq.collisions); > +DEBUGFS_READONLY_FILE(codel_maxpacket, "%u", > + local->cstats.maxpacket); > +DEBUGFS_READONLY_FILE(codel_drop_count, "%u", > + local->cstats.drop_count); > +DEBUGFS_READONLY_FILE(codel_drop_len, "%u", > + local->cstats.drop_len); > +DEBUGFS_READONLY_FILE(codel_ecn_mark, "%u", > + local->cstats.ecn_mark); > +DEBUGFS_READONLY_FILE(codel_ce_mark, "%u", > + local->cstats.ce_mark); > > DEBUGFS_RW_FILE(fq_limit, > DEBUGFS_RW_EXPR_FQ("%u", >fq.limit), > @@ -139,6 +158,18 @@ DEBUGFS_RW_FILE(fq_limit, > DEBUGFS_RW_FILE(fq_quantum, > DEBUGFS_RW_EXPR_FQ("%u", >fq.quantum), > "%u", local->fq.quantum); > +DEBUGFS_RW_FILE(codel_interval, > + DEBUGFS_RW_EXPR_FQ("%u", >cparams.interval), > + "%u", local->cparams.interval); > +DEBUGFS_RW_FILE(codel_target, > + DEBUGFS_RW_EXPR_FQ("%u", >cparams.target), > + "%u", local->cparams.target); > +DEBUGFS_RW_FILE(codel_mtu, > + DEBUGFS_RW_EXPR_FQ("%u", >cparams.mtu), > + "%u", local->cparams.mtu); > +DEBUGFS_RW_FILE(codel_ecn, > + DEBUGFS_RW_BOOL(local->cparams.ecn), > + "%d", local->cparams.ecn ? 1 : 0); > > #ifdef CONFIG_PM > static ssize_t reset_write(struct file *file, const char __user *user_buf, > @@ -333,6 +364,15 @@ void debugfs_hw_add(struct ieee80211_local *local) > DEBUGFS_ADD(fq_collisions); > DEBUGFS_ADD(fq_limit); > DEBUGFS_ADD(fq_quantum); > + DEBUGFS_ADD(codel_maxpacket); > + DEBUGFS_ADD(codel_drop_count); > + DEBUGFS_ADD(codel_drop_len); > + DEBUGFS_ADD(codel_ecn_mark); > + DEBUGFS_ADD(codel_ce_mark); > + DEBUGFS_ADD(codel_interval); > + DEBUGFS_ADD(codel_target); > + DEBUGFS_ADD(codel_mtu); > + DEBUGFS_ADD(codel_ecn); > > statsd = debugfs_create_dir("statistics", phyd); > > -- > 2.1.4 > -- Dave Täht Let's go make home routers and wifi faster! With better software! http://blog.cerowrt.org -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: General VHT rate-ctrl question
On Wed, Apr 13, 2016 at 6:18 AM, Ben Greearwrote: > > > On 04/13/2016 01:01 AM, Johannes Berg wrote: >> >> On Tue, 2016-04-12 at 16:48 -0700, Ben Greear wrote: >>> >>> If a station and it's peer can both do VHT, is there ever a good >>> reason to even try HT rates? >>> >> >> Not really; perhaps if you could do HT greenfield preamble (which VHT >> doesn't have) you could get something out of it, beyond that I don't >> see a reason to try. >> >> Unless, for some strange reason, it supports only single stream VHT and >> dual-stream HT or something really weird? > > > I was wondering if there was ever a reason that, say 450Mbps HT > would work better than MCS-1 for VHT. Or, maybe a mid-rate HT MCS would > have more range than VHT, or something like that. > > After fighting with the firmware's rate-ctrl all day, I am even more > interested > in trying to make it use mistrel_ht. I just put up Andrew's old paper on minstrel, if that helps any. http://blog.cerowrt.org/post/minstrel/ > > Thanks, > Ben > > -- > Ben Greear > Candela Technologies Inc http://www.candelatech.com > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
ietf proposed standards for dscp to 802.11e mappings
If anyone cares https://datatracker.ietf.org/doc/draft-szigeti-tsvwg-ieee-802-11/?include_text=1 these will be discussed on the tsvwg mailing list. https://www.ietf.org/mailman/listinfo/tsvwg There are several things in here I object to - no discussion of multicast, and arbitrarily dropping CS6 and CS7 on APs, but I figure that will be sorted out eventually. -- Dave Täht -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
On Wed, Apr 6, 2016 at 12:21 AM, Johannes Bergwrote: > [removing other lists since they spam me with moderation bounces] I have added your email address be accepted to the codel, make-wifi-fast lists. My apologies for the bounces. The people on those lists generally do not have the time to tackle the volume of traffic on linux-wireless. >> The hope had been the original codel.h would have been reusable, >> which is not the case at present. > > So what's the strategy for making it happen? Strategy? to meander towards a result that gives low latency to all stations, no matter their bandwidth, on several chipsets. The holy grail from my viewpoint is to get airtime fairness, better mac utilization, slow stations not starving fast ones, more stations servicable, and so on, and my focus has generally been on having an architecture that applied equally to APs and clients. Getting clients alone to have a queuing latency reduction of these orders of magnitude on uploads at low rates would be a huge win, but not the holy grail. It was really nice to have michal's proof of concept(s) show up and show fq_codel-like benefits at both low and high speeds on wifi, but it is clear more architectural rework is required to fit the theory into the reality. > Unless there is one, I > don't see the point in making the code more complicated than it already > has to be anyway. +1. Next steps were - get toke's and my testbeds up - avery/tim/myself to keep hammering at the ath9k - michal exploring dql - jonathon poking at it with cake-like ideas - and anyone else that cares to join in on finally fixing bufferbloat on wifi. and maybe put together a videoconference in 2-3 weeks or so with where we are stuck at (felix will be off vacation, too, I think). There are still multiple points where we all talk past each other. Me, for example, am overly fixated on having a per station queue to start with (which in the case of a client is two stations - one multicast/mgtmt frames and regular traffic) and not dealing with tids until much later in the process. Unfortunately it seems the hook is very late in the process. > > johannes -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
thx for the review! On Tue, Apr 5, 2016 at 6:57 AM, Johannes Bergwrote: > On Thu, 2016-03-31 at 12:28 +0200, Michal Kazior wrote: > >> +++ b/net/mac80211/codel.h >> +++ b/net/mac80211/codel_i.h > > Do we really need all this code in .h files? It seems very odd to me to > have all the algorithm implementation there rather than a C file, you > should (can?) only include codel.h into a single C file anyway. The hope had been the original codel.h would have been reusable, which is not the case at present. > >> struct txq_info { >> - struct sk_buff_head queue; >> + struct txq_flow flow; >> + struct list_head new_flows; >> + struct list_head old_flows; > > This is confusing, can you please document that? Why are there two > lists of flows, *and* an embedded flow? Is the embedded flow on any of > the lists? To explain the new and old flow concepts, there's https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06 which is in the ietf editors queue for final publication and doesn't have a final name yet. The embedded flow concept is michal's and I'm not convinced it's the right idea as yet. > >> + u32 backlog_bytes; >> + u32 backlog_packets; >> + u32 drop_codel; > > Would it make some sense to at least conceptually layer this a bit? > I.e. rather than calling this "drop_codel" call it "drop_congestion" or > something like that? Is there a more generic place overall in ieee80211 to record per-sta backlogs, drops and marks? >> + skb = codel_dequeue(flow, >> + >backlog, >> + 0, >> + >cvars, >> + >cparams, >> + codel_get_time(), >> + false); > > What happened here? :) Magic. > >> + if (!skb) { >> + if ((head == >new_flows) && >> + !list_empty(>old_flows)) { >> + list_move_tail(>flowchain, >old_flows); >> + } else { >> + list_del_init(>flowchain); >> + flow->txqi = NULL; >> + } >> + goto begin; >> + } > > Ouch. Any way you can make that easier to follow? It made my brain hurt in the original code, too, but it is eric optimizing out cycles at his finest. if the the new_flows list is expired or done, switch to the old_flows list, if the old_flows list is done, go try selecting another queue to pull from (which may or may not exist). see the pending rfc for a more elongated version. > > johannes -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] ath10k: implement dql for htt tx
As a side note of wifi ideas complementary to codel, please see: http://blog.cerowrt.org/post/selective_unprotect/ On Tue, Mar 29, 2016 at 12:49 AM, Michal Kazior <michal.kaz...@tieto.com> wrote: > On 26 March 2016 at 17:44, Dave Taht <dave.t...@gmail.com> wrote: >> Dear Michal: > [...] >> I am running behind on this patch set, but a couple quick comments. > [...] >>> - no rrul tests, sorry Dave! :) >> >> rrul would be a good baseline to have, but no need to waste your time >> on running it every time as yet. It stresses out both sides of the >> link so whenever you get two devices with these driver changes on them >> it would be "interesting". It's the meanest, nastiest test we have... >> if you can get past the rrul, you've truly won. >> >> Consistently using tcp_fair_up with 1,2,4 flows and 1-4 stations as >> you are now is good enough. >> >> doing a more voip-like test with slamming d-itg into your test would be >> good... >> >>> >>> Observations / conclusions: >>> - DQL builds up throughput slowly on "veryfast"; in some tests it >>> doesn't get to reach peak (roughly 210mbps average) because the test >>> is too short >> >> It looks like having access to the rate control info here for the >> initial and ongoing estimates will react faster and better than dql >> can. I loved the potential here in getting full rate for web traffic >> in the usual 2second burst you get it in (see above blog entries) > > On one hand - yes, rate control should in theory be "faster". > > On the other hand DQL will react also to host system interrupt service > time. On slow CPUs (typically found on routers and such) you might end > up grinding the CPU so much you need deeper tx queues to keep the hw > busy (and therefore keep performance maxed). DQL should automatically > adjust to that while "txop limit" might not. Mmmm current multi-core generation arm routers should be fast enough. Otherwise, point taken (possibly). Even intel i3 boxes need offloads to get to line rate. >> >> It is always good to test codel and fq_codel separately, particularly >> on a new codel implementation. There are so many ways to get codel >> wrong or add an optimization that doesn't work (speaking as someone >> that has got it wrong often) >> >> If you are getting a fq result of 12 ms, that means you are getting >> data into the device with a ~12ms standing queue there. On a good day >> you'd see perhaps 17-22ms for "codel target 5ms" in that case, on the >> rtt_fair_up series of tests. > > This will obviously depend on the number of stations you have data > queued to. Estimating codel target time requires smarter tx > scheduling. My earlier (RFC) patch tried doing that. and I loved it. ;) > >> if you are getting a pure codel result of 160ms, that means the >> implementation is broken. But I think (after having read your >> description twice), the baseline result today of 160ms of queuing was >> with a fq_codel *qdisc* doing the work on top of huge buffers, > > Yes. The 160ms is with fq_codel qdisc with ath10k doing DQL at 6mbps. > Without DQL ath10k would clog up all tx slots (1424 of them) with > frames. At 6mbps you typically want/need a handful (5-10) of frames to > be queued. > >> the >> results a few days ago were with a fq_codel 802.11 layer, and the >> results today you are comparing, are pure fq (no codel) in the 802.11e >> stack, with fixed (and dql) buffering? > > Yes. codel target in fq_codel-in-mac80211 is hardcoded at 20ms now > because there's no scheduling and hence no data to derive the target > dynamically. Well, for these simple 2 station tests, you could halve it, easily. With ecn on on both sides, I tend to look at the groupings of the ecn marks in wireshark. > > >> if so. Yea! Science! >> >> ... >> >> One of the flaws of the flent tests is that conceptually they were >> developed before the fq stuff won so big, and looking hard at the >> per-queue latency for the fat flows requires either looking hard at >> the packet captures or sampling the actual queue length. There is that >> sampling capability in various flent tests, but at the moment it only >> samples what tc provides (Drops, marks, and length) and it does not >> look like there is a snapshot queue length exported from that ath10k >> driver? > > Exporting tx queue length snapshot should be fairly easy. 2 debugfs > entries for ar->htt.max_num_pending_tx and ar->htt.num_pending_tx. K. Still running *way* behind you on getting stuff up and
Re: Bonjour mDNS broacast can be lost during BT-WLAN coexistence schemes?
These folk claim an open source prototype. http://www.sigcomm.org/sites/default/files/ccr/papers/2014/January/2567561-2567567.pdf -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] ath10k: implement dql for htt tx
Dear Michal: I commented on and put up your results for the baseline driver here: http://blog.cerowrt.org/post/rtt_fair_on_wifi/ And the wonderful result you got for the first ever fq_codel-ish implementation here: http://blog.cerowrt.org/post/fq_codel_on_ath10k/ I am running behind on this patch set, but a couple quick comments. On Fri, Mar 25, 2016 at 2:55 AM, Michal Kaziorwrote: > On 25 March 2016 at 10:39, Michal Kazior wrote: >> This implements a very naive dynamic queue limits >> on the flat HTT Tx. In some of my tests (using >> flent) it seems to reduce induced latency by >> orders of magnitude (e.g. when enforcing 6mbps >> tx rates 2500ms -> 150ms). But at the same time it >> introduces TCP throughput buildup over time >> (instead of immediate bump to max). More >> importantly I didn't observe it to make things >> much worse (yet). >> >> Signed-off-by: Michal Kazior >> --- >> >> I'm not sure yet if it's worth to consider this >> patch for merging per se. My motivation was to >> have something to prove mac80211 fq works and to >> see if DQL can learn the proper queue limit in >> face of wireless rate control at all. >> >> I'll do a follow up post with flent test results >> and some notes. > > Here's a short description what-is-what test naming: > - sw/fq contains only txq/flow stuff (no scheduling, no txop queue limits) > - sw/ath10k_dql contains only ath10k patch which applies DQL to > driver-firmware tx queue naively > - sw/fq+ath10k_dql is obvious > - sw/base today's ath.git/master checkout used as base > - "veryfast" tests TCP tput to reference receiver (4 antennas) > - "fast" tests TCP tput to ref receiver (1 antenna) > - "slow" tests TCP tput to ref receiver (1 *unplugged* antenna) > - "fast+slow" tests sharing between "fast" and "slow" > - "autorate" uses default rate control > - "rate6m" uses fixed-tx-rate at 6mbps > - the test uses QCA9880 w/ 10.1.467 > - no rrul tests, sorry Dave! :) rrul would be a good baseline to have, but no need to waste your time on running it every time as yet. It stresses out both sides of the link so whenever you get two devices with these driver changes on them it would be "interesting". It's the meanest, nastiest test we have... if you can get past the rrul, you've truly won. Consistently using tcp_fair_up with 1,2,4 flows and 1-4 stations as you are now is good enough. doing a more voip-like test with slamming d-itg into your test would be good... > > Observations / conclusions: > - DQL builds up throughput slowly on "veryfast"; in some tests it > doesn't get to reach peak (roughly 210mbps average) because the test > is too short It looks like having access to the rate control info here for the initial and ongoing estimates will react faster and better than dql can. I loved the potential here in getting full rate for web traffic in the usual 2second burst you get it in (see above blog entries) > - DQL shows better latency results in almost all cases compared to > the txop based scheduling from my mac80211 RFC (but i haven't > thoroughly looked at *all* the data; I might've missed a case where it > performs worse) Well, if you are not saturating the link, latency will be better. Showing how much less latency is possible, is good too, but > - latency improvement seen on sw/ath10k_dql @ rate6m,fast compared to > sw/base (1800ms -> 160ms) can be explained by the fact that txq AC > limit is 256 and since all TCP streams run on BE (and fq_codel as the > qdisc) the induced txq latency is 256 * (1500 / (6*1024*1024/8.)) / 4 > = ~122ms which is pretty close to the test data (the formula ignores > MAC overhead, so the latency in practice is larger). Once you consider > the overhead and in-flight packets on driver-firmware tx queue 160ms > doesn't seem strange. Moreover when you compare the same case with > sw/fq+ath10k_dql you can clearly see the advantage of having fq_codel > in mac80211 software queuing - the latency drops by (another) order of > magnitude because now incomming ICMPs are treated as new, bursty flows > and get fed to the device quickly. It is always good to test codel and fq_codel separately, particularly on a new codel implementation. There are so many ways to get codel wrong or add an optimization that doesn't work (speaking as someone that has got it wrong often) If you are getting a fq result of 12 ms, that means you are getting data into the device with a ~12ms standing queue there. On a good day you'd see perhaps 17-22ms for "codel target 5ms" in that case, on the rtt_fair_up series of tests. if you are getting a pure codel result of 160ms, that means the implementation is broken. But I think (after having read your description twice), the baseline result today of 160ms of queuing was with a fq_codel *qdisc* doing the work on top of huge buffers, the results a few days ago were with a fq_codel 802.11 layer, and the results today
Re: [RFCv2 0/3] mac80211: implement fq codel
We have a huge cc list on this thread, and admittedly this work does cut across a great deal of wireless, potentially, but does netdev need to be on it? there's been nothing codel specific on it in a while, so I cut those from the cc. On Tue, Mar 22, 2016 at 1:05 AM, Michal Kazior <michal.kaz...@tieto.com> wrote: > On 21 March 2016 at 18:10, Dave Taht <dave.t...@gmail.com> wrote: >> thx. >> >> a lot to digest. >> >> A) quick notes on "flent-gui bursts_11e-2016-03-21T09*.gz" >> >> 1) the new bursts_11e test *should* have stuck stuff in the VI and VO >> queues, and there *should* have been some sort of difference shown on >> the plots with it. There wasn't. > > traffic-gen generates only BE traffic. Everything else runs UDP_RR > which doesn't generate a lot of traffic. > > >> For diffserv markings I used BE=CS0, BK=CS1, VI=CS5, and VO=EF. >> CS6/CS7 should also land in VO (at least with the soft mac handler >> last I looked). Is there a way to check if you are indeed exercising >> all four 802.11e hardware queues in this test? in ath9k it is the >> "xmit" sysfs var > > Hmm.. there are no txq stats. I guess it makes sense to have them? ath9k xmit has been useful to capture. I'm kind of unconvinced those stats are correct, at the moment, but... > There is /sys/kernel/debug/ieee80211/phy*/fq which dumps state of all > queues which will be mostly empty with UDP_RR. You can run netperf UDP > stream with diffserv marking to see onto which tid they are mapped. > You can see tid-AC mappings here: > https://wireless.wiki.kernel.org/en/developers/documentation/mac80211/queues We can try to capture those, but sampling summary per-station stats ties back better to actual traffic analysis. Also useful to capture has been the minstrel stats, the minstrel-blues version provided these in a handy csv format. > I just checked and EF ends up as tid5 which is VI. It's actually the > same as CS5. You can use CS7 to run on tid7 which is VO. The intent of CS6 is somewhat incompatible with VO's intent, but we can argue diffserv's usefulness and mappings another day. I have changed the bursts_11e test to use CS7, which will break parsing our previous test runs' data, but actually test what I'd intended to test in the first place. >> 2) In all the old cases the BE UDP_RR flow died on the first burst >> (why?), and the fullpatch preserved it. > > I think it's related to my setup which involves veth pairs. I use them > to simulate bridging/AP behavior but maybe it's not doing the job > right, hmm.. > > >> (I would have kind of hoped to >> have seen the BK flow die, actually, in the fullpatch) > > There's no extra weight priority to BK. The difference between BE and > BK in 802.11 is contention window access time so BK gets less txops > statistically. Both share the same txop, which is 5.484ms in most > cases. Um, well, another day. > >> 3) I am also confused on 802.11ac - can VO aggregate? ( can't in in 802.11n). > > Yes, it should be albeit VI and VO have shorter txop compared to > BE/BK: 3.008ms and 1.504ms respectively. Not being able to aggregate in VO in n was a bad thing. There is an awful lot I like about ac over n. > > UDP_RR doesn't really create a lot of opportunities for aggregation. > If you want to see how different queues behave when loaded you'll need > to modify traffic-gen and add bursts across different ACs in the > bursts_11e test. or flood the queues with other tests like rrul or toke's enhancement to traffic-gen. :) I liked being able to arbitrarily mark udp packets ecn capable... > > > Michał -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: making fq_codel default
On Tue, Mar 22, 2016 at 3:05 AM, Reinoud Koornstrawrote: > Thanks, that answers my question. Adding this to /etc/sysctl.conf or /etc/sysctl.d/bufferbloat.conf is generally what we do net.core.default_qdisc=fq_codel A lot of us are running ecn by default and put in ecn on also: net.ipv4.tcp_ecn=1 > Ok, so currently Cake didn't make it into the 4.4 kernel yet I noticed. > Are there plans to add this or are there still many issues to be worked out? The last major test series in late december showed issues with the then codel implementation, which was presumably fixed, and then also the new "triple queue isolation" code landed, which is still something of a head scratcher to test for the problem it's trying to solve (better per host fairness while still getting ). It has been my hope to see cake mainlined for about 4 linux kernel versions now, but the code has sprouted a great deal more instrumentation that I, personally, would like (like nearly all the stats below), and additional complexity, that may not be needed, it still could use some fixes (particularly in GRO peeling), diffserv modeling still has disputes, and performance analysis and tuning on the kinds of hardware (arm, mips) it is intended for, especially at a gbit. It is being incorporated in a couple openwrt builds, could use more eyeballs and testers, to firmly escape second system syndrome. Discussions are held on the cake mailing list. https://lists.bufferbloat.net/pipermail/cake/2015-December/001755.html I hope we can resume major testing on it again by the end of april. > In the mean time I followed some instructions to build the module and > iproute2 for cake. Comparison tests on your workloads and your specific devices between pfifo, fq_codel, and cake are welcomed, but I am pessimistic about any effects without a bql-like layer underneath it. The consensus generally is that while some cake-like algos might apply to wifi, the work needs to happen at the ieee80211 layer rather than the qdisc layer. The principal use case for cake for wifi has been in analyzing codel's behavior in the face of shifting rates on tc change dev wifi0 root cake bandwidth 10mbit sleep 2 tc change dev wifi0 root cake bandwidth 50mbit sleep 2 tc change dev wifi0 root cake bandwidth 20mbit > sudo tc qdisc add dev wlp4s0 root cake (iwlwifi) > > reinoud@router-dev:~/Downloads/linux-4.4.5/net/sched$ sudo tc -s qdisc show > > qdisc cake 8002: dev wlp4s0 root refcnt 5 unlimited diffserv4 flows > rtt 100.0ms raw > Sent 71025 bytes 516 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > memory used: 0b of 15140Kb > capacity estimate: 0bit > Tin 0 Tin 1 Tin 2 Tin 3 > thresh0bit0bit0bit0bit > target 5.0ms 5.0ms 5.0ms 5.0ms > interval 100.0ms 100.0ms 100.0ms 100.0ms > Pk-delay 0us 0us 0us 0us > Av-delay 0us 0us 0us 0us > Sp-delay 0us 0us 0us 0us > pkts 0 0 0 0 > bytes0 0 0 0 > way-inds 0 0 0 0 > way-miss 0 0 0 0 > way-cols 0 0 0 0 > drops0 0 0 0 > marks0 0 0 0 > Sp-flows 0 0 0 0 > Bk-flows 0 0 0 0 > last-len 0 0 0 0 > max-len0 0 0 0 Groovy. Go pound it flat with some tests and let us know if any of these statistics are useful to you. > On Tue, Mar 22, 2016 at 3:43 AM, Matthias May > wrote: >> On 22/03/16 10:37, Reinoud Koornstra wrote: >>> >>> Hi Everyone, >>> >>> Everytime I boot I need to set fq_codel for my wireless interface: >>> >>> sudo tc qdisc add dev wlp4s0 root fq_codel >>> >>> I also need to sudo sysctl -w net.core.default_qdisc=fq_codel >>> >>> Is there a good way to have this as the default in the kernel config >>> instead of pfifo? >>> Also, are there plans for cake support or do fq_codel in this case mean >>> cake? >>> Thanks, >>> >>> Reinoud. >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-wireless" >>> in >>> the body of a message to majord...@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> You might want to take a look at the patches in openwrt. >> Specifically this one: >> https://dev.openwrt.org/browser/trunk/target/linux/generic/patches-4.4/662-use_fq_codel_by_default.patch >> >> Best regards >> Matthias > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org >
Re: [RFCv2 0/3] mac80211: implement fq codel
On Thu, Mar 17, 2016 at 1:55 AM, Michal Kazior <michal.kaz...@tieto.com> wrote: > I suspect the BK/BE latency difference has to do with the fact that > there's bulk traffic going on BE queues (this isn't reflected > explicitly in the plots). The `bursts` flent test includes short > bursts of traffic on tid0 (BE) which is shared with ICMP and BE UDP_RR > (seen as green and blue lines on the plot). Due to (intended) limited > outflow (6mbps) BE queues build up and don't drain for the duration of > the entire test creating more opportunities for aggregating BE traffic > while other queues are near-empty and very short (time wise as well). I agree with your explanation. Access to the media and queue length are the two variables at play here. I just committed a new flent test that should exercise the vo,vi,be, and bk queues, "bursts_11e". I dropped the conventional ping from it and just rely on netperf's udp_rr for each queue. It seems to "do the right thing" on the ath9k And while I'm all in favor of getting 802.11e's behaviors more right, and this seems like a good way to get there... netperf's udp_rr is not how much traffic conventionally behaves. It doesn't do tcp slow start or congestion control in particular... In the case of the VO queue, for example, the (2004) intended behavior was 1 isochronous packet per 10ms per voice sending station and one from the ap, not a "ping". And at the time, VI was intended to be unicast video. TCP was an afterthought. (wifi's original (1993) mac was actually designed for ipx/spx!) I long for regular "rrul" and "rrul_be" tests against the new stuff to blow it up thoroughly as references along the way. (tcp_upload, tcp_download, (and several of the rtt_fair tests also between stations)). Will get formal about it here as soon as we end up on the same kernel trees Furthermore 802.11e is not widely used - in particular, not much internet bound/sourced traffic falls into more than BE and BK, presently. and in some cases weirder - comcast remarks a very large percentage of to the home inbound traffic as CS1 (BK), btw, and stations tend to use CS0. Data comes in on BK, acks go out on BE. I/we will try to come up with intermediate tests between the burst tests and the rrul tests as we go along the way. > If you consider Wi-Fi is half-duplex and latency in the entire stack In the context of this test regime... Saying wifi is "half"-duplex is a misleading way to think about it in many respects. it is a shared medium more like early, non-switched ethernet, with a weird mac that governs what sort of packets get access to (a txop) the medium first, across all stations co-operating within EDCA. Half or full duplex is something that mostly applied to p2p serial connections (or p2p wifi), not P2MP. Additionally characteristics like exponential backoff make no sense were wifi any form of duplex, full or half. Certainly much stuff within a txop (block acks for example) can be considered half duplex in a microcosmic context. I wish we actually had words that accurately described wifi's actual behavior. > (for processing ICMP and UDP_RR) is greater than 11e contention window > timings you can get your BE flow responses with extra delay (since > other queues might have responses ready quicker). yes. always having a request pending for each of the 802.11e queues is actually not the best idea, it is better to take advantage of better aggregation afforded by 802.11n/ac, to only have one or two of the queues in use against any given station and promote or demote traffic into a more-right queue. simple example of the damage having all 4 queues always contending is exemplified by running the rrul and rrul_be tests against nearly any given AP. > > I've modified traffic-gen and re-run tests with bursts on all tested > tids/ACs (tid0, tid1, tid5). I'm attaching the results. > > With bursts on all tids you can clearly see BK has much higher latency than > BE. The long term goal here, of course, is for BK (or the other queues) to not have seconds of queuing latency but something more bounded to 2x media access time... > (Note, I've changed my AP to QCA988X with oldie firmware 10.1.467 for > this test; it doesn't have the weird hiccups I was seeing on QCA99X0 > and newer QCA988X firmware reports bogus expected throughput which is > most likely a result of my sloppy proof-of-concept change in ath10k). So I should avoid ben greer's firmware for now? > > > Michał > > On 16 March 2016 at 20:48, Jasmine Strong <j...@eero.com> wrote: >> BK usually has 0 txop, so it doesn't do aggregation. >> >> On Wed, Mar 16, 2016 at 11:55 AM, Bob Copeland <m...@bobcopeland.com> wrote: >>> >>> On Wed, Mar 16, 2016 at 11:36:31AM -0700, Dave Taht wrote: >>> > That is the sanest 802.11e queu
Re: [RFC/RFT] mac80211: implement fq_codel for software queuing
>> regular fq_codel uses 1024 and there has not been much reason to >> change it. In the case of an AP which has more limited memory, 256 or >> 1024 would be a good setting, per station. I'd stick to 1024 for now. > > Do note that the 4096 is shared _across_ station-tid queues. It is not > per-station. If you have 10 stations you still have 4096 flows > (actually 4096 + 16*10, because each tid - and there are 16 - has it's > own fallback flow in case of hash collision on the global flowmap to > maintain per-sta-tid queuing). I have to admit I didn't parse this well - still haven't, I think I need to draw. (got a picture?) Where is this part happening in the code (or firmware?) " because each tid - and there are 16 - has it's own fallback flow in case of hash collision on the global flowmap to maintain per-sta-tid queuing" "fallback flow - hash collision on global flowmap" - huh? > With that in mind do you still think 1024 is enough? Can't answer that question without understanding what you said above. I assembled a few of the patches to date (your fq_codel patch, avery's and tims ath9k stuff) and tested them, to no measurable effect, against linus's tree a day or two back. I also acquired an ath10k card - would one of these suit? http://www.amazon.com/gp/product/B011SIMFR8?psc=1=true_=oh_aui_detailpage_o08_s00 -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
per sta queuing - the ath9k statistics
I have put together some of the patches for fq_codel and per-station queuing inside the mac80211 portion of the stack flying around on linux-wireless, to no real visible effect as yet. Mostly testing uploads at the moment, from an x86 based client. It's not clear if I have the code path enabled, either, nor how to check, from userspace. (?) Topology is x86 <-wifi-> wndr3800 <-ethernet-> pi Latency is still poor, throughput is down slightly. I will start printk-ing tomorrow. I do have a few puzzling things A) re the ath9k statistics At the client (ubuntu x86, 4.5-rc7 + patches, Atheros AR5418 Wireless Network Adapter [AR5008E ) I see http://pastebin.com/rvKJnc1y AMPDUs Queued HW:0 0 0 0 AMPDUs Queued SW:0 0 0 0 AMPDUs Completed: 1098389 7050 14967 0 At the AP (cerowrt 3.10.50) I see http://pastebin.com/RTt7MNT6 AMPDUs Queued SW: 3009455 364214557331 0 AMPDUs Completed: 2961055 363353556982 0 AMPDUs Retried: 115311 7833 21489 0 In both cases the TX-Pkts-all is close to the AMPDUs completed figure. B) In the regular packet captures I see no tcp losses. I can see in an aircap wifi retrying for every lost packet. C) I do not see any ECN marks (presumably would be generated by the codel implementation.) D) I do see things nicely "fq"'d on the captures but that might be by cerowrt rather than the ieee mac E) tc qdisc show dev wlp2s0 shows the tc layer qdisc disabled qdisc noqueue 0: root refcnt 2 which does imply that at least part of the new codepath is working, but there are no stats out of that side yet... Ah, well, at least the patchset compiled and didn't crash the box. -- Dave Täht Let's go make home routers and wifi faster! With better software! https://www.gofundme.com/savewifi -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/RFT] mac80211: implement fq_codel for software queuing
*txq = sta->sta.txq[tid]; > + struct ieee80211_sub_if_data *sdata; > + struct ieee80211_fq *fq; > struct txq_info *txqi; > > if (!txq) > return; > > txqi = to_txq_info(txq); > + sdata = vif_to_sdata(txq->vif); > + fq = >local->fq; > > /* Lock here to protect against further seqno updates on dequeue */ > - spin_lock_bh(>queue.lock); > + spin_lock_bh(>lock); > set_bit(IEEE80211_TXQ_STOP, >flags); > - spin_unlock_bh(>queue.lock); > + spin_unlock_bh(>lock); > } > > static void > diff --git a/net/mac80211/codel.h b/net/mac80211/codel.h > new file mode 100644 > index ..f6f1b9b73a9a > --- /dev/null > +++ b/net/mac80211/codel.h > @@ -0,0 +1,260 @@ > +#ifndef __NET_MAC80211_CODEL_H > +#define __NET_MAC80211_CODEL_H > + > +/* > + * Codel - The Controlled-Delay Active Queue Management algorithm > + * > + * Copyright (C) 2011-2012 Kathleen Nichols <nich...@pollere.com> > + * Copyright (C) 2011-2012 Van Jacobson <v...@pollere.net> > + * Copyright (C) 2016 Michael D. Taht <dave.t...@bufferbloat.net> > + * Copyright (C) 2012 Eric Dumazet <eduma...@google.com> > + * Copyright (C) 2015 Jonathan Morton <chromati...@gmail.com> > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + *notice, this list of conditions, and the following disclaimer, > + *without modification. > + * 2. Redistributions in binary form must reproduce the above copyright > + *notice, this list of conditions and the following disclaimer in the > + *documentation and/or other materials provided with the distribution. > + * 3. The names of the authors may not be used to endorse or promote products > + *derived from this software without specific prior written permission. > + * > + * Alternatively, provided that this notice is retained in full, this > + * software may be distributed under the terms of the GNU General > + * Public License ("GPL") version 2, in which case the provisions of the > + * GPL apply INSTEAD OF those given above. > + * > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH > + * DAMAGE. > + * > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "codel_i.h" > + > +/* Controlling Queue Delay (CoDel) algorithm > + * = > + * Source : Kathleen Nichols and Van Jacobson > + * http://queue.acm.org/detail.cfm?id=2209336 > + * > + * Implemented on linux by Dave Taht and Eric Dumazet > + */ > + > +/* CoDel5 uses a real clock, unlike codel */ > + > +static inline codel_time_t codel_get_time(void) > +{ > + return ktime_get_ns(); > +} > + > +static inline u32 codel_time_to_us(codel_time_t val) > +{ > + do_div(val, NSEC_PER_USEC); > + return (u32)val; > +} > + > +/* sizeof_in_bits(rec_inv_sqrt) */ > +#define REC_INV_SQRT_BITS (8 * sizeof(u16)) > +/* needed shift to get a Q0.32 number from rec_inv_sqrt */ > +#define REC_INV_SQRT_SHIFT (32 - REC_INV_SQRT_BITS) > + > +/* Newton approximation method needs more iterations at small inputs, > + * so cache them. > + */ > + > +static void codel_vars_init(struct codel_vars *vars) > +{ > + memset(vars, 0, sizeof(*vars)); > +} > + > +/* > + * > http://en.wikipedia.org/wiki/Methods_of_computing_square_roots#Iterative_methods_for_reciprocal_square_roots > + * new_invsqrt = (invsqrt / 2) * (3 - count * invsqrt^2) > + * > + * Here, invsqrt is a fixed point number (< 1.0), 32bit mantissa, aka Q0.32 > + */ > +static inline void codel_Newton_step(struct codel_vars *vars) >
Re: [RFC/RFT] mac80211: implement fq_codel for software queuing
On Mon, Mar 7, 2016 at 9:14 AM, Avery Pennarun <apenw...@gmail.com> wrote: > On Mon, Mar 7, 2016 at 11:54 AM, Dave Taht <dave.t...@gmail.com> wrote: >> If I can just get a coherent patch set that I can build, I will gladly >> join you on the testing ath9k in particular... can probably do ath10k, >> too - and do a bit of code review... this week. it's very exciting >> watching all this activity... >> >> but I confess that I've totally lost track of what set of trees and >> patchwork I should try to pull from. wireless-drivers-next? ath10k? >> wireless-next? net-next? toke and I have a ton of x86 platforms >> available to test on >> >> Avery - which patches did you use??? on top of what? > > The patch series I'm currently using can be found here: > > git fetch https://gfiber.googlesource.com/vendor/opensource/backports > ath9k_txq+fq_codel No common commits, but ok, thx for a buildable-looking tree. d@dancer:~/git/linux$ git clone -b ath9k_txq+fq_codel --reference net-next https://gfiber.googlesource.com/vendor/opensource/backports Cloning into 'backports'... warning: no common commits remote: Sending approximately 30.48 MiB ... remote: Counting objects: 4758, done remote: Finding sources: 100% (5/5) remote: Total 19312 (delta 12999), reused 19308 (delta 12999) Receiving objects: 100% (19312/19312), 30.48 MiB | 6.23 MiB/s, done. Resolving deltas: 100% (12999/12999), done. > > That's again backports-20160122, which comes from linux-next as of > 20160122. You can either build backports against whatever kernel > you're using (probably easiest) or try to use that version of > linux-next, or rebase the patches onto your favourite kernel. > >> In terms of "smoothing" codel... >> >> I emphatically do not think codel in it's current form is "ready" for >> wireless, at the very least the target should not be much lower than >> 20ms in your 2 station tests. There is another bit in codel where the >> algo "turns off" with only a single MTU's worth of packets >> outstanding, which could get bumped to the ideal size of the >> aggregate. "ideal" kind of being a variable based on a ton of other >> factors... > > Yeah, I figured that sort of thing would come up. I'm feeling forward > progress just by finally seeing the buggy oscillations finally happen, > though. :) It's *very* exciting to see y'all break things in a measurable, yet positive direction. > >> the underlying code needs to be striving successfully for per-station >> airtime fairness for this to work at all, and the driver/card >> interface nearly as tight as BQL is for the fq portion to behave >> sanely. I'd configure codel at a higher target and try to observe what >> is going on at the fq level til that got saner. > > That seems like two good goals. So Emmanuel's BQL-like thing seems > like we'll need it soon. > > As for per-station airtime fairness, what's a good approximation of > that? Perhaps round-robin between stations, one aggregate per turn, > where each aggregate has a maximum allowed latency? Strict round robin is a start, and simplest, yes. Sure. "Oldest station queues first" on a round (probably) has higher potential for maximizing txops, but requires more overhead. (shortest queue first would be bad). There's another algo based on last received packets from a station possibly worth fiddling with in the long run... as "maximum allowed latency" - well, to me that is eventually also a variable, based on the number of stations that have to be scheduled on that round. Trying to get away from 10 stations eating 5.7ms each + return traffic on a round would be nicer. If you want a constant, for now, aim for 2048us or 1TU. > I don't know how > the current code works, but it's probably almost like that, as long as > we only put one aggregate's worth of stuff into each hwq, which I > guess is what the BQL-like thing will do. I would avoid trying to think about or using 802.11e's 4 queues at the moment[1]. We also have fallout from mu-mimo to deal with, eventually, also, but gang scheduling starts to fall out naturally from these structures and methods... > > So if I understand correctly, what we need is, in the following order: > 1) Disable fq_codel for now, and get BQL-like thing working in ath9k > (and ensure we're getting airtime fairness even without fq_codel); > 2) Re-enable fq_codel and increase fq_codel's target up to 20ms for now; > 3) Tweak fq_codel's "turn off" size to be larger (how important is this?) > > Is that right? Sounds good. I have not reviewed the codel5 based implementation, it may not even have idea "#3" in it at the moment at all. The relevant line in codel.h i
Re: [RFC/RFT] mac80211: implement fq_codel for software queuing
If I can just get a coherent patch set that I can build, I will gladly join you on the testing ath9k in particular... can probably do ath10k, too - and do a bit of code review... this week. it's very exciting watching all this activity... but I confess that I've totally lost track of what set of trees and patchwork I should try to pull from. wireless-drivers-next? ath10k? wireless-next? net-next? toke and I have a ton of x86 platforms available to test on Avery - which patches did you use??? on top of what? In terms of "smoothing" codel... I emphatically do not think codel in it's current form is "ready" for wireless, at the very least the target should not be much lower than 20ms in your 2 station tests. There is another bit in codel where the algo "turns off" with only a single MTU's worth of packets outstanding, which could get bumped to the ideal size of the aggregate. "ideal" kind of being a variable based on a ton of other factors... the underlying code needs to be striving successfully for per-station airtime fairness for this to work at all, and the driver/card interface nearly as tight as BQL is for the fq portion to behave sanely. I'd configure codel at a higher target and try to observe what is going on at the fq level til that got saner. There are so many other variables and things left unaccounted for, as yet. -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC/RFT] mac80211: implement fq_codel for software queuing
On Tue, Mar 1, 2016 at 11:38 PM, Michal Kaziorwrote: > On 1 March 2016 at 15:02, Johannes Berg wrote: >> On Fri, 2016-02-26 at 14:09 +0100, Michal Kazior wrote: >>> >>> +typedef u64 codel_time_t; >> >> Do we really need this? And if yes, does it have to be in the public >> header file? Why a typedef anyway? > > Hmm.. I don't think it's strictly necessary. I just wanted to keep as > much from the original codel implementation as possible. I'm fine with > using just u64. This is an artifact of the original codel keeping time in (nsec >> 10) to fit into a 32 bit int. In codel5 we switched to native 64 bit timekeeping as simpler, to improve logging and reason about. u64 is fine. > >>> - * @txq_ac_max_pending: maximum number of frames per AC pending in >>> all txq >>> - * entries for a vif. >>> + * @txq_cparams: codel parameters to control tx queueing dropping >>> behavior >>> + * @txq_limit: maximum number of frames queuesd >> >> typo - queued >> >>> @@ -2133,7 +2155,8 @@ struct ieee80211_hw { >>> u8 uapsd_max_sp_len; >>> u8 n_cipher_schemes; >>> const struct ieee80211_cipher_scheme *cipher_schemes; >>> - int txq_ac_max_pending; >>> + struct codel_params txq_cparams; >> >> Should this really be a parameter the driver determines? > > I would think so, or at least it should be able to influence it in > *some* way. You can have varying degree of induced latency depending > on fw/hw tx queue depth, air conditions and possible tx rates implying > different/varying RTT. Cake[1] even has a few RTT presets like: lan, > internet, satellite. While those presets have been useful in testing codel and (more generically in cake we can rapidly change the bandwidth from userspace for testing), in the real world you don't move from orbit to desktop and back as rapidly as wifi does. > I don't really have a plan how exactly a driver could make use of it > for benefit though. It might end up not being necessary after all if > generic tx scheduling materializes in mac80211. What we envisioned here is ewma smoothing the target based on the total service time needed for all active stations, per round. (There are other possible approaches) Say you serve 10 stations at 1ms each in one round, a codel target of 5ms will try to push things down too far. If in the next round, you only serve 2 stations at 1ms each (but get back 10 responses at .5ms each), you're still too high. If it's just one station, well, you can get below 2ms if the driver is only sending 1ms, but maybe it's sending 5ms... If you have a large multicast burst, that burp will cause lost packets. Merely getting typical wifi latencies under load down below the 20ms range would be a good start, after that some testing, hard thought, and evaluation are going to be needed. for early testing I think a 20ms fixed target would be safer than the existing 5ms. Pushing the fq part of fq_codel on a per station basis as close to the hardware as possible, and having better airtime fairness between stations is a huge win in itself. > > [1]: http://www.bufferbloat.net/projects/codel/wiki/Cake > > >>> +static void ieee80211_if_setup_no_queue(struct net_device *dev) >>> +{ >>> + ieee80211_if_setup(dev); >>> + dev->priv_flags |= IFF_NO_QUEUE; >>> + /* Note for backporters: use dev->tx_queue_len = 0 instead >>> of IFF_ */ >> >> Heh. Remove that comment; we have an spatch in backports already :) > > I've put it in the RFC specifically in case anyone wants to port this > bypassing backports, e.g. to openwrt's quilt (so when compilation > fails, you can quickly fix it up). I'll remove it before proper > submission obviously :) > > >>> --- a/net/mac80211/sta_info.h >>> +++ b/net/mac80211/sta_info.h >>> @@ -19,6 +19,7 @@ >>> #include >>> #include >>> #include "key.h" >>> +#include "codel_i.h" >>> >>> /** >>> * enum ieee80211_sta_info_flags - Stations flags >>> @@ -327,6 +328,32 @@ struct mesh_sta { >>> >>> DECLARE_EWMA(signal, 1024, 8) >>> >>> +struct txq_info; >>> + >>> +/** >>> + * struct txq_flow - per traffic flow queue >>> + * >>> + * This structure is used to distinguish and queue different traffic >>> flows >>> + * separately for fair queueing/AQM purposes. >>> + * >>> + * @txqi: txq_info structure it is associated at given time >>> + * @flowchain: can be linked to other flows for RR purposes >>> + * @backlogchain: can be linked to other flows for backlog sorting >>> purposes >>> + * @queue: sk_buff queue >>> + * @cvars: codel state vars >>> + * @backlog: number of bytes pending in the queue >>> + * @deficit: used for fair queueing balancing >>> + */ >>> +struct txq_flow { >>> + struct txq_info *txqi; >>> + struct list_head flowchain; >>> + struct list_head backlogchain; >>> + struct sk_buff_head queue; >>> + struct codel_vars cvars; >>> + u32 backlog; >>> + u32 deficit; >>> +}; >>> + >>> /** >>> * struct sta_info - STA information
Re: [PATCH] mac80211: fix AP buffered multicast frames with queue control and txq
On Thu, Mar 3, 2016 at 7:14 AM, Johannes Berg <johan...@sipsolutions.net> wrote: > On Sun, 2016-02-28 at 09:35 -0800, Dave Taht wrote: >> On Sun, Feb 28, 2016 at 6:19 AM, Felix Fietkau <n...@openwrt.org> >> wrote: >> > Buffered multicast frames must be passed to the driver directly via >> > drv_tx instead of going through the txq, otherwise they cannot >> > easily be >> > scheduled to be sent after DTIM. >> > >> > Signed-off-by: Felix Fietkau <n...@openwrt.org> >> > --- >> > net/mac80211/tx.c | 3 ++- >> > 1 file changed, 2 insertions(+), 1 deletion(-) >> > >> > diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c >> > index 3a7475f..b294820 100644 >> > --- a/net/mac80211/tx.c >> > +++ b/net/mac80211/tx.c >> > @@ -1247,7 +1247,8 @@ static void ieee80211_drv_tx(struct >> > ieee80211_local *local, >> > struct txq_info *txqi; >> > u8 ac; >> > >> > - if (info->control.flags & IEEE80211_TX_CTRL_PS_RESPONSE) >> > + if ((info->flags & IEEE80211_TX_CTL_SEND_AFTER_DTIM) || >> > + (info->control.flags & IEEE80211_TX_CTRL_PS_RESPONSE)) >> > goto tx_normal; >> > >> > if (!ieee80211_is_data(hdr->frame_control)) >> > -- >> > 2.2.2 >> >> I would like > > Feel free to propose patches for anything you like :) At the moment all I can do is cheer people on, and try to assemble enough gear to test comprehensively when enough patches have landed in your tree... Go, felix! Go, Michal! Go Mohammed! Go, Ben! Go Tim! Go, Emmanuel! Go Johannes! > > johannes >> -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] mac80211: fix AP buffered multicast frames with queue control and txq
On Sun, Feb 28, 2016 at 6:19 AM, Felix Fietkauwrote: > Buffered multicast frames must be passed to the driver directly via > drv_tx instead of going through the txq, otherwise they cannot easily be > scheduled to be sent after DTIM. > > Signed-off-by: Felix Fietkau > --- > net/mac80211/tx.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c > index 3a7475f..b294820 100644 > --- a/net/mac80211/tx.c > +++ b/net/mac80211/tx.c > @@ -1247,7 +1247,8 @@ static void ieee80211_drv_tx(struct ieee80211_local > *local, > struct txq_info *txqi; > u8 ac; > > - if (info->control.flags & IEEE80211_TX_CTRL_PS_RESPONSE) > + if ((info->flags & IEEE80211_TX_CTL_SEND_AFTER_DTIM) || > + (info->control.flags & IEEE80211_TX_CTRL_PS_RESPONSE)) > goto tx_normal; > > if (!ieee80211_is_data(hdr->frame_control)) > -- > 2.2.2 I would like hooks to emerge to be able to keep the level of multicast at a dull roar, relative to other traffic, and the impact of a multicast bursts measureable (stat exposed to userspace, something reporting back to the main tx queues that one just happened and how long it took). On receive, too. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] codel: add forgotten inline to functions in header file
On Thu, Feb 11, 2016 at 7:05 AM, Grumbach, Emmanuelwrote: > fixing linux-wireless address ... > > On 02/11/2016 04:30 PM, Eric Dumazet wrote: >> On Thu, 2016-02-11 at 16:08 +0200, Emmanuel Grumbach wrote: >>> Signed-off-by: Emmanuel Grumbach >>> --- >>> -static bool codel_should_drop(const struct sk_buff *skb, >>> - struct Qdisc *sch, >>> - struct codel_vars *vars, >>> - struct codel_params *params, >>> - struct codel_stats *stats, >>> - codel_time_t now) >>> +static inline bool codel_should_drop(const struct sk_buff *skb, >>> + struct Qdisc *sch, >>> + struct codel_vars *vars, >>> + struct codel_params *params, >>> + struct codel_stats *stats, >>> + codel_time_t now) >> >> The lack of inline was done on purpose. >> >> This include file is kind of special, being included by codel and >> fq_codel. >> >> Hint : we do not want to force the compiler to inline >> codel_should_drop() (or any other function). >> >> >> See this file as if it was a .c really. >> >> > > Yeah :) codel_should_drop seemed very long indeed... I wanted to use the > codel_get_time and associated utils (_before, _after) in iwlwifi. > They're better than jiffies... So maybe I can just copy that code to > iwlwifi. I need to stress that codel as is is not the right thing for wifi, particularly point to multipoint wifi in highly contended scenarios. It IS a starting point. We have generally felt that the target needs to be offset against the actual service opportunities, and the effects of multicast (with powersave) and other "background" frames, needs to be smoothed out. Lacking hardware that can do that, or adaquate sims, has stalled trying to come up with "the right thing". It looks like you are putting in place more of the pieces to get there in some tree somewhere? -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] codel: add forgotten inline to functions in header file
On Thu, Feb 11, 2016 at 7:29 AM, Grumbach, Emmanuelwrote: > > > On 02/11/2016 05:12 PM, Eric Dumazet wrote: >> On Thu, 2016-02-11 at 15:05 +, Grumbach, Emmanuel wrote: >> >> >>> Yeah :) codel_should_drop seemed very long indeed... I wanted to use the >>> codel_get_time and associated utils (_before, _after) in iwlwifi. >>> They're better than jiffies... So maybe I can just copy that code to >>> iwlwifi. Definately better than jiffies. >> >> You certainly can submit a patch adding the inline, but not on all >> functions present in this file ;) >> >> Thanks ! >> > > Actually... All I need *has* the inline, but if I include codel.h, > codel_dequeue is defined but not used and you definitely don't want to > inline that one. So I guess the only other option I have is to split > that header file which I don't think is really worth it. So, unless you > object it, I'll just copy the code. I think it is best to start with another base implementation of codel for wifi, yes. What I think is the currently best performing codel implementation (on the wire, for ethernet) we have is in: https://github.com/dtaht/bcake/codel5.h which has a few differences from eric's implementation (64 bit timestamps, inlining, not a lot of cpu profiling on it - still aiming for algorithmic correctness here, it is closer to the original paper... We'd used a different means of injecting the callback in it, too) The one currently in the main cake had issues in the last test round but has been updated since. (sch_cake is also on github). In neither case it is the right thing for wifi either. the "starting plan" such as it was was to get to "one aggregate in the hardware, one in the driver, one ready to be formed on the completion interrupt", and pull a smoothed service time from start to completion interrupt to dynamically modify the codel target. (other headaches, like multicast, abound). (It's the per station queue + fq as close to the hardware as possible that matters most, IMHO.) > > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v4] mac80211: add A-MSDU tx support
On Mon, Feb 8, 2016 at 3:23 AM, Felix Fietkauwrote: > On 2016-02-08 12:06, Krishna Chaitanya wrote: >> On Mon, Feb 8, 2016 at 4:34 PM, Felix Fietkau wrote: >>> On 2016-02-08 10:54, Krishna Chaitanya wrote: On Mon, Feb 8, 2016 at 2:56 PM, Emmanuel Grumbach wrote: > On Mon, Feb 8, 2016 at 10:38 AM, Felix Fietkau wrote: >> Requires software tx queueing support. frag_list support (for zero-copy) >> is optional. >> >> Signed-off-by: Felix Fietkau >> --- > > > Ok - looks fine, but... and here comes the hard stuff. > The frame size in the PLCP is limited in a way that you can't - from a > spec POV - enable A-MSDU for low rates. Of course, you don't want to > do that for low rates at all regardless of the spec. > Since you build the A-MSDU in the mac80211 Tx queue which is not aware > of the link quality, how do we prevent A-MSDU if the rate is low / > dropping. > I'd even argue that when the rates get lower, you'll have more > packets piling up in the software queue and ... even more chance to > get A-MSDU in the exact case where you really want to avoid it? Similar to triggering AMPDU setup, we should put this control in RC (minstrel) to start/stop AMSDU based on link quality/if the rates drop below a pre-defined MCS (or) only for best-throughput rates. >>> I think starting/stopping A-MSDU based on the rate is a bad idea. >>> Even with low rates, using A-MSDU can be a good thing (especially for >>> TCP ACKs), it just needs different size limits. >> >> By low rates, i was referring to bad channel conditions (more >> retries/crc errors) >> so using AMSDU might trigger more TCP level retries and for case >> of TCP ACK's its still worse in that it triggers TCP data retires from the >> peer. > Based on the research and data from the Bufferbloat project, I'd say > that in this case the latency due to queue buildup is a lot more harmful > than lost packets. +1. > With unmanaged queues, the latency will cause unnecessary > retransmissions anyway. typically at greater than 250ms, yes. It's my hope that everyone has agreed that > 250ms of queuing is bad, except for actual transit to the moon and back, at this point. My hope is that everyone can aim for < 20ms in the more general case. I was very happy to see the iwl patch go by cutting things down from seconds to 30-250ms under load, but am now puzzled. One iwl user has reported that AMPDUs are disabled by default on iwl, and had to do this: options iwlwifi 11n_disable=8 For a 4fold reduction in latency under load in his tests, iwl to ath9k. What enables/disable AMPDU and AMSDU in the general case? > With managed queues, packet drops start increasing with latency until > TCP starts behaving properly. > In both cases you have extra TCP retransmissions... One thrust of mine has been to get ecn more widely adopted, which would eliminate the tcp retransmit problem on devices using it for congestion control (except on actual loss). Apple turned ecn on universally on all dev devices last august. They ran into trouble with one major ISP and had to turn it off, I am not sure if they shipped with it on or not. https://plus.google.com/u/0/107942175615993706558/posts/1j8nXtLGZDm Stuart Cheshire also points at a real case where excessive queuing hurts and fq doesn't help - screen sharing - with a nice demo combining fq_codel + the tcp_netsent_lowat option. (his talk starts at about 16 minutes in, he goes into the relevant bits at about 24 minutes in, and I'm sorry for always pointing people at talks rather than papers) Toke ("the good, the bad, and the wifi") points to excessive queuing *not* being a problem so long as the fq portion is sufficiently close to the hardware, in other scenarios. http://www.sciencedirect.com/science/article/pii/S1389128615002479 Now, as to whether a driver or device doing retransmits should (or even could) drop or mark packets is a good question. I am biased towards achieving low latency and per device fairness so I lean towards giving up a lot sooner in bad conditions and moving on > With bad conditions you also get a strong increase in per-TXOP latency. Having an agreed upon table for all the "bad" conditions and what should be done in these cases would be good. I don't regard "contention" as a bad condition, but as an indicator to be smarter about bunching up and/or discarding packets. Interference is a "bad" condition. Low rates for one station compared to the others is not a bad condition, as striving for airtime fairness between all stations is a reasonable goal... etc... > With A-MSDU you need fewer TXOPs for the same amount of data in the queue. I like it. :) I note that for stations, I am perpetually seeing this sort of behavior: ap -> transmits a bunch of packets to station station -> takes 2 or more txops to respond
Re: [RFC v2] iwlwifi: pcie: transmit queue auto-sizing
> A bursted txop can be as big as 5-10ms. If you consider you want to > queue 5-10ms worth of data for *each* station at any given time you > obviously introduce a lot of lag. If you have 10 stations you might > end up with service period at 10*10ms = 100ms. This gets even worse if > you consider MU-MIMO because you need to do an expensive sounding > procedure before transmitting. So while SU aggregation can probably > still work reasonably well with shorter bursts (1-2ms) MU needs at > least 3ms to get *any* gain when compared to SU (which obviously means > you want more to actually make MU pay off). I am not sure where you get these numbers. Got a spreadsheet? Gradually reducing the maximum sized txop as a function of the number of stations makes sense. If you have 10 stations pending delivery and reduced the max txop to 1ms, you hurt bandwidth at that instant, but by offering more service to more stations, in less time, they will converge on a reasonable share of the bandwidth for each, faster[1]. And I'm sure that the person videoconferencing on a link like that would appreciate getting some service inside of a 10ms interval, rather than a 100ms. yes, there's overhead, and that's not the right number, which would vary as to g,n,ac and successors. You will also get more opportunities to use mu-mimo with shorter bursts extant and more stations being regularly serviced. [1] https://www.youtube.com/watch?v=Rb-UnHDw02o at about 13:50 > The rule of thumb is the > longer you wait the bigger capacity you can get. This is not strictly true as the "fountain" of packets is regulated by acks on the other side of the link, and ramp up or down as a function of service time and loss. > > Apparently there's interest in maximizing throughput but it stands in > direct opposition of keeping the latency down so I've been thinking > how to satisfy both. > > The current approach ath10k is taking (patches in review [1][2]) is to > use mac80211 software queues for per-station queuing, exposing queue > state to firmware (it decides where frames should be dequeued from) > and making it possible to stop/wake per-station tx subqueue with fake > netdev queues. I'm starting to think this is not the right way though > because it's inherently hard to control latency and there's a huge > memory overhead associated with the fake netdev queues. What is this overhead? Applying things like codel tend to dramatically shorten the amount of skbs extant... modern 802.11ac capable hardware has tons more memory... > Also fq_codel > is a less effective with this kind of setup. fq_codel's principal problems with working with wifi are long and documented in the talk above. > My current thinking is that the entire problem should be solved via > (per-AC) qdiscs, e.g. fq_codel. I guess one could use > limit/target/interval/quantum knobs to tune it for higher latency of > aggregation-oriented Wi-Fi links where long service time (think > 100-200ms) is acceptable. However fq_codel is oblivious to how Wi-Fi > works in the first place, i.e. Wi-Fi gets better throughput if you > deliver bursts of packets destined to the same station. Moreover this > gets even more complicated with MU-MIMO where you may want to consider > spatial location (which influences signal quality when grouped) of > each station when you decide which set of stations you're going to > aggregate to in parallel. Since drivers have a finite tx ring this it > is important to deliver bursts that can actually be aggregated > efficiently. This means driver would need to be able to tell qdisc > about per-flow conditions to influence the RR scheme in some way > (assuming a qdiscs even understands flows; do we need a unified way of > talking about flows between qdiscs and drivers?). This is a very good summary of the problems in layering fq_codel as it exists today on top of wifi as it exists today. :/ Our conclusion several years ago was that as the information needed to do things more right was in the mac80211 layer that we could not evolve the qdisc layer to suit, and needed to move the core ideas into the mac80211 layer. Things have evolved since, but I still think we can't get enough info up to the qdisc layer (locks and so on) to use it sanely. > > [1]: https://www.spinics.net/lists/linux-wireless/msg146187.html > [2]: https://www.spinics.net/lists/linux-wireless/msg146512.html I will review! > For reference, ath10k has around 1400 tx descriptors, though in practice not all are usable, and in stock firmware, I'm guessing the NIC will never be able to actually fill up it's tx descriptors and stop traffic. Instead, it just allows the stack to try to TX, then drops the frame... >>> >>> >>> 1400 descriptors, ok... but they are not organised in queues? >>> (forgive my ignorance of athX drivers) >> >> >> I think all the details are in the firmware, at least for now. > > Yeah. Basically ath10k has a flat set of tx descriptors which are > AC-agnostic.
Re: [RFC RESEND] iwlwifi: pcie: transmit queue auto-sizing
ook to do that is in use on the mt72 chipset that felix is working on... but nowhere else so far as I know (as yet). the iwl does it's own aggregation (I think(?))... but estimates can still be made... There are WAY more details of course - per station queuing, a separate multicast queue, only some in that talk!, but my hope was that under good conditions we'd get wireless-n down below 12ms driver overhead, even at 6mbit, before something like fq_codel could kick in (under good conditions! Plenty of other potential latency sources beside excessive queuing in wifi!). My ideal world would be to hold it at under 1250us at higher rates Periodically sampling seems like a reasonable approach under lab conditions but it would be nicer to have feedback from the firmware - "I transmitted the last tx as an X byte aggregate, at MCS1, I had to retransmit a few packets once, it took me 6ms to acquire the media, I heard 3 other stations transmitting, etc.". The above info we know we can get from a few chipsets, but not enough was known about the iwl last I looked. And one reason why fq_codel - unassisted - is not quite the right thing on top of this is that multicast can take a really long time... Regardless, I'd highly love to see/use this patch myself in a variety of real world conditions and see what happens. And incremental progress is the only way forward. Thx for cheering me up. > > Cc: Stephen Hemminger <step...@networkplumber.org> > Cc: Dave Taht <dave.t...@gmail.com> > Cc: Jonathan Corbet <cor...@lwn.net> > Signed-off-by: Emmanuel Grumbach <emmanuel.grumb...@intel.com> > --- > Fix Dave's email address > --- > drivers/net/wireless/intel/iwlwifi/pcie/internal.h | 6 > drivers/net/wireless/intel/iwlwifi/pcie/tx.c | 32 > -- > 2 files changed, 35 insertions(+), 3 deletions(-) > > diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/internal.h > b/drivers/net/wireless/intel/iwlwifi/pcie/internal.h > index 2f95916..d83eb56 100644 > --- a/drivers/net/wireless/intel/iwlwifi/pcie/internal.h > +++ b/drivers/net/wireless/intel/iwlwifi/pcie/internal.h > @@ -192,6 +192,11 @@ struct iwl_cmd_meta { > u32 flags; > }; > > +struct iwl_txq_auto_size { > + int min_space; > + unsigned long reset_ts; > +}; > + > /* > * Generic queue structure > * > @@ -293,6 +298,7 @@ struct iwl_txq { > bool block; > unsigned long wd_timeout; > struct sk_buff_head overflow_q; > + struct iwl_txq_auto_size auto_sz; > }; > > static inline dma_addr_t > diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/tx.c > b/drivers/net/wireless/intel/iwlwifi/pcie/tx.c > index 837a7d5..4d1dee6 100644 > --- a/drivers/net/wireless/intel/iwlwifi/pcie/tx.c > +++ b/drivers/net/wireless/intel/iwlwifi/pcie/tx.c > @@ -572,6 +572,8 @@ static int iwl_pcie_txq_init(struct iwl_trans *trans, > struct iwl_txq *txq, > > spin_lock_init(>lock); > __skb_queue_head_init(>overflow_q); > + txq->auto_sz.min_space = 240; > + txq->auto_sz.reset_ts = jiffies; > > /* > * Tell nic where to find circular buffer of Tx Frame Descriptors for > @@ -1043,10 +1045,14 @@ void iwl_trans_pcie_reclaim(struct iwl_trans *trans, > int txq_id, int ssn, > q->read_ptr != tfd_num; > q->read_ptr = iwl_queue_inc_wrap(q->read_ptr)) { > struct sk_buff *skb = txq->entries[txq->q.read_ptr].skb; > + struct ieee80211_tx_info *info; > + unsigned long tx_time; > > if (WARN_ON_ONCE(!skb)) > continue; > > + info = IEEE80211_SKB_CB(skb); > + > iwl_pcie_free_tso_page(skb); > > __skb_queue_tail(skbs, skb); > @@ -1056,6 +1062,18 @@ void iwl_trans_pcie_reclaim(struct iwl_trans *trans, > int txq_id, int ssn, > iwl_pcie_txq_inval_byte_cnt_tbl(trans, txq); > > iwl_pcie_txq_free_tfd(trans, txq); > + > + tx_time = > (uintptr_t)info->driver_data[IWL_TRANS_FIRST_DRIVER_DATA + 2]; > + if (time_before(jiffies, tx_time + msecs_to_jiffies(10))) { > + txq->auto_sz.min_space -= 10; > + txq->auto_sz.min_space = > + max(txq->auto_sz.min_space, txq->q.high_mark); > + } else if (time_after(jiffies, > + tx_time + msecs_to_jiffies(20))) { > + txq->auto_sz.min_space += 10; > + txq->auto_sz.min_space = > + min(txq->auto_sz.min_space, 252); >
Re: [ath9k-devel] AR9462 problems connecting again..
On Tue, Feb 24, 2015 at 2:26 AM, Jouni Malinen j...@w1.fi wrote: On Tue, Feb 24, 2015 at 01:29:27PM +1100, Andrew McGregor wrote: Over the weekend I found a bug in minstrel-ht that might well be implicated here. The last retransmit rate is meant to be a 'get the packet there reliably' rate; minstrel-ht doesn't do that right, and can pick a fairly flaky rate instead. Can't generate a proper patch right now, so this diff might not apply cleanly, but the fix is simply to change 75 to 99 in the two places below: While this may indeed be helpful, I don't think it is sufficient for this EAPOL frame related issue. What I would like to see is minstrel_ht using a basic rate (something non-HT) at the end of the retry series for EAPOL frames. The current behavior looks very suspicious to me. The early EAPOL frames after association are being used to probe for higher rates. This results in the total number of retry attempts actually getting smaller than any other frame, i.e., minstrel_ht seems to be using significantly _less_ robust choices for the EAPOL frames than following normal data frames! This should really be the other way around.. As an example, I'm seeing this on 5 GHz band (with the 75 to 99 change in place, but behavior was more or less identical without it): - the first EAPOL frame (msg 2/4) getting one attempt at MCS 3, 2 attempts at MCS 0, 2 attempts at MCS 0 (yes, identical to the previous one) with total maximum of 5 attempts - the second EAPOL frame (msg 4/4) getting one attempt at MCS 9, 2 attempts at MCS 0, 2 attempts at MCS 0 with total maximum of 5 attempts - another data frame after this: 5 attempts at MCS 9, 5 attempts at MCS 3, 5 attempts at MCS 3 with total maximum of 15 attempts(!!) I would in general prefer that the excessive retries in the present driver layers in wifi be dramatically reduced, the packet dropped and the problem punted to higher layers. This cannot be the best approach here.. Falling back faster to the lowest possible rate with minimum retries, and then giving up sooner would be better. 15 attempts? jeeze For the IEEE80211_TX_CTRL_PORT_CTRL_PROTO cases, there are identified issues where failing to deliver the frame results is significant issues either in getting connected in the first place or getting disconnected if rekeying fails. I'm not sure how this would be implemented cleanly in minstrel_ht or whether that is even the best place (i.e., rate.c could do this instead), but I'd like that third attempt for control port cases to be dropped to use a (lowish) basic rate and non-MCS at that since there may be some interop issues with HT MCS early during association. Alternatively with drivers like ath9k that support 4 rate values, it would also be fine to add this basic rate attempt (or well, I'd have multiple, say 4, such attempts) as an additional 4th entry which does not currently seem to get used with minstrel at all. The (lowish) basic rate here could be defined as 6 Mbps OFDM for 5 GHz band and either that or maybe even 2 Mbps or 5.5 Mbps on 2.4 GHz (if included by the AP in basic rate set). -- Jouni MalinenPGP id EFC895FA ___ ath9k-devel mailing list ath9k-de...@lists.ath9k.org https://lists.ath9k.org/mailman/listinfo/ath9k-devel -- Dave Täht Let's make wifi fast, less jittery and reliable again! https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb -- To unsubscribe from this list: send the line unsubscribe linux-wireless in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [ath9k-devel] AR9462 problems connecting again..
On Sun, Feb 22, 2015 at 10:30 AM, Linus Torvalds torva...@linux-foundation.org wrote: On Sun, Feb 22, 2015 at 10:24 AM, Adrian Chadd adr...@freebsd.org wrote: Just a wild shot - try disabling fast authentication and see if that makes a difference? wpa_supplicant.conf: fast_reauth=0 I recall having issues with fast_reauth once, but I never stuck around that location long enough to debug it. Nope. Did that, killed wpa_supplicant (which restarts it), tried connecting, still failed. Hint: Several unifi (and most ubnt) products are well supported by openwrt directly, which by reflashing your device to it via their web interface would A) probably fix the problem, and B) give you more insight into fixing it, if it persists, by giving you full access on both sides of the connection. https://downloads.openwrt.org/snapshots/trunk/ar71xx/generic/ I have been replacing ubnt´s default firmware on first boot for 6+ years now. It is good hardware, after you do that. -- Dave Täht http://www.bufferbloat.net/projects/cerowrt/wiki -- To unsubscribe from this list: send the line unsubscribe linux-wireless in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Wifi outside the faraday cage (was: Throughput regression with `tcp: refine TSO autosizing`)
On Fri, Feb 6, 2015 at 1:57 AM, Michal Kazior michal.kaz...@tieto.com wrote: On 5 February 2015 at 20:50, Dave Taht dave.t...@gmail.com wrote: [...] And I really, really, really wish, that just once during this thread, someone had bothered to try running a test at a real world MCS rate - say MCS1, or MCS4, and measured the latency under load of that... Time between frame submission to firmware and tx-completion on one of my ath10k machines: THANK YOU for running these tests! Legacy 54mbps: ~18ms Legacy 6mbps: ~37ms legacy rates are what many people actually achieve, given the limited market penetration ac has on clients and APs. 11n MCS 3 (nss=0): ~13ms 11n MCS 8 (nss=1): ~6-8ms 11ac NSS=1 MCS=2: ~4-6ms 11ac NSS=2 MCS=0: ~5-8ms Keep in mind this is a clean room environment so retransmissions are kept at minimum. Obviously with a noisy environment you'll get retries at different rates and higher latency. It is difficult to reconcile the results you get in the clean room with the results I get from measurements in the real wold. I encourage you to go test your code in coffee shops, in offices with wifi, and in hotels and apartment buildings in preference to testing in the lab. I typically measure induced delays in the 3 to 6 second range in your typical conference scenario, which I measure at every conference I go to. The latest talk, including data on that, is friday morning, starting at 2:15 or so, at nznog: http://new.livestream.com/i-filmservices/NZNOG2015/videos/75358960 1) In the real world, I rarely see the maximum *rates*. I am personally quite fond of designing stuff with gears out of the middle of the Boston Gear Catalog. [1]. In looking over my largely outdoor wifi network, I see a cluster of values around mcs11, followed\by mcs4,3, 7 and 0, and *nothing* with MCS15. David lang is planning on doing some measurements at the SCALE conference next week, and I expect heaps of data from that, but I strongly suspect that the vast majority of connections in every circumstance except the test-bench are not even coming close to the maximum MCS rate in the standard(s). I would have thought that the lessons of the chromecast, where *every* attempt at reviewing it in an office environment failed, might have supplied industry clue that even 20Mbit to a given station is impossible in many cases due to airtime contention. Aggregates should be sized to have a maximum of 2 full ones stacked up at the rate being achieved for the destination, and the rest backlogged in the qdisc layer, if possible. 37ms backed up in the firmware is a lot, considering that the test above had no airtime contention in it, and no multicast. Drivers need to be aware that every TXOP is precious. I could see having a watchdog timer set on getting one packet into a wifi driver to wait a few hundred usec longer to fire off the write to the hardware in order to maximize aggregation by accumulating more packets to aggregate. I have hopes for xmit_more also being useful, but I am really not sure how well that works on single cores, interactions with napi, and with other wifi aggregates. It looks like adding xmit_more to the ag71xx driver will be easy... 2) In the real world I see media acquisition times *far* greater than 1ms. Please feel free to test your drivers in coffee shops, in the office, at hotels, in apartments... And retries... let's not talk about retries... 3) Longer AMPDUs lead to more tail loss and retries I have a paper around here somewhere that shows AMPDU loss and retries go up disproportionately as the length of transmission approaches 4ms. I hate drawing a conclusion from a paper I can't find, but my overall take on it is that as media acquisition time and retransmits go up, reducing AMPDU size from the maximum down to about 1ms at the current rate would lead to more fair, responsive, and fast-feeling wifi for everyone, improve ack clocking, flow mixing for web traffic, etc, etc. 4) There is some fairly decent academic work on other aspects of excessive buffering at lower rates http://hph16.uwaterloo.ca/~bshihada/publications/buffer-AMPDU.pdf (there are problems with this paper, but at least it tests n) and see google scholar for bufferbloat related papers in 2014 and later on wifi and LTE. 5) As for rate control, Minstrel was designed in an era when there wasn't one AP for every 4 people in the USA. Other people's rate controllers are even dumber, and minstrel-ht itself needs a hard look at n speeds, much less ac speeds. 6) Everything I say above applies to both stations and APs. APs have FAR worse problems, where per-tid (station) queuing is really needed in order to effectively aggregate when two or more stations are in use. Statistically, with two or more stations using traffic, aggregation possibilities will go down rapidly on a FIFO, (and go down even faster with FQ in place without per sta queuing!), and with the usual fixed buffersize underneath that, without per-tid
Re: Throughput regression with `tcp: refine TSO autosizing`
On Wed, Feb 11, 2015 at 11:48 PM, Michal Kazior michal.kaz...@tieto.com wrote: On 11 February 2015 at 09:57, Michal Kazior michal.kaz...@tieto.com wrote: On 10 February 2015 at 15:19, Johannes Berg johan...@sipsolutions.net wrote: On Tue, 2015-02-10 at 11:33 +0100, Michal Kazior wrote: + if (msdu-sk) { + ewma_add(ar-tx_delay_us, +ktime_to_ns(ktime_sub(ktime_get(), skb_cb-stamp)) / +NSEC_PER_USEC); + + ACCESS_ONCE(msdu-sk-sk_tx_completion_delay_cushion) = + (ewma_read(ar-tx_delay_us) * +msdu-sk-sk_pacing_rate) 20; + } To some extent, every wifi driver is going to have this problem. Perhaps we should do this in mac80211? Good point. I was actually thinking about it. I can try cooking a patch unless you want to do it yourself :-) I've taken a look into this. The most obvious place to add the timestamp for each packet would be ieee80211_tx_info (i.e. the skb-cb[48]). The problem is it's very tight there. Even squeezing 2 bytes (allowing up to 64ms of tx completion delay which I'm worried I will argue strongly in favor of never allowing more than 4ms packets to accumulate in the firmware. won't be enough) will be troublesome. Some drivers already use every last byte of their allowance on 64bit archs (e.g. ar5523 uses entire 40 bytes of driver_data). I wonder if it's okay to bump skb-cb to 56 bytes to avoid the cascade of changes required to implement the tx completion delay accounting? Michał -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Dave Täht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks -- To unsubscribe from this list: send the line unsubscribe linux-wireless in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Throughput regression with `tcp: refine TSO autosizing`
On Fri, Feb 6, 2015 at 2:44 AM, Michal Kazior michal.kaz...@tieto.com wrote: On 5 February 2015 at 14:19, Eric Dumazet eric.duma...@gmail.com wrote: On Thu, 2015-02-05 at 04:57 -0800, Eric Dumazet wrote: The intention is to control the queues to the following : 1 ms of buffering, but limited to a configurable value. On a 40Gbps flow, 1ms represents 5 MB, which is insane. We do not want to queue 5 MB of traffic, this would destroy latencies for all concurrent flows. (Or would require having fq_codel or fq as packet schedulers, instead of default pfifo_fast) This is why having 1.5 ms delay between the transmit and TX completion is a problem in your case. I do get your point. But 1.5ms is really tough on Wi-Fi. Just look at this: ; ping 192.168.1.2 -c 3 PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data. 64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=1.83 ms 64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=2.02 ms 64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=1.98 ms ; ping 192.168.1.2 -c 3 -Q 224 PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data. 64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.939 ms 64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=0.906 ms 64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=0.946 ms This was run with no load so batching code in the driver itself should have no measurable effect. The channel was near-ideal: low noise floor, cabled rf, no other traffic. The lower latency ping is when 802.11 QoS Voice Access Category is used. I also get 400mbps instead of 250mbps in this case with 5 flows (net/master). The VO queue is now nearly useless in a real world environment. Whlle it does grab the media mildly faster in some cases, on a good day with no other competing APs, it cannot aggregate packets, and wastes TXOPS. It is far saner to aim for better aggregate (use the VI queue if you must try to get better media acquisition). It is disabled in multiple products I know of. And I really, really, really wish, that just once during this thread, someone had bothered to try running a test at a real world MCS rate - say MCS1, or MCS4, and measured the latency under load of that... or tried talking to two or more stations at the same time. Instead of trying for 1.5Gbits in a faraday cage. Dealing with black box firmware blobs is a pain. +10 Note that TCP stack could detect when this happens, *if* ACK where delivered before the TX completions, or when TX completion happens, we could detect that the clone of the freed packet was freed. In my test, when I did ethtool -C eth0 tx-usecs 1024 tx-frames 64, and disabling GSO, TCP stack sends a bunch of packets (a bit less than 64), blocks on tcp_limit_output_bytes. Then we receive 2 stretch ACKS after ~50 usec. TCP stack tries to push again some packets but blocks on tcp_limit_output_bytes again. 1ms later, TX completion happens, tcp_wfree() is called, and TCP stack push following ~60 packets. TCP could eventually dynamically adjust the tcp_limit_output_bytes, using a per flow dynamic value, but I would rather not add a kludge in TCP stack only to deal with a possible bug in ath10k driver. niu has a similar issue and simply had to call skb_orphan() : drivers/net/ethernet/sun/niu.c:6669:skb_orphan(skb); Ok. I tried calling skb_orphan() right after I submit each Tx frame (similar to niu which does this in start_xmit): --- a/drivers/net/wireless/ath/ath10k/htt_tx.c +++ b/drivers/net/wireless/ath/ath10k/htt_tx.c @@ -564,6 +564,8 @@ int ath10k_htt_tx(struct ath10k_htt *htt, struct sk_buff *msdu) if (res) goto err_unmap_msdu; + skb_orphan(msdu); + return 0; err_unmap_msdu: Now, with {net/master + ath10k GRO + the above} I get 620mbps on a single flow (even better then before). Wow. Does this look ok/safe as a solution to you? Michał -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Dave Täht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks -- To unsubscribe from this list: send the line unsubscribe linux-wireless in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html