[Bloat] Getting Google to index. was:Re: [Starlink] [LibreQoS] [Rpm] net neutrality back in the news

[Rodney W. Grimes via Bloat]

> I love that there are oh, 700+ people on these mailing lists, but we
> have zero visibility due to google not indexing them, where hackernews
> does. This is going to be an issue dominating the web (again, sadly)
> for a few weeks at least, and it would really help to be doing it
> there, rather than here:

You can correct the lack of indexing by google if you put a
properly built web interface to the archives of the list.

 [snip]
> Dave T?ht CSO, LibreQos

--
Rod Grimes rgri...@freebsd.org
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Rpm] [Starlink] [LibreQoS] net neutrality back in the news

[dan via Bloat]

ok, lots and lots of great comments here for sure.

bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't
carrying to many places at affordable rates.  I've pulled quotes from Lumen
and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of
service.  This isn't abundance and it's widespread and it leaves only
major providers that can afford/amortize out 100G transits etc.
My answer to this is one that Dave and I have bounced back and forth is the
idea of micro IXs in every municipality and having that somehow tied into
access to the ROW in counties etc.  Not fully hashed out, but the fiber is
in the ground, it could be sold, but the carriers are not well incentivised
to sell it.  It takes the better part of a year to get a DIA within 100ft
of a Lumen hut sometimes...  Heck, it could even be a government program to
get an μ*IX* with x feet of every school, city hall, and library.  I don't
care how it's done but this would get abundance NEAR end users and open up
essentially every town to competition.

monopoly.  This is a historical thing for most cable and DSL incumbents.
They have enjoyed virtual monopolies with cable owning population centers
and DSL owning the outskirts and there is no product darwinism here where
customer satisfaction is a pressure.  That may not be the future but it
definitely is the past.  These companies may have to shift into customer
satisfaction as a major part instead of a minor part of their corporate
culture to fend off fttx and ultra-modern wisps.

Starlink is not offering significant competition to major carriers.
Starlink's 1.5 million customers are at LEAST 90% pulled from other
satellite services and small ISPs.  Spectrum and Comcast's losses to
starlink are measured in decimal points.

Only fttx and ultra-modern wireless tech really threatens these
incumbents.  Typical wisps aren't putting a dent in these guys, just
scraping the paint off their bumper.  We're pulling customers at the scale
of 'dozens' for example.  Spectrum's management doesn't know we exist we're
such a small threat to them.

But to further the point here, these fttx and ultra-modern wisps can only
exist in places where there is adequate carrier services to start with.  In
areas where they spend the money and do the build but there aren't good
carrier services, those fiber services suck and the wISPs start to claw
back even with inferior technology.  We've pulled quite a few customers off
fttx deployments because of this sort of situation.


On Fri, Sep 29, 2023 at 7:28 AM Rich Brown  wrote:

> Thank you Jonathan for this clear description of the issues and their
> history. I wonder if there's a fourth one - privacy.
>
> Rosenworcel's talk
> https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out
> that ISPs might want to monetize our traffic patterns and location data.
> (This is less of an issue in the EU, but the US remains a Wild West in this
> regard.)
>
> I am hopeful that the FCC will include this in their NPRM (which must be
> available by now but I haven't looked...)
>
> - Rich Brown
>
> > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <
> r...@lists.bufferbloat.net> wrote:
> >
> >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <
> bloat@lists.bufferbloat.net> wrote:
> >>
> >> Dave T called out earlier that the rise of bittorrent was a large part
> of the inital NN discussion here in the US. But a second large portion was
> a money grab from ISPs thinking that they could hold up large paid websites
> (netflix for example) for additional fees by threatening to make their
> service less useful to their users (viewing their users as an asset to be
> marketed to the websites rather than customers to be satisfied by providing
> them access to the websites)
> >>
> >> I don't know if a new round of "it's not fair that Netflix doesn't pay
> us for the bandwidth to service them" would fall flat at this point or not.
> >
> > I think there were three more-or-less separate concerns which have, over
> time, fallen under the same umbrella:
> >
> >
> > 1:  Capacity-seeking flows tend to interfere with latency-sensitive
> flows, and the "induced demand" phenomenon means that increases in link
> rate do not in themselves solve this problem, even though they may be sold
> as doing so.
> >
> > This is directly addressed by properly-sized buffers and/or AQM, and
> even better by FQ and SQM.  It's a solved problem, so long as the solutions
> are deployed.  It's not usually necessary, for example, to specifically
> enhance service for latency-sensitive traffic, if FQ does a sufficiently
> good job.  An increased link rate *does* enhance service quality for both
> latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> >
> >
> > 2:  Swarm traffic tends to drown out conventional traffic, due to
> congestion control algorithms which try to be more-or-less fair on a
> per-flow basis, and the substantially larger number of parallel flows used

Re: [Bloat] [LibreQoS] [Starlink] [Rpm] net neutrality back in the news

[dan via Bloat]

On Fri, Sep 29, 2023 at 7:17 AM Livingood, Jason via LibreQoS <
libre...@lists.bufferbloat.net> wrote:

> On 9/29/23, 00:54, "Jonathan Morton"  chromati...@gmail.com>> wrote:
> > Some ISPs began to actively degrade Netflix traffic, in particular by
> refusing to provision adequate peering capacity at the nodes through which
> Netflix traffic predominated
>
> That is not true and really not worth re-litigating here.
>
> > NN regulations forced ISPs to carry Netflix traffic with reasonable
> levels of service, even though they didn't want to for purely selfish and
> greedy commercial reasons.
>
> NN regulations played no role whatsoever in the resolution of that
> conflict - a business arrangement was reached, just as it was in the SK
> Telecom example recently:
> https://about.netflix.com/en/news/sk-telecom-sk-broadband-and-netflix-establish-strategic-partnership-to
>
> > ISPs behind L4S actively do not want a technology that works end-to-end
> over the general Internet.
>
> That's simply not true. As someone running an L4S field trial right now -
> we want the technology to get the widest possible deployment and be fully
> end-to-end. Why else would there be so much effort to ensure that ECN and
> DSCP marks can traverse network domain boundaries for example? Why else
> would there be strong app developer interest? What evidence do you have to
> show that anyone working on L4S want to create a walled garden? If
> anything, it seems the opposite of 5G network slicing, which seems to me
> personally to be another 3GPP run at walled garden stuff (like IMS).
> Ultimately it is like a lot of other IETF work -- it is an interesting
> technology and we'll have to see whether it gets good adoption - the
> 'market' will decide.
>
> > They want something that can provide a domination service within their
> own walled gardens.
>
> Also not correct. And last time I checked the balance sheets of companies
> in these sectors - video streaming services were losing money while
> provision of internet services were financially healthy.
>
> JL
>
>
>
I think this stuff degrades into conspiracy theory often enough.  While I
don't discount the possibility of collusion, I don't give these
people/groups credit enough to pull of a mass scale conspiracy either
If netflix is jammed down to small of a pipe at an ISP, that's more likely
(IMO...) disorganization or incompetence or disinterest over conspiracy.
 I feel the same about government in general...
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Rpm] [Starlink] [LibreQoS] net neutrality back in the news

[Livingood, Jason via Bloat]

On 9/29/23, 09:29, "Rich Brown" mailto:richb.hano...@gmail.com>> wrote:
> Thank you Jonathan for this clear description of the issues and their 
> history. I wonder if there's a fourth one - privacy. 
> Rosenworcel's talk also points out that ISPs might want to monetize our 
> traffic patterns and location data. (This is less of an issue in the EU, but 
> the US remains a Wild West in this regard.) 

That reference is to mobile networks in the US - but the US-EU contrast you 
make is a good one! The EU IMO does privacy right - it is not sector-specific 
regulation but is general privacy protecting law that protects user data no 
matter the entity collecting/aggregating/sharing. In the US we seem to pursue 
sector-specific privacy law - like specific to credit cards. What we end up 
with is a real mess and I would love to see comprehensive national data privacy 
legislation - but our legislative body can’t even agree right now to keep our 
government funded past this coming Sunday. ;-)

IANAL but it seems like if the US wanted to provide comprehensive location data 
privacy then it would have a uniform law that applied not just to a MNO with 
towers that can locate a handset, but also what the apps loaded on that handset 
with access to GPS can do with the data as well - and any other party that 
might be able to collect data.

JL 


___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Rpm] [Starlink] [LibreQoS] net neutrality back in the news

[Rich Brown via Bloat]

Thank you Jonathan for this clear description of the issues and their history. 
I wonder if there's a fourth one - privacy. 

Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf 
also points out that ISPs might want to monetize our traffic patterns and 
location data. (This is less of an issue in the EU, but the US remains a Wild 
West in this regard.) 

I am hopeful that the FCC will include this in their NPRM (which must be 
available by now but I haven't looked...)

- Rich Brown

> On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm 
>  wrote:
> 
>> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat 
>>  wrote:
>> 
>> Dave T called out earlier that the rise of bittorrent was a large part of 
>> the inital NN discussion here in the US. But a second large portion was a 
>> money grab from ISPs thinking that they could hold up large paid websites 
>> (netflix for example) for additional fees by threatening to make their 
>> service less useful to their users (viewing their users as an asset to be 
>> marketed to the websites rather than customers to be satisfied by providing 
>> them access to the websites)
>> 
>> I don't know if a new round of "it's not fair that Netflix doesn't pay us 
>> for the bandwidth to service them" would fall flat at this point or not.
> 
> I think there were three more-or-less separate concerns which have, over 
> time, fallen under the same umbrella:
> 
> 
> 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, 
> and the "induced demand" phenomenon means that increases in link rate do not 
> in themselves solve this problem, even though they may be sold as doing so.
> 
> This is directly addressed by properly-sized buffers and/or AQM, and even 
> better by FQ and SQM.  It's a solved problem, so long as the solutions are 
> deployed.  It's not usually necessary, for example, to specifically enhance 
> service for latency-sensitive traffic, if FQ does a sufficiently good job.  
> An increased link rate *does* enhance service quality for both 
> latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> 
> 
> 2:  Swarm traffic tends to drown out conventional traffic, due to congestion 
> control algorithms which try to be more-or-less fair on a per-flow basis, and 
> the substantially larger number of parallel flows used by swarm traffic.  
> This also caused subscribers using swarm traffic to impair the service of 
> subscribers who had nothing to do with it.
> 
> FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I 
> think it was occasionally used as an argument for *not* deploying FQ.  ISPs' 
> initial response was to outright block swarm traffic where they could 
> identify it, which was then softened to merely throttling it heavily, before 
> NN regulations intervened.  Usage quotas also showed up around this time, and 
> were probably related to this problem.
> 
> This has since been addressed by several means.  ISPs may use FQ on a 
> per-subscriber basis to prevent one subscriber's heavy traffic from degrading 
> service for another.  Swarm applications nowadays tend to employ altruistic 
> congestion control which deliberately compensates for the large number of 
> flows, and/or mark them with one or more of the Least Effort class DSCPs.  
> Hence, swarm applications are no longer as damaging to service quality as 
> they used to be.  Usage quotas, however, still remain in use as a profit 
> centre, to the point where an "unlimited" service is a rare and precious 
> specimen in many jurisdictions.
> 
> 
> 3:  ISPs merged with media distribution companies, creating a conflict of 
> interest in which the media side of the business wanted the internet side to 
> actively favour "their own" media traffic at the expense of "the 
> competition".  Some ISPs began to actively degrade Netflix traffic, in 
> particular by refusing to provision adequate peering capacity at the nodes 
> through which Netflix traffic predominated, or by zero-rating (for the 
> purpose of usage quotas) traffic from their own media empire while refusing 
> to do the same for Netflix traffic.
> 
> **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to 
> carry Netflix traffic with reasonable levels of service, even though they 
> didn't want to for purely selfish and greedy commercial reasons.  NN 
> succeeded in curbing an anti-competitive and consumer-hostile practice, which 
> I am perfectly sure would resume just as soon as NN regulations were repealed.
> 
> And this type of practice is just the sort of thing that technologies like 
> L4S are designed to support.  The ISPs behind L4S actively do not want a 
> technology that works end-to-end over the general Internet.  They want 
> something that can provide a domination service within their own walled 
> gardens.  That's why L4S is a NN hazard, and why they actively resisted all 
> attempts to displace it with SCE.
> 
> 
> All of the above 

Re: [Bloat] [Starlink] [LibreQoS] [Rpm] net neutrality back in the news

[Livingood, Jason via Bloat]

On 9/29/23, 00:54, "Jonathan Morton" mailto:chromati...@gmail.com>> wrote:
> Some ISPs began to actively degrade Netflix traffic, in particular by 
> refusing to provision adequate peering capacity at the nodes through which 
> Netflix traffic predominated

That is not true and really not worth re-litigating here. 

> NN regulations forced ISPs to carry Netflix traffic with reasonable levels of 
> service, even though they didn't want to for purely selfish and greedy 
> commercial reasons. 

NN regulations played no role whatsoever in the resolution of that conflict - a 
business arrangement was reached, just as it was in the SK Telecom example 
recently: 
https://about.netflix.com/en/news/sk-telecom-sk-broadband-and-netflix-establish-strategic-partnership-to
 

> ISPs behind L4S actively do not want a technology that works end-to-end over 
> the general Internet. 

That's simply not true. As someone running an L4S field trial right now - we 
want the technology to get the widest possible deployment and be fully 
end-to-end. Why else would there be so much effort to ensure that ECN and DSCP 
marks can traverse network domain boundaries for example? Why else would there 
be strong app developer interest? What evidence do you have to show that anyone 
working on L4S want to create a walled garden? If anything, it seems the 
opposite of 5G network slicing, which seems to me personally to be another 3GPP 
run at walled garden stuff (like IMS). Ultimately it is like a lot of other 
IETF work -- it is an interesting technology and we'll have to see whether it 
gets good adoption - the 'market' will decide. 

> They want something that can provide a domination service within their own 
> walled gardens. 

Also not correct. And last time I checked the balance sheets of companies in 
these sectors - video streaming services were losing money while provision of 
internet services were financially healthy. 

JL



___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [EXTERNAL] Re: [Rpm] net neutrality back in the news

[Erik Auerswald via Bloat]

Hi,

On Thu, Sep 28, 2023 at 09:07:23PM +, Livingood, Jason via Bloat wrote:
> […]
> Side note - a lot of IETF work flowed from the P2P workshop the IETF
> held in response to all this, including new WGs - LEDBAT, ALTO, CDNI.
> 
> JL
> 
> PS - some of the P2P workshop papers would be interesting to read again if we 
> could find them:

The wiki link in RFC 5594  seems 
to have been lost, but archive.org has
at least some of the contents, see, e.g.,
.

> [1] Nick Weaver - The case for "Ugly Now" User Fairness

https://web.archive.org/web/20090721015803/http://trac.tools.ietf.org/area/rai/trac/attachment/wiki/PeerToPeerInfrastructure/1%20p2pi-weaver.txt

> [2] Paul Jessop - Position paper of the RIAA

That's missing from archive.org.

> [3] Nikloaos Laotaris, Pablo Rodriguez, Laurent Massoulie - ECHOES:
> Edge Capacity Hosting Overlays of Nano Data Centers
> […]

I did not look for the other position papers.

Cheers,
Erik
-- 
If things aren’t simple, they won’t work.
-- Radia Perlman
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Starlink] [Rpm] [LibreQoS] net neutrality back in the news

[Sebastian Moeller via Bloat]

Hi Gert,


> On Sep 29, 2023, at 08:31, Gert Doering  wrote:
> 
> Hi,
> 
> On Fri, Sep 29, 2023 at 08:24:13AM +0200, Sebastian Moeller via Starlink 
> wrote:
>>  [SM] In the EU we have this as a continuous lobbying effort by big 
>> incumbent ISPs (a move to have the large content providers (CAPs) shoulder 
>> their "fair" share of the cost of modernizing the networks*), why this flys 
>> with at least some EU politicians is that the intended payees of this scheme 
>> are all located outside the EU and hence will have little support by the EU 
>> citizenry... (The latter is IMHO not fully undeserved either, the days of 
>> "do no evil" are long behind us and big tech often forgets that we are all 
>> in this together, but I digress). In the EU one of these days such an effort 
>> might actually succeed, as much as I dislike this.
> 
> And then the local incumbent uses that line of argument to arm-twist
> all the smaller ISPs to pay them for traffic into their network...
> (and calling up fees well above normal market rates for "transit").

Indeed, but that only flies because the regulators so far only feel 
responsible for the end-customer to internet access provider links, and 
explicitly exempt AS interconnect from their regulatory efforts. Given how 
complicated this can become I have some sympathy for their position, the 
national incumbent however plays a somewhat dangerous game, if he makes things 
too obvious it will likely result in regulatory interventions. This is also why 
the product sold is not "access to our eye-balls" but access "to the whole 
internet, including our eye-balls" yet at a cost that nobody is likely to use 
to access anything but that ISPs eye-balls. As much as it pains me that is 
behavior not untypical for large corporations these days...

Regards
Sebastian


> 
> Gert Doering
>-- NetMaster
> -- 
> have you enabled IPv6 on something today...?
> 
> SpaceNet AG  Vorstand: Sebastian v. Bomhard, Michael Emmer
> Joseph-Dollinger-Bogen 14Aufsichtsratsvors.: A. Grundner-Culemann
> D-80807 Muenchen HRB: 136055 (AG Muenchen)
> Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Rpm] [Starlink] [LibreQoS] net neutrality back in the news

[Sebastian Moeller via Bloat]

Hi David,


> On Sep 29, 2023, at 00:19, David Lang via Rpm  
> wrote:
> 
> On Thu, 28 Sep 2023, Livingood, Jason via Bloat wrote:
> 
>> Date: Thu, 28 Sep 2023 20:48:58 +
>> From: "Livingood, Jason via Bloat" 
>> Reply-To: "Livingood, Jason" 
>> To: dan , Dave Taht 
>> Cc: Rpm ,
>>Dave Taht via Starlink ,
>>bloat ,
>>libreqos ,
>>Jamal Hadi Salim 
>> Subject: Re: [Bloat] [Starlink] [LibreQoS] [Rpm] net neutrality back in the
>>news
>>> dan  wrote:
>> 
>>> "(I assume most ISPs want happy customers)."
> 
>>> made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' 
>>> in terms of customers being served by puts the Spectrums and Comcasts in 
>>> the mix (in the US) and they don't care about happy customers they care 
>>> about defacto monopolies in markets so that they don't have to care about 
>>> happy customers. 
>> 
>> In that context, happy customers stay longer (less churn) and spend more 
>> (upgrades, multiple services). And unhappy customers generate costs via 
>> disconnects (loss of revenue, costs to replace them with a new customer to 
>> just stay at the same subscriber levels), and costs via customer contacts 
>> (call center staff).
> 
> Except when you have a monopoly in an area, at which point the ability of 
> customers to leave is minimal, and years of bad customer service means that 
> people don't bother complaining, so the call center staffing costs are lower 
> than they should be.
> 
>>> For the last mile, I'm actually less concerned with pure NN and more 
>>> concerned with no-blocking or 'brand' prioritization and required/label 
>>> transparency...
>> 
>> The two thoughts your comments (thanks for the response BTW!) trigger are:
> 
>> 1 - Often regulation looks to the past - in this case maybe an era of 
>> bandwidth scarcity where prioritization may have mattered. I think we're in 
>> the midst of a shift into bandwidth abundance where priority does not 
>> matter. What will is latency/responsiveness, content/compute localization, 
>> reliability, consistency, security, etc.
> 
>> 2 - If an ISP blocked YouTube or Netflix, they'd incur huge customer care 
>> (contact) costs and would see people start to immediately shift to 
>> competitors (5G FWA, FTTP or DOCSIS, WISP, Starlink/LEO, etc.). It just does 
>> not seem like something that could realistically happen any longer in the US.
> 
> Dave T called out earlier that the rise of bittorrent was a large part of the 
> inital NN discussion here in the US. But a second large portion was a money 
> grab from ISPs thinking that they could hold up large paid websites (netflix 
> for example) for additional fees by threatening to make their service less 
> useful to their users (viewing their users as an asset to be marketed to the 
> websites rather than customers to be satisfied by providing them access to 
> the websites)
> 
> I don't know if a new round of "it's not fair that Netflix doesn't pay us for 
> the bandwidth to service them" would fall flat at this point or not.

[SM] In the EU we have this as a continuous lobbying effort by big 
incumbent ISPs (a move to have the large content providers (CAPs) shoulder 
their "fair" share of the cost of modernizing the networks*), why this flys 
with at least some EU politicians is that the intended payees of this scheme 
are all located outside the EU and hence will have little support by the EU 
citizenry... (The latter is IMHO not fully undeserved either, the days of "do 
no evil" are long behind us and big tech often forgets that we are all in this 
together, but I digress). In the EU one of these days such an effort might 
actually succeed, as much as I dislike this.




*) This argument about fairness is indeed made by the same ISPs that already 
charge their eye-ball customers for the same capacity they say they need to 
built with particpatoin of the CAPs


> 
> David Lang___
> Rpm mailing list
> r...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat