Re: [opensource-dev] New HTTP Library & Project Viewer

[Tateru Nino]

On 31/07/2012 9:13 AM, Henri Beauchamp wrote:
> On Mon, 30 Jul 2012 13:08:18 -0400, Oz Linden (Scott Lawrence) wrote:
>
>> HTTP itself recommends that clients not maintain more than 2 connections 
>> to the same service [1].  I don't know exactly what limit we will decide 
>> is reasonable (I expect that 2 will be ok, but don't know whether or not 
>> some larger number will be also).
> I bet 2 persistent connections will not be as performant as 16 (or even
> only 8) transient ones... Especially for oversea users, when links go
> through a load of routers before the viewer can actually communicate with
> LL's servers (and I have seen numerous times buggy/badly configured routers
> dropping "persistent" connections).

The pipelining of requests *should* contribute a great deal to
per-connection performance - there are outliers where that won't happen
(stuck/stalled connections, broken connections, and so forth), but in
the main properly-implemented pipelining, smartly managed by the viewer,
should provide quite a boost under the most common use-cases, IMO.
> Not to mention the underusage of the additional CPU cores for image decode
> and caching threads (that will have to wait each time and will, at best,
> process 2 images before the next 'bunch' (if 2 is already a 'bunch')
> arrives).
I'm not sure that's actually much of a concern. The number of textures
actually *ready* for decoding at any tick of the clock when they can be
scheduled is generally quite small (just one or two), even under the old
UDP system - more cores for texture decoding tasks primarily benefit the
slower systems where the decoding starts to eat up significant chunks of
user-scale time - or if you're really sucking down oodles of data at
speeds many of us overseas users can't actually pull. (I think it is
generally safe to assume a median data transfer rate of 400Kbps - this
is everyone's cue to jump up and tell me that the number is vastly
over/under any sane view of reality)

So, confining the number of persistent connections to an arbitrary X
(say 2 or 3) doesn't *seem* to me like it would make that much of a
difference in practical terms to many users. It will to *some*,
absolutely, who could doubtless get a *heap* more performance from X+1
or X+2 pipelines. There will always be those cases, IMO.


___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Henri Beauchamp]

On Mon, 30 Jul 2012 13:08:18 -0400, Oz Linden (Scott Lawrence) wrote:

> On 2012-07-29 04:07 , Henri Beauchamp wrote:
> > With just a few tweakings to the texture fetcher (and in particular
> > a setting that permits up to 32 concurrent HTTP fetches),
> 
> Caution... once we begin supporting persistent connections and pipelined 
> requests on the server side (we're working on it), opening that many 
> connections will very likely be considered abuse of the service.  
> Allowing a single server to have that many potentially long-lived 
> connections to the same server would starve the file descriptor pool 
> very quickly at the server end, adversely affecting other users.

Don't put word in my mouths, that I never pronounced... I was not speaking
about *persistent* connections. The current HTTP fetches are not persistent
connections, and that's why 32 simultaneous requests are not really an issue
(and it was actually the limit in Snowglobe v1.5, thus why I kept it in the
Cool VL Viewer, but made it configurable (and defaulting to 16), when I
noticed that the v2 viewers changed that limit down to 8 then up to 16 and
then to whatever number it is now hardcoded with).

> HTTP itself recommends that clients not maintain more than 2 connections 
> to the same service [1].  I don't know exactly what limit we will decide 
> is reasonable (I expect that 2 will be ok, but don't know whether or not 
> some larger number will be also).

I bet 2 persistent connections will not be as performant as 16 (or even
only 8) transient ones... Especially for oversea users, when links go
through a load of routers before the viewer can actually communicate with
LL's servers (and I have seen numerous times buggy/badly configured routers
dropping "persistent" connections).
Not to mention the underusage of the additional CPU cores for image decode
and caching threads (that will have to wait each time and will, at best,
process 2 images before the next 'bunch' (if 2 is already a 'bunch')
arrives).

> Please bear this in mind as you think about your designs.

Please, don't underestimate my thinking capacity... I also think I proved
numerous times with the Cool VL Viewer that I was very careful and quite
reasonnable (refusing to implement things that could overload servers:
I can give you half a dozen of examples if you want).

Henri.
___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Liny Odell]

On Mon, Jul 30, 2012 at 3:04 PM, Monty Brandenberg  wrote:
> On 7/30/2012 5:03 PM, Tateru Nino wrote:
>>
>> Heck, I know one person with two. A
>> Belkin G *and* a Linksys WRT.
>
> There's someone who I would like to buy a drink.

Im using my old wrt54gs with dd-wrt as a wireless access point/wired
switch and set up an old amd k6 computer with 2 nics and slapped
pfsense onto it and am using that as my router. Now if only I could
change my ISP from ATT, but sadly, thats _all_ thats available where I
live.
___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Monty Brandenberg]

On 7/30/2012 5:03 PM, Tateru Nino wrote:
>
> Heck, I know one person with two. A
> Belkin G *and* a Linksys WRT.

There's someone who I would like to buy a drink.

___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Tateru Nino]

On 31/07/2012 3:20 AM, Oz Linden (Scott Lawrence) wrote:
> On 2012-07-29 13:58 , Sheet Spotter wrote:
>> Does increasing the HTTP connection limit also increase the burden on the
>> server and network?
>>
>> Increasing the HTTP connection limit on one client might improve the
>> experience for one person. It might also diminish the experience for
>> everyone else on the same server.
> Exactly so.  The effect is more complex than just consuming file 
> descriptors on the server side, as well.  TCP congestion control works 
> far better when dealing with a smaller number of connections. Any 
> packets that are not subject to congestion control that are sharing the 
> same network path as a TCP connection increase the likelihood of a 
> dropped packet in the connection flow, which causes the connection to 
> immediately cut its transmission window by half. The viewer already puts 
> a lot of UDP traffic into the same path. The handshake packets that open 
> and close TCP connections are also not subject to the congestion control 
> - which means that opening each new connection increases the chances 
> that each of your existing connections will slow down by half.  Opening 
> many connections also puts additional strain on routers in the path that 
> are doing connection tracking (which too many do), adding another 
> possible source of problems.
>
> In general, when persistent connections and pipelined requests (having 
> more than one request in flight on a connection at a time) are used, 
> HTTP performs far better with a small number of connections than with 
> more.  As we begin deploying support for this way of using it on the 
> server side, we'll be able to do good experiments to determine what the 
> right tradeoffs are for network behavior and server load to provide 
> optimal experience for users.
>
>
Also, a badly behaving connection (eg: one that is dropping packets)
leaves the connection open longer, while delivering less data per unit
time. That's basically a *waste* of server resources. If I overload my
crappy consumer router with connections it can't properly handle, I'm
indirectly impacting the experience of other users. Not a desirable
situation.

___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Tateru Nino]

On 31/07/2012 5:28 AM, Monty Brandenberg wrote:
> On 7/30/2012 2:15 PM, Celierra Darling wrote:
>
>> FYI, the Firefox folks had a conversation in 2008 and decided to bump up
>> from 2 to 6 by default at that time (partly because everyone else was
>> raising it).[1]  (And for what it's worth, I found a mention from '06
>> that "anything above 10 is excessive".[2])  That doesn't necessarily
>> mean SL viewers should use the same values, but I think it probably
>> demonstrates where people might try to push that setting, at least at first.
>>
>> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=423377#c4
>> [2]
>> http://kb.mozillazine.org/index.php?title=Network.http.max-persistent-connections-per-server&diff=28784&oldid=28783
> On the limiting side of things, we have to deal with this
> unfortunately:
>
> http://www.smallnetbuilder.com/wireless/wireless-reviews/26843-linksyswrt54gv5reallyisalousyrouter?showall=&start=4
>
> Finding a one-size-fits-all solution is a challenge when the consumer
> space performance range spans a 3000:1 ratio.
>
> I also did some tests on throughput-vs-concurrency and at 10
> connections we're falling away from linear speedup.  The example
> code in the new library happens to be a performance test framework
> should anyone want to play...
>
It seems almost everyone I know has a linksys WRT. Heck *I* have one,
myself - the rev 2 - and there's three of us feeding through that. (Many
of the non US models have their firmware in ROM, rather than in flash
and can't be updated with DD-WRT). Heck, I know one person with two. A
Belkin G *and* a Linksys WRT.

Rubbish consumer routers are hard to ignore, even if you've got a good
one yourself.

___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Kadah]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 7/30/2012 10:20 AM, Oz Linden (Scott Lawrence) wrote:
> Opening many connections also puts additional strain on routers in
> the path that are doing connection tracking (which too many do),
> adding another possible source of problems.

It's a known issue that with the default concurrent setting of _4_
that many consumer level networking equipment can quickly overloaded
by the viewer. This project will eventually address local open
connection overloading (ie, few connection and allowing keep-alive).
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQFvBEAAoJEIdLfPRu7qE2nAgH/RZvaliZ9F/7YnNmskJYwMKJ
o3udee56w29zy5IRB/frL/82N8T7/MLkH6iF2en7BHgJpNG3rnN6O5mElAdwmkWN
WfzbVqL1sSoiUyzI6pHDISAhjyR/4yxtX8UEJ9ofVgY9Wr/BvPfS25V/jdr1OD0r
2vMm8neGGoQ3V5RgQXTmFlAz/6BgmFH/Ov198IMD1RuIs3k46x1xzVuMz6P//zTa
nNo/19O6y7Oog7/75nb/AufkvcARzjI7H6F4JlAQhKs1PA0VATsvNXAtiGSAF93K
UvQ91TJtBW1A5G3lcOvXVS5w8OjRNgN6FfieFilMjary23psq000bYkMzd91CR0=
=FhoG
-END PGP SIGNATURE-
___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Jonathan Ballard]

That's two connections at the process level, so that is not the limit for
each machine.  For example, in hardware, each server machine may have eight
network cards on PC motherboards (by what was on the market), so each card
can have 12 connections for each server process, and each client process
has 2 connections, and each network has 254 connections (IPv4), so there is
leverage for maximum throughput up to the switch.

On Monday, July 30, 2012, Oz Linden (Scott Lawrence) wrote:

>  On 2012-07-29 04:07 , Henri Beauchamp wrote:
>
> With just a few tweakings to the texture fetcher (and in particular
> a setting that permits up to 32 concurrent HTTP fetches),
>
>
> Caution... once we begin supporting persistent connections and pipelined
> requests on the server side (we're working on it), opening that many
> connections will very likely be considered abuse of the service.  Allowing
> a single server to have that many potentially long-lived connections to the
> same server would starve the file descriptor pool very quickly at the
> server end, adversely affecting other users.
>
> HTTP itself recommends that clients not maintain more than 2 connections
> to the same service [1].  I don't know exactly what limit we will decide is
> reasonable (I expect that 2 will be ok, but don't know whether or not some
> larger number will be also).
>
> Please bear this in mind as you think about your designs.  I will keep you
> all informed as things develop.
>
>
> [1] RFC 2616, section 8.1.4
>
>
___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Monty Brandenberg]

On 7/30/2012 2:15 PM, Celierra Darling wrote:

> FYI, the Firefox folks had a conversation in 2008 and decided to bump up
> from 2 to 6 by default at that time (partly because everyone else was
> raising it).[1]  (And for what it's worth, I found a mention from '06
> that "anything above 10 is excessive".[2])  That doesn't necessarily
> mean SL viewers should use the same values, but I think it probably
> demonstrates where people might try to push that setting, at least at first.
>
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=423377#c4
> [2]
> http://kb.mozillazine.org/index.php?title=Network.http.max-persistent-connections-per-server&diff=28784&oldid=28783

On the limiting side of things, we have to deal with this
unfortunately:

http://www.smallnetbuilder.com/wireless/wireless-reviews/26843-linksyswrt54gv5reallyisalousyrouter?showall=&start=4

Finding a one-size-fits-all solution is a challenge when the consumer
space performance range spans a 3000:1 ratio.

I also did some tests on throughput-vs-concurrency and at 10
connections we're falling away from linear speedup.  The example
code in the new library happens to be a performance test framework
should anyone want to play...

m

___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Celierra Darling]

On Mon, Jul 30, 2012 at 1:08 PM, Oz Linden (Scott Lawrence)
 wrote:
>
> On 2012-07-29 04:07 , Henri Beauchamp wrote:
>
> With just a few tweakings to the texture fetcher (and in particular
> a setting that permits up to 32 concurrent HTTP fetches),
>
>
> Caution... once we begin supporting persistent connections and pipelined
requests on the server side (we're working on it), opening that many
connections will very likely be considered abuse of the service.  Allowing
a single server to have that many potentially long-lived connections to the
same server would starve the file descriptor pool very quickly at the
server end, adversely affecting other users.
>
> HTTP itself recommends that clients not maintain more than 2 connections
to the same service [1].  I don't know exactly what limit we will decide is
reasonable (I expect that 2 will be ok, but don't know whether or not some
larger number will be also).
>
> Please bear this in mind as you think about your designs.  I will keep
you all informed as things develop.
>
>
> [1] RFC 2616, section 8.1.4


FYI, the Firefox folks had a conversation in 2008 and decided to bump up
from 2 to 6 by default at that time (partly because everyone else was
raising it).[1]  (And for what it's worth, I found a mention from '06 that
"anything above 10 is excessive".[2])  That doesn't necessarily mean SL
viewers should use the same values, but I think it probably demonstrates
where people might try to push that setting, at least at first.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=423377#c4
[2]
http://kb.mozillazine.org/index.php?title=Network.http.max-persistent-connections-per-server&diff=28784&oldid=28783

Celi
___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Oz Linden (Scott Lawrence)]

On 2012-07-29 13:58 , Sheet Spotter wrote:
> Does increasing the HTTP connection limit also increase the burden on the
> server and network?
>
> Increasing the HTTP connection limit on one client might improve the
> experience for one person. It might also diminish the experience for
> everyone else on the same server.

Exactly so.  The effect is more complex than just consuming file 
descriptors on the server side, as well.  TCP congestion control works 
far better when dealing with a smaller number of connections. Any 
packets that are not subject to congestion control that are sharing the 
same network path as a TCP connection increase the likelihood of a 
dropped packet in the connection flow, which causes the connection to 
immediately cut its transmission window by half. The viewer already puts 
a lot of UDP traffic into the same path. The handshake packets that open 
and close TCP connections are also not subject to the congestion control 
- which means that opening each new connection increases the chances 
that each of your existing connections will slow down by half.  Opening 
many connections also puts additional strain on routers in the path that 
are doing connection tracking (which too many do), adding another 
possible source of problems.

In general, when persistent connections and pipelined requests (having 
more than one request in flight on a connection at a time) are used, 
HTTP performs far better with a small number of connections than with 
more.  As we begin deploying support for this way of using it on the 
server side, we'll be able to do good experiments to determine what the 
right tradeoffs are for network behavior and server load to provide 
optimal experience for users.


___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges

Re: [opensource-dev] New HTTP Library & Project Viewer

[Oz Linden (Scott Lawrence)]

On 2012-07-29 04:07 , Henri Beauchamp wrote:

With just a few tweakings to the texture fetcher (and in particular
a setting that permits up to 32 concurrent HTTP fetches),


Caution... once we begin supporting persistent connections and pipelined 
requests on the server side (we're working on it), opening that many 
connections will very likely be considered abuse of the service.  
Allowing a single server to have that many potentially long-lived 
connections to the same server would starve the file descriptor pool 
very quickly at the server end, adversely affecting other users.


HTTP itself recommends that clients not maintain more than 2 connections 
to the same service [1].  I don't know exactly what limit we will decide 
is reasonable (I expect that 2 will be ok, but don't know whether or not 
some larger number will be also).


Please bear this in mind as you think about your designs.  I will keep 
you all informed as things develop.



[1] RFC 2616, section 8.1.4 



___
Policies and (un)subscribe information available here:
http://wiki.secondlife.com/wiki/OpenSource-Dev
Please read the policies before posting to keep unmoderated posting privileges