Okay, I've turned the setting on with git master 352002f. Let's see how it
goes.
I prefer a ramp up because of the massive packet loss we always saw at the
start of a connection.
I'm somewhat surprised that the client itself doesn't do this throttling - I thought that's why it was sending throttle
requests and adjustments in the first place.
On 12/10/11 19:08, Mic Bowman wrote:
The adaptive throttle is basically a slow start mechanism. It starts with a low
throttle rate & increases the rate until
packets start to be dropped. At that point, it backs off. TCP slow start grows
very slowly and cuts the rate very
rapidly. The current configuration in the adaptive throttle starts a little faster
& doesn't back off quite so much. I
think you can use show pqueues and show throttles to get a feel for how quickly
it adapts.
The biggest issue I had from people who tested it (while it is experimental its
been the default on scisim since the
code went in originally) was the comment "hey, my network settings are 2Mbps but I'm
only seeing 500kbps"... never mind
that the reason is that their network connection couldn't handle 2Mbps... so...
I'm a bit leary of the perception.
One possible way to address that concern is to start with the adaptive throttle
fully open & let it shrink back rather
than start slowly and grow. I'm a bit concerned that once you get into a bad
state, you'll have a hard time getting out
& the initial scene download puts a lot of pressure on the network connection.
Either way... I'm good with trying out
some changes in the configuration.
--mic
On Wed, Oct 12, 2011 at 10:57 AM, Justin Clark-Casey <[email protected]
<mailto:[email protected]>> wrote:
You're right - the mass resend is consistently overloading the connection.
Logging client acks shows that the
resent packet bursts are not acked. Raising the default RTO only delays
the resend since the client doesn't send
acks no matter how long the wait.
Setting enable_adaptive_throttles = true makes the problem go away for me -
the client now receives the object
update packets at a rate that it can cope with. Is there really any reason
not to turn this on by default? I hear
what you're saying about potential issues but it seems far better to get
real world data to iron them out than
persist with our current naive implementation. At the worst we can always
just disable it again.
Lowering the bw in the client settings also works, though it still allows a
noticeable overload at the beginning of
the client connection. I'm very surprised that the throttling isn't being
properly adjusted automatically, or is
this what enable_adaptive_throttles does (I traced it into the
AdaptiveTokenBucket but no further yet)?
On 12/10/11 16:32, Mic Bowman wrote:
Couple things...
First... here's the link to the thread on some of the original
issues... not sure if this went in to dan's
retransmission fix but i'll get to that in a minute:
https://lists.berlios.de/__pipermail/opensim-dev/2011-__March/010029.html
<https://lists.berlios.de/pipermail/opensim-dev/2011-March/010029.html>
Second... In spite of all the questions/comments below that this is the
"expected" behavior... there may be a
bug & we
need more data to find it.
Question... do you have the adaptive throttling turned on? Look for
enable_adaptive_throttles. It is turned off by
default. That is the switch that is designed to fix the problem you are
seeing... but the code has some other
behaviors
which are correct but... "different" than what most admins expect (the
throttle mimics the TCP slow start to
identify
and avoid congestion though not quite as conservatively as the TCP
algorithm).
One explanation for what you are seeing is that the network simply
cannot send as many packets as you are
asking. This
is frequently true for long-haul connections. When we did our analysis, we saw a
"cliff". Once you pass the
point where
the number of outgoing packets exceeds the physical networks ability to
deliver them ACKs are not processed in
time and
very bad things happen. That is... enough ACKs are coming late enough
so that the resend packet is already in
the queue.
The current version of the code handles that situation far better than
the old version (the current version is
bound by
the number of dynamic objects... the old version would hold *every*
update to that object... even ones that were
superceded by later updates).
The only real solution to this problem is to stop sending updates so
fast or to be much more conservative about
resends.
The first can be addressed one of three ways... First, configure the
simulator so that the outgoing bw per
connection is
throttled much lower. Second, turn down the BW setting on the viewer.
Third, turn on the adaptive throttling
code in the
simulator.
Being conservative about resends has its own share of problems...
Basically what this means is that you wait
longer to
resend a packet. However... if you wait too long to retransmit, then a
portion of your scene is not updating. The
algorithm for determining when to retransmit is based on a commonly
used algorithm described in one of the RFCs (the
actual RFC is referenced in a comment in the opensim code).
As to the UDP packet vs going back through the SceneEntityUpdate
queue... well that one is a no brainer... If
you use
the UDP packet for retranmission you are almost always sending useless
old data and just filling up the network with
fluff. Outside of the initial scene load, you see streams of updates
for objects more often than a single
update. If you
retransmit the UDP packet, you are almost always sending old data.
Going through the sceneentityupdate queue
basically
says "retransmit the current state of the object".
--mic
On Wed, Oct 12, 2011 at 5:56 AM, Justin Clark-Casey <[email protected]
<mailto:[email protected]>
<mailto:jjustincc@googlemail.__com <mailto:[email protected]>>>
wrote:
Yes, I may not have the correct cause in my original e-mail but I'm
certainly seeing a lot of resends that never
appear to stop. The effect is not consistent - sometimes I can log
in to the remote sim and there will be no
continuous resend. But most of the time a large proportion of
updates appear to be continuously resent.
I'm surprised no-one else has reported it though I did see a lot of
continuous updating on Wright Plaza @
osgrid.org <http://osgrid.org>
<http://osgrid.org> last night, which I don't think is related to
changes made by scripts. Perhaps it's only really
noticeable if you start looking, have visual object updates
development/advanced settings on in a viewer or are
wondering why viewer received packets keeps spiking.
As far as I can see from LLClientView/LLUDPServer, both prim and
property updates resends are happening but
I need
to study the code more closely and possibly old e-mails.
I hope to do some more analysis later on today and be around on IRC.
On 12/10/11 09:20, Lake, Dan wrote:
Thanks for the additional info on this, Melanie. You are
correct that the resend should get a new sequence
number and that old updates for an object will no longer get
sent when there is a newer update to send.
Properties and updates are handled almost the same but
different code. I'll look it over with Mic in the
morning
so I understand what is going on.
~Dan
-----Original Message-----
From: opensim-dev-bounces@lists.__be__rlios.de
<http://berlios.de>
<mailto:opensim-dev-bounces@__lists.berlios.de
<mailto:[email protected]>>
[mailto:opensim-dev-bounces@
<mailto:opensim-dev-bounces@>____lists.berlios.de <http://lists.berlios.de>
<mailto:opensim-dev-bounces@__lists.berlios.de
<mailto:[email protected]>>] On Behalf Of Melanie
Sent: Tuesday, October 11, 2011 11:22 PM
To: [email protected]
<mailto:[email protected]>
<mailto:opensim-dev@lists.__berlios.de
<mailto:[email protected]>>
Subject: Re: [Opensim-dev] Continual object update resends if
acks are missed
It appears this mechanism has already been removed for prim
updates
because intermediate updates are needed fro proper motion of
physical prims, but was left in for property updates. However,
culling of later updates as well as preservation of sequence
numbers
was not done.
Melanie
On 12/10/2011 07:43, Mic Bowman wrote:
We should collect more information on what is actually
happening. The old
behavior was to resend infinitely (and very badly). It
would send old
updates even when there were new updates for the same
object. We changed the
resend to transmit the current data rather than old data.
If there are still
no acks coming back after some time then we need to figure
out what the
viewer is expecting& when it no longer acks packets.
There is a write up on the procedure in an old email
message on this list.
--mic
On Tue, Oct 11, 2011 at 3:55 PM, Justin Clark-Casey<
[email protected] <mailto:[email protected]>
<mailto:jjustincc@googlemail.__com
<mailto:[email protected]>>> wrote:
Hi dslake (since this is chiefly addressed to you :)
In commit b5ab33b5 back on Wednesday 20th April 2011,
you made a change so
that the resend of object updates or property updates
is threaded back
through Resend methods on LLClientView rather than the
normal resend within
UDPServer.
The normal resend only performs the resend once, while
going through the
LLClientView.Resent*() methods will continually attempt
the resend until
acked as they put the requests back on the
m_entityUpdates/m_entityProps
queue.
From my experience, often the client will not reack
such packets. This
means that they are continuously resent until the
client logs out. I can
see this happening by uncommenting the log lines in
LLClientView.**____ResendPrimUpdates()
and ResentPropertyUpdates(). This is chiefly seen on
remote regions (I
can't repro on a localhost).
What do you think? Can we resend such packets just
once?
Thanks,
--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
__________________________________**_________________
Opensim-dev mailing list
[email protected] <mailto:[email protected]>
<mailto:Opensim-dev@lists.__berlios.de
<mailto:[email protected]>>
https://lists.berlios.de/**____mailman/listinfo/opensim-dev
<https://lists.berlios.de/**__mailman/listinfo/opensim-dev>
<https://lists.berlios.de/**__mailman/listinfo/opensim-dev
<https://lists.berlios.de/**mailman/listinfo/opensim-dev>><__h__ttps://lists.berlios.de/____mailman/listinfo/opensim-dev
<http://lists.berlios.de/__mailman/listinfo/opensim-dev>
<https://lists.berlios.de/__mailman/listinfo/opensim-dev
<https://lists.berlios.de/mailman/listinfo/opensim-dev>>>
___________________________________________________
Opensim-dev mailing list
[email protected] <mailto:[email protected]>
<mailto:Opensim-dev@lists.__berlios.de
<mailto:[email protected]>>
https://lists.berlios.de/____mailman/listinfo/opensim-dev
<https://lists.berlios.de/__mailman/listinfo/opensim-dev>
<https://lists.berlios.de/__mailman/listinfo/opensim-dev
<https://lists.berlios.de/mailman/listinfo/opensim-dev>>
___________________________________________________
Opensim-dev mailing list
[email protected] <mailto:[email protected]>
<mailto:Opensim-dev@lists.__berlios.de
<mailto:[email protected]>>
https://lists.berlios.de/____mailman/listinfo/opensim-dev
<https://lists.berlios.de/__mailman/listinfo/opensim-dev>
<https://lists.berlios.de/__mailman/listinfo/opensim-dev
<https://lists.berlios.de/mailman/listinfo/opensim-dev>>
___________________________________________________
Opensim-dev mailing list
[email protected] <mailto:[email protected]>
<mailto:Opensim-dev@lists.__berlios.de
<mailto:[email protected]>>
https://lists.berlios.de/____mailman/listinfo/opensim-dev
<https://lists.berlios.de/__mailman/listinfo/opensim-dev>
<https://lists.berlios.de/__mailman/listinfo/opensim-dev
<https://lists.berlios.de/mailman/listinfo/opensim-dev>>
--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
___________________________________________________
Opensim-dev mailing list
[email protected] <mailto:[email protected]>
<mailto:Opensim-dev@lists.__berlios.de
<mailto:[email protected]>>
https://lists.berlios.de/____mailman/listinfo/opensim-dev
<https://lists.berlios.de/__mailman/listinfo/opensim-dev>
<https://lists.berlios.de/__mailman/listinfo/opensim-dev
<https://lists.berlios.de/mailman/listinfo/opensim-dev>>
_________________________________________________
Opensim-dev mailing list
[email protected] <mailto:[email protected]>
https://lists.berlios.de/__mailman/listinfo/opensim-dev
<https://lists.berlios.de/mailman/listinfo/opensim-dev>
--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
_________________________________________________
Opensim-dev mailing list
[email protected] <mailto:[email protected]>
https://lists.berlios.de/__mailman/listinfo/opensim-dev
<https://lists.berlios.de/mailman/listinfo/opensim-dev>
_______________________________________________
Opensim-dev mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-dev
--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
_______________________________________________
Opensim-dev mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/opensim-dev
