* Matthew Toseland <toad at amphibian.dyndns.org> [2008-07-10 16:55:23]:
> On Thursday 10 July 2008 10:44, Florent Daigni?re wrote:
> > * Matthew Toseland <toad at amphibian.dyndns.org> [2008-07-07 12:17:33]:
> > > 7. Automatic bandwidth limit calibration. (Toad)
> > >
> > > Several other p2p apps implement this, we should too. Bandwidth is *the*
> > > scarce resource most of the time, we want to use as much of it as we can
> > > without significantly slowing down the user's internet connection (see
> > > above).
> >
> > I don't think that such a thing can reliably work. It might work in 80%
> > of the cases but will badly screw up in others.
>
> It works for other p2p's. What specifically is the problem for Freenet? Small
> number of connections?
Small number of connections *and* usage of UDP! Do you know any p2p
protocol which uses mainly UDP and does what you call "automatic
bandwidth limit calibration"?
E2k uses TCP, bittorrent uses TCP... As far as I know, only their links
to DHTs use UDP (Kademilia); they don't use it for data transfert.
> >
> > > TO IMPLEMENT IN 0.9:
> > >
> > > 1. Streams (low level optimisation)
> > >
> > > At the moment, Freenet is quite inefficient in bandwidth usage: the
> > > payload percentage, especially on nodes with low bandwidth limits but
> > > also on nodes with high bandwidth limits, can be quite low, 70%, maybe
> > > 50%... this is unacceptable. A lot of this is due to message padding. If
> > > we split block transfers (and large messages) into streams which can be
> > > sliced up byte-wise however big we want them, rather than into 1K blocks,
> > > then we will need much less padding and so acheive a higher payload
> > > percentage. This also has significant side benefits for steganography and
> > > for connections with low MTU, since we can (on a link with live streams)
> > > construct useful packets of almost any size.
> >
> > I am not convinced that I understand the logic here... I can't believe
> > you guys plan on implementing (7) without that.
>
> How is it related?
> >
> > In case you didn't suspect it: here is a big news:
> > "available and effective throughput depend on packet sizes amongst other
> > things". At the moment we have oversized packets with a per-link MTU...
> > which is poorly guessed and never adapted.
>
> And how exactly do you propose we figure out the MTU from Java? More JNI? :(
We can do statistics on each link and detect for which MTU value the
efficiency is maximum... but that requires to be able to have arbitrary
long data packets ;)
Doing it the other way around (doing statistics to determine the
send-rate with a fixed, arbitrary MTU) is just stupid.
Most of the QoS schemes on UDP are using the following criteria:
- size of packets (we can't act on that until we have streams)
- send rate -frequency- (if we don't act on size of packets, it
affects the throughput if we change it)
- port number used (we can't act on that until we have
"transport plugins")
Keep in mind that UDP is stateless; most of the time QoS is used to
prevent bandwidth to be monopolized by one low-priority application...
QoS doesn't work well on UDP flows because it's stateless! Hence it's
either under-prioritized or over-prioritized (people don't want their
dns requests to take ages). Most of the time there is QoS happening on
the ISP's side as well (they tend to prioritize ICMP because gamers are
interested in round-trip time and most of them don't suspect that they
are different policies in between ICMP and UDP -most games are using
UDP-).
My point is: if you want auto-bandwidth calibration to work and to be
effective, you *will* have to adapt the per-link MTU and as far as I
know that can be done only dynamically, using a stream based transport.
On the other hand if the only plan is to increase the bandwidth usage
(and not the actual throughput)... it will work ;)
> >
> > > DATA COLLECTION:
> > >
> > > 1. Push/pull tests.
> > >
> > > We should do push/pull tests regularly on the real network. On darknet
> this
> > > won't work well because of the difficulty of getting somebody you're not
> > > near to to fetch your data, although this is IMHO possible with FMS etc
> > > volunteers. On opennet we can simply select a different pair of peers
> every
> > > time to avoid the two nodes converging on each other as they are reused.
> >
> > What's the point here? Getting some data from the real network to run
> > the simulations? Testing availability of data, paths length, end
> > user experience?
> >
> > I haven't found the old-fashioned threaded-probe-requests you were
> > planning on implementing in your sum up; isn't it something we could use
> > here?
>
> They were implemented some time ago. They don't tell us anything about data
> reachability.
Okay, I've mis-formulated the question: what do you plan to use that
data for?
Are you trying to improve the data reachability itself?
Do you want to measure which effects "switching to a random-replacement
policy" will have on it?
Or do you plan to do it on a regular basis and see how it evolves with
the network growth and code changes?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20080711/6367a39c/attachment.pgp>
