[freenet-dev] Putting stuff in manifests sucks
I have introduced an easily comparable version number for plugins via the FredPluginRealVersioned. I have added this to all the default plugins, the few plugins that I missed are IMHO not used much (e.g. Librarian), and may not be versioned at all (ditto). We have always assumed that this would be compiled in to the jar so we can check it before we deploy the jar. However, THIS SUCKS: - It is not cross platform, unless we call java code from ant to determine it and stick it in a variable. - If we don't, it relies on unpublished build scripts, which devs developing the plugins will not use. We need the plugin to be versioned even if it is an unofficial build! - If we do, we have to protect the Version.java from unauthorized commits. Bombe gives the example: For the time being I am just going to load the plugin. If it fails to load, it probably isn't compatible. If it loads and has an incompatible version, we can unload it without starting it. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090409/b50935b5/attachment.pgp>
[freenet-dev] [freenet-cvs] r26638 - trunk/freenet/test/freenet/support
On Thu, Apr 9, 2009 at 8:57 PM, Matthew Toseland wrote: > On Thursday 09 April 2009 11:27:04 Daniel Cheng wrote: >> On Thu, Apr 9, 2009 at 6:12 AM, Matthew Toseland >> wrote: >> > On Wednesday 08 April 2009 16:11:31 Daniel Cheng wrote: >> >> On Wed, Apr 8, 2009 at 8:26 PM, Matthew Toseland >> >> wrote: >> >> > On Wednesday 08 April 2009 06:04:48 j16sdiz at freenetproject.org wrote: >> >> >> Author: j16sdiz >> >> >> Date: 2009-04-08 05:04:47 + (Wed, 08 Apr 2009) >> >> >> New Revision: 26638 >> >> >> >> >> >> Added: >> >> >>> ? ?trunk/freenet/test/freenet/support/PrioritizedSerialExecutorTest.java >> >> >> Log: >> >> >> JUnit for PrioritizedSerialExecutor >> >> > >> >> > Race conditions??? >> >> >> >> The test case using locks and SynchronousQueue to make sure >> >> the test case are run in the predictable order. >> > >> > And at the expected rate? >> > >> >> not :) >> i have tried my best to make this work in all machines i have access to. > > How about barriers? Don't let it finish a job until the test case tells it it > can? SynchronousQueue is doing that internally. The problem is not some task finish too fast, it's some task not started when I check for it. (the executor is just sleeping), I have added some Thread.sleep(10) to workaround this, but this is not reliable.
[freenet-dev] [freenet-cvs] r26635 - trunk/freenet/src/freenet/io/comm
On Thu, Apr 9, 2009 at 6:13 AM, Matthew Toseland wrote: > On Wednesday 08 April 2009 16:08:58 Daniel Cheng wrote: >> On Wed, Apr 8, 2009 at 11:05 PM, Daniel Cheng > wrote: >> > On Wed, Apr 8, 2009 at 8:23 PM, Matthew Toseland >> > wrote: >> >> On Wednesday 08 April 2009 06:03:34 j16sdiz at freenetproject.org wrote: >> >>> Author: j16sdiz >> >>> Date: 2009-04-08 05:03:33 + (Wed, 08 Apr 2009) >> >>> New Revision: 26635 >> >>> >> >>> Modified: >> >>> ? ?trunk/freenet/src/freenet/io/comm/UdpSocketHandler.java >> >>> Log: >> >>> Reduce SocketTimeOutException >> >>> >> >>> SocketTimeOutException was causing the thread wake up >> >>> every 1 second and creating (slow) Exceptions. >> >>> >> >>> This patch, however, effectivly broke the Hang Checker, >> >>> hence must be applied after the "Remove UDPSocket Hang >> >>> Checker" patch. >> >> >> >> As I understand it, waiting for a packet cannot be interrupted. Which > means >> > >> > It can interrupted _ONLY IF_ you call close(). >> > That's why I have r25533. >> >> Oops... I think I have handled that. but i didn't >> This is fixed at r26654 >> >> >> that after this patch is applied, shutting down opennet and then starting > it >> >> up again within 2 minutes will break with a binding error. Correct? >> >> fixed > > Tested? > tested, disable and re-enable opennet works.
[freenet-dev] [freenet-cvs] r26638 - trunk/freenet/test/freenet/support
On Thu, Apr 9, 2009 at 6:12 AM, Matthew Toseland wrote: > On Wednesday 08 April 2009 16:11:31 Daniel Cheng wrote: >> On Wed, Apr 8, 2009 at 8:26 PM, Matthew Toseland >> wrote: >> > On Wednesday 08 April 2009 06:04:48 j16sdiz at freenetproject.org wrote: >> >> Author: j16sdiz >> >> Date: 2009-04-08 05:04:47 + (Wed, 08 Apr 2009) >> >> New Revision: 26638 >> >> >> >> Added: >> >> ? ?trunk/freenet/test/freenet/support/PrioritizedSerialExecutorTest.java >> >> Log: >> >> JUnit for PrioritizedSerialExecutor >> > >> > Race conditions??? >> >> The test case using locks and SynchronousQueue to make sure >> the test case are run in the predictable order. > > And at the expected rate? > not :) i have tried my best to make this work in all machines i have access to.
[freenet-dev] [freenet-cvs] r26609 - trunk/freenet/src/freenet/crypt
On Thu, Apr 9, 2009 at 6:14 AM, Matthew Toseland
wrote:
> On Wednesday 08 April 2009 01:28:41 Daniel Cheng wrote:
>> On Wed, Apr 8, 2009 at 4:02 AM, Matthew Toseland
>> wrote:
>> > On Tuesday 07 April 2009 16:26:42 j16sdiz at freenetproject.org wrote:
>> >> Author: j16sdiz
>> >> Date: 2009-04-07 15:26:41 + (Tue, 07 Apr 2009)
>> >> New Revision: 26609
>> >>
>> >> Modified:
>> >> ? ?trunk/freenet/src/freenet/crypt/Yarrow.java
>> >> Log:
>> >> Use int[] instead of Integer, save autoboxing and put()
>> >>
>> >> Modified: trunk/freenet/src/freenet/crypt/Yarrow.java
>> >> ===
>> >> --- trunk/freenet/src/freenet/crypt/Yarrow.java ? ? ? 2009-04-07 15:06:39
> UTC (rev
>> > 26608)
>> >> +++ trunk/freenet/src/freenet/crypt/Yarrow.java ? ? ? 2009-04-07 15:26:41
> UTC (rev
>> > 26609)
>> >> @@ -444,12 +444,12 @@
>> >> ? ? ? private MessageDigest fast_pool, ?slow_pool;
>> >> ? ? ? private int fast_entropy, ?slow_entropy;
>> >> ? ? ? private boolean fast_select;
>> >> - ? ? private Map entropySeen;
>> >> + ? ? private Map entropySeen;
>> >>
>> >> ? ? ? private void accumulator_init(String digest) throws
>> > NoSuchAlgorithmException {
>> >> ? ? ? ? ? ? ? fast_pool = MessageDigest.getInstance(digest);
>> >> ? ? ? ? ? ? ? slow_pool = MessageDigest.getInstance(digest);
>> >> - ? ? ? ? ? ? entropySeen = new HashMap();
>> >> + ? ? ? ? ? ? entropySeen = new HashMap();
>> >> ? ? ? }
>> >>
>> >> ? ? ? @Override
>> >> @@ -515,21 +515,21 @@
>> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? slow_entropy += actualEntropy;
>> >>
>> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if(source != null) {
>> >> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Integer contributedEntropy =
> entropySeen.get(source);
>> >> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if(contributedEntropy == null)
>> >> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? contributedEntropy =
> Integer.valueOf(actualEntropy);
>> >> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? else
>> >> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? contributedEntropy =
> Integer.valueOf(actualEntropy +
>> > contributedEntropy.intValue());
>> >> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? entropySeen.put(source,
> contributedEntropy);
>> >> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? int[] contributedEntropy =
> entropySeen.get(source);
>> >> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if(contributedEntropy == null) {
>> >> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? contributedEntropy = new
> int[] { actualEntropy };
>> >> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? entropySeen.put(source,
> contributedEntropy);
>> >> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? } else
>> >> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? contributedEntropy[0]++;
>> >
>> > Wrong.
>> > contributedEntropy[0] += actualEntropy;
>> > Please do not make mistakes in crypt/!
>> > Otherwise it's a neat optimisation, although using a MutableInt would be
>> > cleaner.
>>
>> fixed in r26629.
>>
>> (this is in the reseed from entropy source code path.)
>> btw, the current code maybe reseeding too infrequent.
>>
>> This is the current reseed logic:
>> ? ?(1) we have more then ?(SLOW_THRESHOLD * 2) bits in slow_entropy
>> ?_AND_
>> ? ?(2) those entropy are from TWO DIFFERENT source, each contribute more
> then
>> ? ? ? ? ? THRESHOLD bits
>>
>> However, we have ONLY TWO sources,
>> the TWO DIFFERENT sources requirement is too hard to meet.
>
> Yeah. And one of the sources (incoming packet hashes) always reports 0, as it
> may be observable/influencible. Ideas?
>
> BTW, what are the two sources?
>
FNPPacketMangler ,
private final EntropySource fnpTimingSource;
private final EntropySource myPacketDataSource;
fnpTimeingSource is the packet arrivial time
myPacketDataSource is the hash of packets.
[freenet-dev] Putting stuff in manifests sucks
I have introduced an easily comparable version number for plugins via the FredPluginRealVersioned. I have added this to all the default plugins, the few plugins that I missed are IMHO not used much (e.g. Librarian), and may not be versioned at all (ditto). We have always assumed that this would be compiled in to the jar so we can check it before we deploy the jar. However, THIS SUCKS: - It is not cross platform, unless we call java code from ant to determine it and stick it in a variable. - If we don't, it relies on unpublished build scripts, which devs developing the plugins will not use. We need the plugin to be versioned even if it is an unofficial build! - If we do, we have to protect the Version.java from unauthorized commits. Bombe gives the example: For the time being I am just going to load the plugin. If it fails to load, it probably isn't compatible. If it loads and has an incompatible version, we can unload it without starting it. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] [freenet-cvs] r26638 - trunk/freenet/test/freenet/support
On Thursday 09 April 2009 11:27:04 Daniel Cheng wrote: > On Thu, Apr 9, 2009 at 6:12 AM, Matthew Toseland > wrote: > > On Wednesday 08 April 2009 16:11:31 Daniel Cheng wrote: > >> On Wed, Apr 8, 2009 at 8:26 PM, Matthew Toseland > >> wrote: > >> > On Wednesday 08 April 2009 06:04:48 j16sdiz at freenetproject.org wrote: > >> >> Author: j16sdiz > >> >> Date: 2009-04-08 05:04:47 + (Wed, 08 Apr 2009) > >> >> New Revision: 26638 > >> >> > >> >> Added: > >> >> ? ?trunk/freenet/test/freenet/support/PrioritizedSerialExecutorTest.java > >> >> Log: > >> >> JUnit for PrioritizedSerialExecutor > >> > > >> > Race conditions??? > >> > >> The test case using locks and SynchronousQueue to make sure > >> the test case are run in the predictable order. > > > > And at the expected rate? > > > > not :) > i have tried my best to make this work in all machines i have access to. How about barriers? Don't let it finish a job until the test case tells it it can? -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090409/be83b6df/attachment.pgp>
[freenet-dev] Client apps limiting load in bad ways
On Thursday 09 April 2009 10:55:10 xor wrote: > > > -Original Message- > > From: devl-bounces at freenetproject.org > > [mailto:devl-bounces at freenetproject.org] On Behalf Of Matthew Toseland > > Sent: Thursday, April 09, 2009 12:28 AM > > To: Discussion of development issues > > Subject: [freenet-dev] Client apps limiting load in bad ways > > > > FMS does bad things to limit load: Rather than queueing every > > key at MaxRetries=-1, it does all the scheduling itself, > > Freetalk and WoT can do -1 because we/I can patch them right now. > This can be done for any SSK/CHK URIs which the plugins know for sure that > they exist, i.e. for messages and message lists which they obtained the URIs > of through other peoples message lists... This is really for polling. > > However Freetalk and WoT will have to periodically try to download message > lists and trust lists of ALL identities to ensure that nothing is missed if > the propagation through third party lists does not work. Yeah... you may not want to fetch 3 times every half hour for the medium-trusted identities, but you do want to be notified of messages. Hence DontFetch=true for these identities, which means that after you try occasionally to fetch them, if they are found nearby they will be offered. > > > The reason that FMS's behaviour is a problem is that ULPRs > > rely on nodes knowing which keys they are interested in. > > ULPRs are the main network level optimisation for SSK > > polling/chat/WoT apps, so this is potentially a big source of > > problems - excessive load, slow propagation of messages etc. > > > > Possible solutions: > > > > DontFetch=true : if set, a request would be purely passive: > > no requests would be started as a result of that request, but > > the node would know it wants those blocks, and if the blocks > > do come in, the request would progress. > > The probability of such a request to succeed will be horribly low, won't it? Yes, under normal circumstances. It's a way to tell the node that we want the key, without necessarily requesting it regularly. If other nodes nearby are requesting it, we will get it, even during the periods when we are not actually fetching it. Really it's equivalent to MaxRetries=-1, but instead of having 3 fetches every half hour fetches are entirely up to the client (and other nodes). > > > > Client configurable cooldown: Running a request with > > MaxRetries=-1 means fetching it 3 times every half hour. We > > could make the half hour bit configurable, and maybe the 3 > > times too. > > Good idea IMHO, especially for FT/WoT. Note that ULPRs expire after 1 hour... -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090409/a860c22b/attachment.pgp>
[freenet-dev] Client apps limiting load in bad ways
On Thursday 09 April 2009 10:43:32 bo-le at web.de wrote:
> > -Urspr?ngliche Nachricht-
> > Von: "Matthew Toseland"
> > Gesendet: 09.04.09 00:28:48
> > An: Discussion of development issues
> > Betreff: [freenet-dev] Client apps limiting load in bad ways
>
> > FMS does bad things to limit load: Rather than queueing every key at
> > MaxRetries=-1, it does all the scheduling itself, doing simple ClientGet's
> > with no maxretries and no priority (hence prio 2), in order to enforce a
> > maximum number of parallel requests. Other client apps probably do similar
> > things. MiniChat presumably does the opposite, constantly requesting keys,
> > starting a new request for the same key when the last one has finished
> > (saces: is this true?).
> yes. the code 'emulates' a (passive)request with short timeout (a few
minutes) ;)
>
> while (currenttime() < timeout) {
> sendrequest();
> if sucess break;
> wait(500);
> }
>
> >
> > The reason that FMS's behaviour is a problem is that ULPRs rely on nodes
> > knowing which keys they are interested in. ULPRs are the main network
level
> > optimisation for SSK polling/chat/WoT apps, so this is potentially a big
> > source of problems - excessive load, slow propagation of messages etc.
> >
> > Possible solutions:
> >
> > DontFetch=true : if set, a request would be purely passive: no requests
would
> > be started as a result of that request, but the node would know it wants
> > those blocks, and if the blocks do come in, the request would progress.
> > Suggested usage is that a client app would set up DontFetch requests for
> > everything it is interested in, with Verbosity sufficiently high, and then
> > poll using whatever load heuristics they want. When a block (typically an
SSK
> > block when polling SSK outqueues) is found, either the request succeeds,
> > fatally fails, or it is a redirect. If it is a redirect, the client would
get
> > a SimpleProgress and would schedule a normal request to fetch the rest of
the
> > key. (This should be fairly easy to implement)
> >
> > Client configurable cooldown: Running a request with MaxRetries=-1 means
> > fetching it 3 times every half hour. We could make the half hour bit
> > configurable, and maybe the 3 times too. Note that it is quite likely that
we
> > will eventually impose network level limits on polling (e.g. using
> > RecentlyFailed), so setting this low will not necessarily (in the long
run)
> > yield better performance. (This should be moderately easy to implement)
> >
> > Any other ideas? Is this the right approach? IMHO we can implement
whatever
> > API fairly easily, but maybe there are some other ideas?
>
> If everybody is listen a key with DontFetch=True, the key/data will be known
along the regular insert chain,
> but how is this further spreaded?
Via ULPRs. Any node which has recently requested the key, or which we've
routed a request for it to, will be offered the key, and if it wants the key,
or in some cases if it has peers that want the key, it will fetch it, and
further propagate it. This means tracking sensitive information (who has
asked for and who we have sent requests to for a specific key) for 1 hour,
after that we delete it.
>
> MfG
> saces
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20090409/962720b9/attachment.pgp>
[freenet-dev] Client apps limiting load in bad ways
On Thursday 09 April 2009 09:22:08 VolodyA! V Anarhist wrote: > Matthew Toseland wrote: > > FMS does bad things to limit load: Rather than queueing every key at > > MaxRetries=-1, it does all the scheduling itself, doing simple ClientGet's > > with no maxretries and no priority (hence prio 2), in order to enforce a > > maximum number of parallel requests. Other client apps probably do similar > > things. MiniChat presumably does the opposite, constantly requesting keys, > > starting a new request for the same key when the last one has finished > > (saces: is this true?). > > > > The reason that FMS's behaviour is a problem is that ULPRs rely on nodes > > knowing which keys they are interested in. ULPRs are the main network level > > optimisation for SSK polling/chat/WoT apps, so this is potentially a big > > source of problems - excessive load, slow propagation of messages etc. > > > > Possible solutions: > > > > DontFetch=true : if set, a request would be purely passive: no requests would > > A minor point probably, but is it at all possible not to use negative there? It > sometimes make FCPv2 standard harder to follow: DontFailToNotFetch=false takes a > few extra seconds to figure out than Fetch=true Well, the normal behaviour is to fetch... -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090409/3b644767/attachment.pgp>
[freenet-dev] [freenet-cvs] r26557 -trunk/freenet/src/freenet/support
> -Original Message- > From: devl-bounces at freenetproject.org > [mailto:devl-bounces at freenetproject.org] On Behalf Of Matthew Toseland > Sent: Tuesday, April 07, 2009 9:28 PM > To: devl at freenetproject.org > Subject: Re: [freenet-dev] [freenet-cvs] r26557 > -trunk/freenet/src/freenet/support > > On Monday 06 April 2009 07:51:13 xor at freenetproject.org wrote: > > Author: xor > > Date: 2009-04-06 06:51:12 + (Mon, 06 Apr 2009) New > Revision: 26557 > > > > Modified: > >trunk/freenet/src/freenet/support/DoublyLinkedList.java > > Log: > > Add more comments to the FIXME. > > So can we un-deprecate it? Done.
[freenet-dev] Git vs hg: we must decide now!
On Wednesday 08 April 2009 13:28:28 Matthew Toseland wrote: > Assuming I haven't missed anyone and there are no disputes, clearly git has > it. I can haz a happy! David -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090409/6e67c166/attachment.pgp>
[freenet-dev] Client apps limiting load in bad ways
> -Original Message- > From: devl-bounces at freenetproject.org > [mailto:devl-bounces at freenetproject.org] On Behalf Of Matthew Toseland > Sent: Thursday, April 09, 2009 12:28 AM > To: Discussion of development issues > Subject: [freenet-dev] Client apps limiting load in bad ways > > FMS does bad things to limit load: Rather than queueing every > key at MaxRetries=-1, it does all the scheduling itself, Freetalk and WoT can do -1 because we/I can patch them right now. This can be done for any SSK/CHK URIs which the plugins know for sure that they exist, i.e. for messages and message lists which they obtained the URIs of through other peoples message lists... However Freetalk and WoT will have to periodically try to download message lists and trust lists of ALL identities to ensure that nothing is missed if the propagation through third party lists does not work. > The reason that FMS's behaviour is a problem is that ULPRs > rely on nodes knowing which keys they are interested in. > ULPRs are the main network level optimisation for SSK > polling/chat/WoT apps, so this is potentially a big source of > problems - excessive load, slow propagation of messages etc. > > Possible solutions: > > DontFetch=true : if set, a request would be purely passive: > no requests would be started as a result of that request, but > the node would know it wants those blocks, and if the blocks > do come in, the request would progress. The probability of such a request to succeed will be horribly low, won't it? > > Client configurable cooldown: Running a request with > MaxRetries=-1 means fetching it 3 times every half hour. We > could make the half hour bit configurable, and maybe the 3 > times too. Good idea IMHO, especially for FT/WoT.
[freenet-dev] Client apps limiting load in bad ways
> -Urspr?ngliche Nachricht-
> Von: "Matthew Toseland"
> Gesendet: 09.04.09 00:28:48
> An: Discussion of development issues
> Betreff: [freenet-dev] Client apps limiting load in bad ways
> FMS does bad things to limit load: Rather than queueing every key at
> MaxRetries=-1, it does all the scheduling itself, doing simple ClientGet's
> with no maxretries and no priority (hence prio 2), in order to enforce a
> maximum number of parallel requests. Other client apps probably do similar
> things. MiniChat presumably does the opposite, constantly requesting keys,
> starting a new request for the same key when the last one has finished
> (saces: is this true?).
yes. the code 'emulates' a (passive)request with short timeout (a few minutes)
;)
while (currenttime() < timeout) {
sendrequest();
if sucess break;
wait(500);
}
>
> The reason that FMS's behaviour is a problem is that ULPRs rely on nodes
> knowing which keys they are interested in. ULPRs are the main network level
> optimisation for SSK polling/chat/WoT apps, so this is potentially a big
> source of problems - excessive load, slow propagation of messages etc.
>
> Possible solutions:
>
> DontFetch=true : if set, a request would be purely passive: no requests would
> be started as a result of that request, but the node would know it wants
> those blocks, and if the blocks do come in, the request would progress.
> Suggested usage is that a client app would set up DontFetch requests for
> everything it is interested in, with Verbosity sufficiently high, and then
> poll using whatever load heuristics they want. When a block (typically an SSK
> block when polling SSK outqueues) is found, either the request succeeds,
> fatally fails, or it is a redirect. If it is a redirect, the client would get
> a SimpleProgress and would schedule a normal request to fetch the rest of the
> key. (This should be fairly easy to implement)
>
> Client configurable cooldown: Running a request with MaxRetries=-1 means
> fetching it 3 times every half hour. We could make the half hour bit
> configurable, and maybe the 3 times too. Note that it is quite likely that we
> will eventually impose network level limits on polling (e.g. using
> RecentlyFailed), so setting this low will not necessarily (in the long run)
> yield better performance. (This should be moderately easy to implement)
>
> Any other ideas? Is this the right approach? IMHO we can implement whatever
> API fairly easily, but maybe there are some other ideas?
If everybody is listen a key with DontFetch=True, the key/data will be known
along the regular insert chain,
but how is this further spreaded?
MfG
saces
__
Verschicken Sie SMS direkt vom Postfach aus - in alle deutschen und viele
ausl?ndische Netze zum gleichen Preis!
https://produkte.web.de/webde_sms/sms
[freenet-dev] git/hg hosting
On Wednesday 08 April 2009 16:22:03 guido wrote: > > Your pgp signature is charset corrupted. > The pgp signature is pure ASCII, how can it possibly be charset corrupted? Because there are encodings out there that don?t care about the lowest 7 bit being ASCII. Admittedly, they are far and in-between. David -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090409/41c9df12/attachment.pgp>
[freenet-dev] Client apps limiting load in bad ways
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Toseland wrote: > FMS does bad things to limit load: Rather than queueing every key at > MaxRetries=-1, it does all the scheduling itself, doing simple ClientGet's > with no maxretries and no priority (hence prio 2), in order to enforce a > maximum number of parallel requests. Other client apps probably do similar > things. MiniChat presumably does the opposite, constantly requesting keys, > starting a new request for the same key when the last one has finished > (saces: is this true?). > > The reason that FMS's behaviour is a problem is that ULPRs rely on nodes > knowing which keys they are interested in. ULPRs are the main network level > optimisation for SSK polling/chat/WoT apps, so this is potentially a big > source of problems - excessive load, slow propagation of messages etc. > > Possible solutions: > > DontFetch=true : if set, a request would be purely passive: no requests would A minor point probably, but is it at all possible not to use negative there? It sometimes make FCPv2 standard harder to follow: DontFailToNotFetch=false takes a few extra seconds to figure out than Fetch=true > be started as a result of that request, but the node would know it wants > those blocks, and if the blocks do come in, the request would progress. > Suggested usage is that a client app would set up DontFetch requests for > everything it is interested in, with Verbosity sufficiently high, and then > poll using whatever load heuristics they want. When a block (typically an SSK > block when polling SSK outqueues) is found, either the request succeeds, > fatally fails, or it is a redirect. If it is a redirect, the client would get > a SimpleProgress and would schedule a normal request to fetch the rest of the > key. (This should be fairly easy to implement) > > Client configurable cooldown: Running a request with MaxRetries=-1 means > fetching it 3 times every half hour. We could make the half hour bit > configurable, and maybe the 3 times too. Note that it is quite likely that we > will eventually impose network level limits on polling (e.g. using > RecentlyFailed), so setting this low will not necessarily (in the long run) > yield better performance. (This should be moderately easy to implement) > > Any other ideas? Is this the right approach? IMHO we can implement whatever > API fairly easily, but maybe there are some other ideas? > > > > > ___ > Devl mailing list > Devl at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl - -- http://freedom.libsyn.com/ Echo of Freedom, Radical Podcast http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal http://www.freedomporn.org/ Freedom Porn, anarchist and activist smut "None of us are free until all of us are free."~ Mihail Bakunin -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkndsCwACgkQuWy2EFICg+05NACfR91vYCW2StcpNFC/6FTlYrro j9cAoLqQ7apxP3XLP2CBOW8vYRciOkZx =dJXO -END PGP SIGNATURE-
Re: [freenet-dev] [freenet-cvs] r26638 - trunk/freenet/test/freenet/support
On Thu, Apr 9, 2009 at 8:57 PM, Matthew Toseland wrote: > On Thursday 09 April 2009 11:27:04 Daniel Cheng wrote: >> On Thu, Apr 9, 2009 at 6:12 AM, Matthew Toseland >> wrote: >> > On Wednesday 08 April 2009 16:11:31 Daniel Cheng wrote: >> >> On Wed, Apr 8, 2009 at 8:26 PM, Matthew Toseland >> >> wrote: >> >> > On Wednesday 08 April 2009 06:04:48 j16s...@freenetproject.org wrote: >> >> >> Author: j16sdiz >> >> >> Date: 2009-04-08 05:04:47 + (Wed, 08 Apr 2009) >> >> >> New Revision: 26638 >> >> >> >> >> >> Added: >> >> >>> trunk/freenet/test/freenet/support/PrioritizedSerialExecutorTest.java >> >> >> Log: >> >> >> JUnit for PrioritizedSerialExecutor >> >> > >> >> > Race conditions??? >> >> >> >> The test case using locks and SynchronousQueue to make sure >> >> the test case are run in the predictable order. >> > >> > And at the expected rate? >> > >> >> not :) >> i have tried my best to make this work in all machines i have access to. > > How about barriers? Don't let it finish a job until the test case tells it it > can? SynchronousQueue is doing that internally. The problem is not some task finish too fast, it's some task not started when I check for it. (the executor is just sleeping), I have added some Thread.sleep(10) to workaround this, but this is not reliable. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] [freenet-cvs] r26638 - trunk/freenet/test/freenet/support
On Thursday 09 April 2009 11:27:04 Daniel Cheng wrote: > On Thu, Apr 9, 2009 at 6:12 AM, Matthew Toseland > wrote: > > On Wednesday 08 April 2009 16:11:31 Daniel Cheng wrote: > >> On Wed, Apr 8, 2009 at 8:26 PM, Matthew Toseland > >> wrote: > >> > On Wednesday 08 April 2009 06:04:48 j16s...@freenetproject.org wrote: > >> >> Author: j16sdiz > >> >> Date: 2009-04-08 05:04:47 + (Wed, 08 Apr 2009) > >> >> New Revision: 26638 > >> >> > >> >> Added: > >> >> trunk/freenet/test/freenet/support/PrioritizedSerialExecutorTest.java > >> >> Log: > >> >> JUnit for PrioritizedSerialExecutor > >> > > >> > Race conditions??? > >> > >> The test case using locks and SynchronousQueue to make sure > >> the test case are run in the predictable order. > > > > And at the expected rate? > > > > not :) > i have tried my best to make this work in all machines i have access to. How about barriers? Don't let it finish a job until the test case tells it it can? signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Client apps limiting load in bad ways
On Thursday 09 April 2009 10:55:10 xor wrote: > > > -Original Message- > > From: devl-boun...@freenetproject.org > > [mailto:devl-boun...@freenetproject.org] On Behalf Of Matthew Toseland > > Sent: Thursday, April 09, 2009 12:28 AM > > To: Discussion of development issues > > Subject: [freenet-dev] Client apps limiting load in bad ways > > > > FMS does bad things to limit load: Rather than queueing every > > key at MaxRetries=-1, it does all the scheduling itself, > > Freetalk and WoT can do -1 because we/I can patch them right now. > This can be done for any SSK/CHK URIs which the plugins know for sure that > they exist, i.e. for messages and message lists which they obtained the URIs > of through other peoples message lists... This is really for polling. > > However Freetalk and WoT will have to periodically try to download message > lists and trust lists of ALL identities to ensure that nothing is missed if > the propagation through third party lists does not work. Yeah... you may not want to fetch 3 times every half hour for the medium-trusted identities, but you do want to be notified of messages. Hence DontFetch=true for these identities, which means that after you try occasionally to fetch them, if they are found nearby they will be offered. > > > The reason that FMS's behaviour is a problem is that ULPRs > > rely on nodes knowing which keys they are interested in. > > ULPRs are the main network level optimisation for SSK > > polling/chat/WoT apps, so this is potentially a big source of > > problems - excessive load, slow propagation of messages etc. > > > > Possible solutions: > > > > DontFetch=true : if set, a request would be purely passive: > > no requests would be started as a result of that request, but > > the node would know it wants those blocks, and if the blocks > > do come in, the request would progress. > > The probability of such a request to succeed will be horribly low, won't it? Yes, under normal circumstances. It's a way to tell the node that we want the key, without necessarily requesting it regularly. If other nodes nearby are requesting it, we will get it, even during the periods when we are not actually fetching it. Really it's equivalent to MaxRetries=-1, but instead of having 3 fetches every half hour fetches are entirely up to the client (and other nodes). > > > > Client configurable cooldown: Running a request with > > MaxRetries=-1 means fetching it 3 times every half hour. We > > could make the half hour bit configurable, and maybe the 3 > > times too. > > Good idea IMHO, especially for FT/WoT. Note that ULPRs expire after 1 hour... signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Client apps limiting load in bad ways
On Thursday 09 April 2009 10:43:32 bo...@web.de wrote:
> > -Ursprüngliche Nachricht-
> > Von: "Matthew Toseland"
> > Gesendet: 09.04.09 00:28:48
> > An: Discussion of development issues
> > Betreff: [freenet-dev] Client apps limiting load in bad ways
>
> > FMS does bad things to limit load: Rather than queueing every key at
> > MaxRetries=-1, it does all the scheduling itself, doing simple ClientGet's
> > with no maxretries and no priority (hence prio 2), in order to enforce a
> > maximum number of parallel requests. Other client apps probably do similar
> > things. MiniChat presumably does the opposite, constantly requesting keys,
> > starting a new request for the same key when the last one has finished
> > (saces: is this true?).
> yes. the code 'emulates' a (passive)request with short timeout (a few
minutes) ;)
>
> while (currenttime() < timeout) {
> sendrequest();
> if sucess break;
> wait(500);
> }
>
> >
> > The reason that FMS's behaviour is a problem is that ULPRs rely on nodes
> > knowing which keys they are interested in. ULPRs are the main network
level
> > optimisation for SSK polling/chat/WoT apps, so this is potentially a big
> > source of problems - excessive load, slow propagation of messages etc.
> >
> > Possible solutions:
> >
> > DontFetch=true : if set, a request would be purely passive: no requests
would
> > be started as a result of that request, but the node would know it wants
> > those blocks, and if the blocks do come in, the request would progress.
> > Suggested usage is that a client app would set up DontFetch requests for
> > everything it is interested in, with Verbosity sufficiently high, and then
> > poll using whatever load heuristics they want. When a block (typically an
SSK
> > block when polling SSK outqueues) is found, either the request succeeds,
> > fatally fails, or it is a redirect. If it is a redirect, the client would
get
> > a SimpleProgress and would schedule a normal request to fetch the rest of
the
> > key. (This should be fairly easy to implement)
> >
> > Client configurable cooldown: Running a request with MaxRetries=-1 means
> > fetching it 3 times every half hour. We could make the half hour bit
> > configurable, and maybe the 3 times too. Note that it is quite likely that
we
> > will eventually impose network level limits on polling (e.g. using
> > RecentlyFailed), so setting this low will not necessarily (in the long
run)
> > yield better performance. (This should be moderately easy to implement)
> >
> > Any other ideas? Is this the right approach? IMHO we can implement
whatever
> > API fairly easily, but maybe there are some other ideas?
>
> If everybody is listen a key with DontFetch=True, the key/data will be known
along the regular insert chain,
> but how is this further spreaded?
Via ULPRs. Any node which has recently requested the key, or which we've
routed a request for it to, will be offered the key, and if it wants the key,
or in some cases if it has peers that want the key, it will fetch it, and
further propagate it. This means tracking sensitive information (who has
asked for and who we have sent requests to for a specific key) for 1 hour,
after that we delete it.
>
> MfG
> saces
signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Client apps limiting load in bad ways
On Thursday 09 April 2009 09:22:08 VolodyA! V Anarhist wrote: > Matthew Toseland wrote: > > FMS does bad things to limit load: Rather than queueing every key at > > MaxRetries=-1, it does all the scheduling itself, doing simple ClientGet's > > with no maxretries and no priority (hence prio 2), in order to enforce a > > maximum number of parallel requests. Other client apps probably do similar > > things. MiniChat presumably does the opposite, constantly requesting keys, > > starting a new request for the same key when the last one has finished > > (saces: is this true?). > > > > The reason that FMS's behaviour is a problem is that ULPRs rely on nodes > > knowing which keys they are interested in. ULPRs are the main network level > > optimisation for SSK polling/chat/WoT apps, so this is potentially a big > > source of problems - excessive load, slow propagation of messages etc. > > > > Possible solutions: > > > > DontFetch=true : if set, a request would be purely passive: no requests would > > A minor point probably, but is it at all possible not to use negative there? It > sometimes make FCPv2 standard harder to follow: DontFailToNotFetch=false takes a > few extra seconds to figure out than Fetch=true Well, the normal behaviour is to fetch... signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] [freenet-cvs] r26557 -trunk/freenet/src/freenet/support
> -Original Message- > From: devl-boun...@freenetproject.org > [mailto:devl-boun...@freenetproject.org] On Behalf Of Matthew Toseland > Sent: Tuesday, April 07, 2009 9:28 PM > To: devl@freenetproject.org > Subject: Re: [freenet-dev] [freenet-cvs] r26557 > -trunk/freenet/src/freenet/support > > On Monday 06 April 2009 07:51:13 x...@freenetproject.org wrote: > > Author: xor > > Date: 2009-04-06 06:51:12 + (Mon, 06 Apr 2009) New > Revision: 26557 > > > > Modified: > >trunk/freenet/src/freenet/support/DoublyLinkedList.java > > Log: > > Add more comments to the FIXME. > > So can we un-deprecate it? Done. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] [freenet-cvs] r26635 - trunk/freenet/src/freenet/io/comm
On Thu, Apr 9, 2009 at 6:13 AM, Matthew Toseland wrote: > On Wednesday 08 April 2009 16:08:58 Daniel Cheng wrote: >> On Wed, Apr 8, 2009 at 11:05 PM, Daniel Cheng > wrote: >> > On Wed, Apr 8, 2009 at 8:23 PM, Matthew Toseland >> > wrote: >> >> On Wednesday 08 April 2009 06:03:34 j16s...@freenetproject.org wrote: >> >>> Author: j16sdiz >> >>> Date: 2009-04-08 05:03:33 + (Wed, 08 Apr 2009) >> >>> New Revision: 26635 >> >>> >> >>> Modified: >> >>> trunk/freenet/src/freenet/io/comm/UdpSocketHandler.java >> >>> Log: >> >>> Reduce SocketTimeOutException >> >>> >> >>> SocketTimeOutException was causing the thread wake up >> >>> every 1 second and creating (slow) Exceptions. >> >>> >> >>> This patch, however, effectivly broke the Hang Checker, >> >>> hence must be applied after the "Remove UDPSocket Hang >> >>> Checker" patch. >> >> >> >> As I understand it, waiting for a packet cannot be interrupted. Which > means >> > >> > It can interrupted _ONLY IF_ you call close(). >> > That's why I have r25533. >> >> Oops... I think I have handled that. but i didn't >> This is fixed at r26654 >> >> >> that after this patch is applied, shutting down opennet and then starting > it >> >> up again within 2 minutes will break with a binding error. Correct? >> >> fixed > > Tested? > tested, disable and re-enable opennet works. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Git vs hg: we must decide now!
On Wednesday 08 April 2009 13:28:28 Matthew Toseland wrote: > Assuming I haven't missed anyone and there are no disputes, clearly git has > it. I can haz a happy! David signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] [freenet-cvs] r26638 - trunk/freenet/test/freenet/support
On Thu, Apr 9, 2009 at 6:12 AM, Matthew Toseland wrote: > On Wednesday 08 April 2009 16:11:31 Daniel Cheng wrote: >> On Wed, Apr 8, 2009 at 8:26 PM, Matthew Toseland >> wrote: >> > On Wednesday 08 April 2009 06:04:48 j16s...@freenetproject.org wrote: >> >> Author: j16sdiz >> >> Date: 2009-04-08 05:04:47 + (Wed, 08 Apr 2009) >> >> New Revision: 26638 >> >> >> >> Added: >> >> trunk/freenet/test/freenet/support/PrioritizedSerialExecutorTest.java >> >> Log: >> >> JUnit for PrioritizedSerialExecutor >> > >> > Race conditions??? >> >> The test case using locks and SynchronousQueue to make sure >> the test case are run in the predictable order. > > And at the expected rate? > not :) i have tried my best to make this work in all machines i have access to. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] [freenet-cvs] r26609 - trunk/freenet/src/freenet/crypt
On Thu, Apr 9, 2009 at 6:14 AM, Matthew Toseland
wrote:
> On Wednesday 08 April 2009 01:28:41 Daniel Cheng wrote:
>> On Wed, Apr 8, 2009 at 4:02 AM, Matthew Toseland
>> wrote:
>> > On Tuesday 07 April 2009 16:26:42 j16s...@freenetproject.org wrote:
>> >> Author: j16sdiz
>> >> Date: 2009-04-07 15:26:41 + (Tue, 07 Apr 2009)
>> >> New Revision: 26609
>> >>
>> >> Modified:
>> >> trunk/freenet/src/freenet/crypt/Yarrow.java
>> >> Log:
>> >> Use int[] instead of Integer, save autoboxing and put()
>> >>
>> >> Modified: trunk/freenet/src/freenet/crypt/Yarrow.java
>> >> ===
>> >> --- trunk/freenet/src/freenet/crypt/Yarrow.java 2009-04-07 15:06:39
> UTC (rev
>> > 26608)
>> >> +++ trunk/freenet/src/freenet/crypt/Yarrow.java 2009-04-07 15:26:41
> UTC (rev
>> > 26609)
>> >> @@ -444,12 +444,12 @@
>> >> private MessageDigest fast_pool, slow_pool;
>> >> private int fast_entropy, slow_entropy;
>> >> private boolean fast_select;
>> >> - private Map entropySeen;
>> >> + private Map entropySeen;
>> >>
>> >> private void accumulator_init(String digest) throws
>> > NoSuchAlgorithmException {
>> >> fast_pool = MessageDigest.getInstance(digest);
>> >> slow_pool = MessageDigest.getInstance(digest);
>> >> - entropySeen = new HashMap();
>> >> + entropySeen = new HashMap();
>> >> }
>> >>
>> >> @Override
>> >> @@ -515,21 +515,21 @@
>> >> slow_entropy += actualEntropy;
>> >>
>> >> if(source != null) {
>> >> - Integer contributedEntropy =
> entropySeen.get(source);
>> >> - if(contributedEntropy == null)
>> >> - contributedEntropy =
> Integer.valueOf(actualEntropy);
>> >> - else
>> >> - contributedEntropy =
> Integer.valueOf(actualEntropy +
>> > contributedEntropy.intValue());
>> >> - entropySeen.put(source,
> contributedEntropy);
>> >> + int[] contributedEntropy =
> entropySeen.get(source);
>> >> + if(contributedEntropy == null) {
>> >> + contributedEntropy = new
> int[] { actualEntropy };
>> >> + entropySeen.put(source,
> contributedEntropy);
>> >> + } else
>> >> + contributedEntropy[0]++;
>> >
>> > Wrong.
>> > contributedEntropy[0] += actualEntropy;
>> > Please do not make mistakes in crypt/!
>> > Otherwise it's a neat optimisation, although using a MutableInt would be
>> > cleaner.
>>
>> fixed in r26629.
>>
>> (this is in the reseed from entropy source code path.)
>> btw, the current code maybe reseeding too infrequent.
>>
>> This is the current reseed logic:
>> (1) we have more then (SLOW_THRESHOLD * 2) bits in slow_entropy
>> _AND_
>> (2) those entropy are from TWO DIFFERENT source, each contribute more
> then
>> THRESHOLD bits
>>
>> However, we have ONLY TWO sources,
>> the TWO DIFFERENT sources requirement is too hard to meet.
>
> Yeah. And one of the sources (incoming packet hashes) always reports 0, as it
> may be observable/influencible. Ideas?
>
> BTW, what are the two sources?
>
FNPPacketMangler ,
private final EntropySource fnpTimingSource;
private final EntropySource myPacketDataSource;
fnpTimeingSource is the packet arrivial time
myPacketDataSource is the hash of packets.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Client apps limiting load in bad ways
> -Original Message- > From: devl-boun...@freenetproject.org > [mailto:devl-boun...@freenetproject.org] On Behalf Of Matthew Toseland > Sent: Thursday, April 09, 2009 12:28 AM > To: Discussion of development issues > Subject: [freenet-dev] Client apps limiting load in bad ways > > FMS does bad things to limit load: Rather than queueing every > key at MaxRetries=-1, it does all the scheduling itself, Freetalk and WoT can do -1 because we/I can patch them right now. This can be done for any SSK/CHK URIs which the plugins know for sure that they exist, i.e. for messages and message lists which they obtained the URIs of through other peoples message lists... However Freetalk and WoT will have to periodically try to download message lists and trust lists of ALL identities to ensure that nothing is missed if the propagation through third party lists does not work. > The reason that FMS's behaviour is a problem is that ULPRs > rely on nodes knowing which keys they are interested in. > ULPRs are the main network level optimisation for SSK > polling/chat/WoT apps, so this is potentially a big source of > problems - excessive load, slow propagation of messages etc. > > Possible solutions: > > DontFetch=true : if set, a request would be purely passive: > no requests would be started as a result of that request, but > the node would know it wants those blocks, and if the blocks > do come in, the request would progress. The probability of such a request to succeed will be horribly low, won't it? > > Client configurable cooldown: Running a request with > MaxRetries=-1 means fetching it 3 times every half hour. We > could make the half hour bit configurable, and maybe the 3 > times too. Good idea IMHO, especially for FT/WoT. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Client apps limiting load in bad ways
> -Ursprüngliche Nachricht-
> Von: "Matthew Toseland"
> Gesendet: 09.04.09 00:28:48
> An: Discussion of development issues
> Betreff: [freenet-dev] Client apps limiting load in bad ways
> FMS does bad things to limit load: Rather than queueing every key at
> MaxRetries=-1, it does all the scheduling itself, doing simple ClientGet's
> with no maxretries and no priority (hence prio 2), in order to enforce a
> maximum number of parallel requests. Other client apps probably do similar
> things. MiniChat presumably does the opposite, constantly requesting keys,
> starting a new request for the same key when the last one has finished
> (saces: is this true?).
yes. the code 'emulates' a (passive)request with short timeout (a few minutes)
;)
while (currenttime() < timeout) {
sendrequest();
if sucess break;
wait(500);
}
>
> The reason that FMS's behaviour is a problem is that ULPRs rely on nodes
> knowing which keys they are interested in. ULPRs are the main network level
> optimisation for SSK polling/chat/WoT apps, so this is potentially a big
> source of problems - excessive load, slow propagation of messages etc.
>
> Possible solutions:
>
> DontFetch=true : if set, a request would be purely passive: no requests would
> be started as a result of that request, but the node would know it wants
> those blocks, and if the blocks do come in, the request would progress.
> Suggested usage is that a client app would set up DontFetch requests for
> everything it is interested in, with Verbosity sufficiently high, and then
> poll using whatever load heuristics they want. When a block (typically an SSK
> block when polling SSK outqueues) is found, either the request succeeds,
> fatally fails, or it is a redirect. If it is a redirect, the client would get
> a SimpleProgress and would schedule a normal request to fetch the rest of the
> key. (This should be fairly easy to implement)
>
> Client configurable cooldown: Running a request with MaxRetries=-1 means
> fetching it 3 times every half hour. We could make the half hour bit
> configurable, and maybe the 3 times too. Note that it is quite likely that we
> will eventually impose network level limits on polling (e.g. using
> RecentlyFailed), so setting this low will not necessarily (in the long run)
> yield better performance. (This should be moderately easy to implement)
>
> Any other ideas? Is this the right approach? IMHO we can implement whatever
> API fairly easily, but maybe there are some other ideas?
If everybody is listen a key with DontFetch=True, the key/data will be known
along the regular insert chain,
but how is this further spreaded?
MfG
saces
__
Verschicken Sie SMS direkt vom Postfach aus - in alle deutschen und viele
ausländische Netze zum gleichen Preis!
https://produkte.web.de/webde_sms/sms
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] git/hg hosting
On Wednesday 08 April 2009 16:22:03 guido wrote: > > Your pgp signature is charset corrupted. > The pgp signature is pure ASCII, how can it possibly be charset corrupted? Because there are encodings out there that don’t care about the lowest 7 bit being ASCII. Admittedly, they are far and in-between. David signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Client apps limiting load in bad ways
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Toseland wrote: > FMS does bad things to limit load: Rather than queueing every key at > MaxRetries=-1, it does all the scheduling itself, doing simple ClientGet's > with no maxretries and no priority (hence prio 2), in order to enforce a > maximum number of parallel requests. Other client apps probably do similar > things. MiniChat presumably does the opposite, constantly requesting keys, > starting a new request for the same key when the last one has finished > (saces: is this true?). > > The reason that FMS's behaviour is a problem is that ULPRs rely on nodes > knowing which keys they are interested in. ULPRs are the main network level > optimisation for SSK polling/chat/WoT apps, so this is potentially a big > source of problems - excessive load, slow propagation of messages etc. > > Possible solutions: > > DontFetch=true : if set, a request would be purely passive: no requests would A minor point probably, but is it at all possible not to use negative there? It sometimes make FCPv2 standard harder to follow: DontFailToNotFetch=false takes a few extra seconds to figure out than Fetch=true > be started as a result of that request, but the node would know it wants > those blocks, and if the blocks do come in, the request would progress. > Suggested usage is that a client app would set up DontFetch requests for > everything it is interested in, with Verbosity sufficiently high, and then > poll using whatever load heuristics they want. When a block (typically an SSK > block when polling SSK outqueues) is found, either the request succeeds, > fatally fails, or it is a redirect. If it is a redirect, the client would get > a SimpleProgress and would schedule a normal request to fetch the rest of the > key. (This should be fairly easy to implement) > > Client configurable cooldown: Running a request with MaxRetries=-1 means > fetching it 3 times every half hour. We could make the half hour bit > configurable, and maybe the 3 times too. Note that it is quite likely that we > will eventually impose network level limits on polling (e.g. using > RecentlyFailed), so setting this low will not necessarily (in the long run) > yield better performance. (This should be moderately easy to implement) > > Any other ideas? Is this the right approach? IMHO we can implement whatever > API fairly easily, but maybe there are some other ideas? > > > > > ___ > Devl mailing list > Devl@freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl - -- http://freedom.libsyn.com/ Echo of Freedom, Radical Podcast http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal http://www.freedomporn.org/ Freedom Porn, anarchist and activist smut "None of us are free until all of us are free."~ Mihail Bakunin -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkndsCwACgkQuWy2EFICg+05NACfR91vYCW2StcpNFC/6FTlYrro j9cAoLqQ7apxP3XLP2CBOW8vYRciOkZx =dJXO -END PGP SIGNATURE- ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
