[freenet-dev] [freenet-support] Freenet 0.7.5 build 1345

2011-02-10 Thread Matthew Toseland
On Tuesday 08 Feb 2011 02:33:28 Ian Clarke wrote:
> On Mon, Feb 7, 2011 at 6:49 PM, Matthew Toseland
> wrote:
> 
> > Freenet 0.7.5 build 1345 is now available, it will be mandatory on
> > Thursday, please upgrade ASAP. This is just a small bugfix for a problem
> > pointed out on FMS (a NullPointerException). Eleriseth even provided a
> > partial fix, which has been merged along with some related fixes. IMHO this
> > only happens if a peer gets to too-high ping times *before* it is first
> > backed off, so I doubt it's the cause of the problems we've been having
> > lately (most notably it taking *ages* to bootstrap a new node onto opennet),
> > but it's still important to fix it.
> >
> 
> Are you sure its wise to give people so little time to upgrade for what
> appears to be a minor fix?

I just explained that it could be a serious problem.

However, UOM may be broken, in which case a short period is bad. I've been 
working on that ...
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20110210/cc70dbaf/attachment.pgp>


[freenet-dev] [freenet-support] Call for seednodes and explanation of current problems

2011-02-10 Thread Juiceman
On Thu, Feb 10, 2011 at 8:49 PM, Matthew Toseland
 wrote:
> On Saturday 05 Feb 2011 19:26:46 Juiceman wrote:
>> On Sat, Feb 5, 2011 at 2:24 PM, Juiceman  wrote:
>> > On Sat, Feb 5, 2011 at 2:17 PM, Juiceman  wrote:
>> >> On Sat, Feb 5, 2011 at 1:39 PM, Matthew Toseland
>> >>  wrote:
>> >>> We need more seednodes. I will explain the broader situation below. If 
>> >>> you can run a seednode - which means you need a forwarded port, a 
>> >>> reasonably static IP address (or dyndns name), and a reasonable amount 
>> >>> of bandwidth (especially upstream), and a reasonably stable node, please 
>> >>> send me your opennet noderef (from the strangers page in advanced mode), 
>> >>> and enable "Be a seednode" in the advanced config. Thanks.
>> >>>
>> >>> Details:
>> >>>
>> >>> One of the problems Freenet has at the moment is that bootstrapping a 
>> >>> new node can take an awfully long time - 20 minutes or more sometimes. 
>> >>> It is not clear why; we seem to either get rejected by seednodes (most 
>> >>> of the time), or they return nothing, maybe a few "not wanted" notices, 
>> >>> or they return lots of noderefs and we manage to announce.
>> >>>
>> >>> This might be due to bugs. 1343 fixed a bug that apparently badly 
>> >>> affected some seednodes. However it appears most seednodes have upgraded 
>> >>> now.
>> >>>
>> >>> There doesn't seem to be a problem with losing connections - backoff yes 
>> >>> but once a node is connected it seems to mostly stay connected.
>> >>>
>> >>> The most likely answer seems to be that we just don't have enough 
>> >>> seednodes to cope with the load.
>> >>>
>> >>> It is also possible that this is due to an attack. It did come on 
>> >>> relatively suddenly a few weeks ago (it was bad before but it got much 
>> >>> worse), and it seems to have got significantly worse in the last week. 
>> >>> It is not clear how we would identify an attack if that was the problem; 
>> >>> there are no obvious signs so far.
>> >>>
>> >>> It is also possible it is a client-side bug. Testing of the master 
>> >>> branch would be useful, it has some small changes.
>> >>>
>> >>
>> >> Errors in my log (build 1344)
>> >
>> > Also nodestats have been horrible on my seednode for last week or two.
>> > ?I am noticing node ping times in the 1500 - 3500 ms range. ?Mostly
>> > during daytime hours here (GMT -5). ?Not sure if ATT Uverse has
>> > quietly started throttling p2p (they say they don't), or if it is an
>> > attack or bug. ?Seeding for 206 is typical and didn't kill my node in
>> > the past. ?Let me know what logger settings to set and I can send you
>> > my logs if you want.
>> >
>> > Peer statistics
>> >
>> > ? ?* Connected: 15
>> > ? ?* Backed off: 3
>> > ? ?* Too old: 67
>> > ? ?* Disconnected: 13
>> > ? ?* Never connected: 5
>> > ? ?* Clock Problem: 1
>> > ? ?* Seeding for: 206
>> > ? ?* Max peers: 36
>> > ? ?* Max strangers: 36
>> >
>> > Bandwidth
>> >
>> > ? ?* Input Rate: 48.2 KiB/s (of 1.0 MiB/s)
>> > ? ?* Output Rate: 31.6 KiB/s (of 105 KiB/s)
>> > ? ?* Session Total Input: 49.2 MiB (43.7 KiB/s average)
>> > ? ?* Session Total Output: 35.7 MiB (31.7 KiB/s average)
>> > ? ?* Payload Output: 214 KiB (190 B/sec)(0%)
>> >
>>
>> Node status overview
>>
>> ? ? * bwlimitDelayTime: 2947ms
>> ? ? * bwlimitDelayTimeBulk: 2893ms
>> ? ? * bwlimitDelayTimeRT: 9078ms
>> ? ? * nodeAveragePingTime: 2315ms
>> ? ? * darknetSizeEstimateSession: 0 nodes
>> ? ? * opennetSizeEstimateSession: 833 nodes
>> ? ? * nodeUptimeSession: 26m34s
>> ? ? * nodeUptimeTotal: 8w2d
>> ? ? * routingMissDistanceLocal: 0.0650
>> ? ? * routingMissDistanceRemote: 0.0141
>> ? ? * routingMissDistanceOverall: 0.0261
>> ? ? * backedOffPercent: 22.3%
>> ? ? * pInstantReject: 95.8%
>> ? ? * unclaimedFIFOSize: 2663
>> ? ? * RAMBucketPoolSize: 12.8 MiB / 150 MiB
>> ? ? * uptimeAverage: 99.3%
>
> Ping times that high mean your node won't accept any requests at all, 
> although it might accept some announcements.
>
> Usually this is caused by network or CPU problems. When I've run my seednode 
> lately it hasn't had high ping times, nor have any of my other nodes.
>
> The NPE is fixed btw.

I have a tech coming out to diagnose my DSL issues, but the last
several builds have improved my ping times.



[freenet-dev] Wonder why bootstrapping is slow

2011-02-10 Thread Juiceman
On Thu, Feb 10, 2011 at 8:47 PM, Matthew Toseland
 wrote:
>
> On Tuesday 08 Feb 2011 04:16:25 Juiceman wrote:
> > Bootstrapping from seednodes is really slow right now. ?I run one of
> > those seednodes. ?Here are my stats, they have been horrible for about
> > 2 weeks now. ?I don't suppose my node is serving many announcements,
> > huh? ?Are other seednodes in similarly bad shape?
>
> I have some theories.
>
> It looks like most of the announcement load is caused by old nodes trying and 
> repeatedly failing to update using Update Over Mandatory.
>
> Recent builds should help with this, 1349 should help even more (and might 
> help improve announcement performance generally).
>

Much better!? 1349 stats:

Node status overview

bwlimitDelayTime:?1152ms
bwlimitDelayTimeBulk:?1152ms
bwlimitDelayTimeRT:?444ms
nodeAveragePingTime:?356ms
darknetSizeEstimateSession:?0?nodes
opennetSizeEstimateSession:?1804?nodes
nodeUptimeSession:?1h28m
nodeUptimeTotal:?9w21h
routingMissDistanceLocal:?0.1760
routingMissDistanceRemote:?0.1475
routingMissDistanceOverall:?0.1481
backedOffPercent:?36.7%
pInstantReject:?0.0%
unclaimedFIFOSize:?1398
RAMBucketPoolSize:?66.2 MiB / 150 MiB
uptimeAverage:?90.1%

Peer statistics

Connected:?26
Backed off:?9
Disconnected:?1
Never connected:?1
Seeding for:?32
Max peers: 36
Max strangers: 36

Bandwidth

Input Rate: 31.7?KiB/s (of 1.0?MiB/s)
Output Rate: 33.4?KiB/s (of 105?KiB/s)
Session Total Input: 267?MiB (51.4?KiB/s average)
Session Total Output: 311?MiB (60.0?KiB/s average)
Payload Output: 236?MiB (45.4?KiB/sec)(75%)
Global Total Input: 369?GiB
Global Total Output: 433?GiB
Request output (excluding payload): CHK -6.61?MiB SSK 23.6?MiB.
Insert output (excluding payload): CHK -2.26?MiB SSK 492?KiB.
Offered keys: sending keys -79.8?KiB, sending offers 307?KiB
Swapping Output: 2.75?MiB.
Connection setup: 4.78?MiB output
Ack-only packets: 6.74?MiB
Resent bytes: 6.70?MiB (2%)
Updater Output: 113?KiB
Announcement output: 9.14?MiB (transferring node refs payload 8.26?MiB)
Admin bytes: 313?KiB initial messages, 0?B IP change messages, 917?B
disconnection notifications, 0?B routing status
Debugging bytes: 0?B network coloring, 0?B ping, 0?B probe requests, 0
B routed test messages.
Node to node messages: 564?KiB
Load allocation notices: 1.08?MiB
Other output: 28.0?MiB (8%)
Total non-request overhead: 6.19 KiB/sec (10%).



[freenet-dev] Freenet 0.7.5 build 1348

2011-02-10 Thread Matthew Toseland
Freenet 0.7.5 build 1348 is now available, please upgrade, it will be mandatory 
on Monday. This build fixes a bug in 1347 that caused us to reject all TOO OLD 
nodes, at announcement time, on the nodes it is announced to. They could still 
swamp the seednodes, and probably are still doing so. Hopefully the changes in 
1347 and 1348 combined will result in old nodes being able to announce and 
update via Update Over Mandatory, and thus to stop spamming announcement, and 
hopefully that will result, in a few days, in significantly improved opennet 
bootstrapping performance.

Please let us know how it goes.

Thanks!
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20110210/237fb7fa/attachment.pgp>


[freenet-dev] Freenet 0.7.5 build 1347

2011-02-10 Thread Matthew Toseland
Freenet 0.7.5 build 1347 is now available. This build improves update over 
mandatory and related code. It appears that there are a significant number of 
nodes around build 1320 that are unable to update via UOM, and are constantly 
announcing as a result. Hopefully this will fix this. Amongst other changes it 
introduces a limit on the number of TOO OLD peers and makes the node stay 
connected to them as long as a viable UOM transfer is running. Please upgrade! 
Thanks. 1347 will be mandatory on Monday, but hopefully it will be possible to 
test the changes tomorrow.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20110210/f1419cfa/attachment.pgp>


[freenet-dev] Freenet 0.7.5 build 1346

2011-02-10 Thread Matthew Toseland
Freenet 0.7.5 build 1346 is now available. Please upgrade, it will be mandatory 
on Friday. 
Changes include:
- Do not accept too-old opennet nodes as peers of seednodes in announcement. We 
won't be able to send the update to them because UOM doesn't work on seeds, and 
some versions around 1320 got severely broken when this happened. Hopefully 
some of them will now be able to upgrade via other peers, or at least more 
quickly, and thus will stop hammering the seednodes quite so much!
- Better disconnection detection. We were thinking we are still connected 
sometimes when we were in fact disconnected and receiving some early auth 
packets but not the late ones (as with the above case).
- Send keepalive messages more often. (Note that this has zero overhead on a 
typical busy node).
- Fix a rare synchronization bug.

Please upgrade, and let us know how Freenet goes! Thanks.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20110210/6d514a88/attachment.pgp>


[freenet-dev] Freetalk status update

2011-02-10 Thread xor
Hi,
I've been ill for almost the past 7 days, thats why there is no RC2 of 
Freetalk/WoT yet.
I will resume work as soon as I'm healthy again, probably by the weekend.

The amount of bugs which showed up in RC1 is manageable:
- There is one complex bug in the WoT score computation of which I have no 
idea why it is happening and which might take some time to track down. It was 
not happening in the pre-RC1 WoT so it has probably been introduced by the 
heavy refactoring which is contained in RC1 - therefore it  should be a "real" 
bug and not a design failure in the WoT algorithm which means that it will 
probably be easy to fix once it was tracked down.
- There is a handful of non complex bugs which must be fixed but are easy to 
fix.

So maybe the RC-phase will take around 2 weeks due to the illness, but overall 
the stuff is behaving quite well.

Grettings, xor
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20110210/5629b652/attachment.pgp>


[freenet-dev] Freenet 0.7.5 build 1346

2011-02-10 Thread Matthew Toseland
Freenet 0.7.5 build 1346 is now available. Please upgrade, it will be mandatory 
on Friday. 
Changes include:
- Do not accept too-old opennet nodes as peers of seednodes in announcement. We 
won't be able to send the update to them because UOM doesn't work on seeds, and 
some versions around 1320 got severely broken when this happened. Hopefully 
some of them will now be able to upgrade via other peers, or at least more 
quickly, and thus will stop hammering the seednodes quite so much!
- Better disconnection detection. We were thinking we are still connected 
sometimes when we were in fact disconnected and receiving some early auth 
packets but not the late ones (as with the above case).
- Send keepalive messages more often. (Note that this has zero overhead on a 
typical busy node).
- Fix a rare synchronization bug.

Please upgrade, and let us know how Freenet goes! Thanks.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Freenet 0.7.5 build 1347

2011-02-10 Thread Matthew Toseland
Freenet 0.7.5 build 1347 is now available. This build improves update over 
mandatory and related code. It appears that there are a significant number of 
nodes around build 1320 that are unable to update via UOM, and are constantly 
announcing as a result. Hopefully this will fix this. Amongst other changes it 
introduces a limit on the number of TOO OLD peers and makes the node stay 
connected to them as long as a viable UOM transfer is running. Please upgrade! 
Thanks. 1347 will be mandatory on Monday, but hopefully it will be possible to 
test the changes tomorrow.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Freenet 0.7.5 build 1348

2011-02-10 Thread Matthew Toseland
Freenet 0.7.5 build 1348 is now available, please upgrade, it will be mandatory 
on Monday. This build fixes a bug in 1347 that caused us to reject all TOO OLD 
nodes, at announcement time, on the nodes it is announced to. They could still 
swamp the seednodes, and probably are still doing so. Hopefully the changes in 
1347 and 1348 combined will result in old nodes being able to announce and 
update via Update Over Mandatory, and thus to stop spamming announcement, and 
hopefully that will result, in a few days, in significantly improved opennet 
bootstrapping performance.

Please let us know how it goes.

Thanks!


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] [freenet-support] Freenet 0.7.5 build 1345

2011-02-10 Thread Matthew Toseland
On Tuesday 08 Feb 2011 02:33:28 Ian Clarke wrote:
 On Mon, Feb 7, 2011 at 6:49 PM, Matthew Toseland
 t...@amphibian.dyndns.orgwrote:
 
  Freenet 0.7.5 build 1345 is now available, it will be mandatory on
  Thursday, please upgrade ASAP. This is just a small bugfix for a problem
  pointed out on FMS (a NullPointerException). Eleriseth even provided a
  partial fix, which has been merged along with some related fixes. IMHO this
  only happens if a peer gets to too-high ping times *before* it is first
  backed off, so I doubt it's the cause of the problems we've been having
  lately (most notably it taking *ages* to bootstrap a new node onto opennet),
  but it's still important to fix it.
 
 
 Are you sure its wise to give people so little time to upgrade for what
 appears to be a minor fix?

I just explained that it could be a serious problem.

However, UOM may be broken, in which case a short period is bad. I've been 
working on that ...


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Wonder why bootstrapping is slow

2011-02-10 Thread Matthew Toseland
On Tuesday 08 Feb 2011 04:16:25 Juiceman wrote:
 Bootstrapping from seednodes is really slow right now.  I run one of
 those seednodes.  Here are my stats, they have been horrible for about
 2 weeks now.  I don't suppose my node is serving many announcements,
 huh?  Are other seednodes in similarly bad shape?

I have some theories.

It looks like most of the announcement load is caused by old nodes trying and 
repeatedly failing to update using Update Over Mandatory.

Recent builds should help with this, 1349 should help even more (and might help 
improve announcement performance generally).
 
 Node status overview
 
 bwlimitDelayTime: 6706ms
 bwlimitDelayTimeBulk: 6706ms
 bwlimitDelayTimeRT: 0ms
 nodeAveragePingTime: 2452ms
 darknetSizeEstimateSession: 0 nodes
 opennetSizeEstimateSession: 762 nodes
 nodeUptimeSession: 44m57s
 nodeUptimeTotal: 8w5d
 routingMissDistanceLocal: 0.1065
 routingMissDistanceRemote: 0.1270
 routingMissDistanceOverall: 0.1234
 backedOffPercent: 34.4%
 pInstantReject: 92.5%
 unclaimedFIFOSize: 4155
 RAMBucketPoolSize: 12.5 MiB / 150 MiB
 uptimeAverage: 97.2%
 
 Peer statistics
 
 Connected: 17
 Backed off: 4
 Too old: 65
 Disconnected: 9
 Never connected: 6
 Disconnecting: 1
 Seeding for: 199
 Max peers: 36
 Max strangers: 36
 
 Bandwidth
 
 Input Rate: 42.2 KiB/s (of 1.0 MiB/s)
 Output Rate: 31.2 KiB/s (of 105 KiB/s)
 Session Total Input: 115 MiB (43.8 KiB/s average)
 Session Total Output: 86.0 MiB (32.6 KiB/s average)
 Payload Output: 253 KiB (96 B/sec)(0%)
 Global Total Input: 357 GiB
 Global Total Output: 422 GiB
 Request output (excluding payload): CHK 39.7 KiB SSK 111 KiB.
 Insert output (excluding payload): CHK 7.53 KiB SSK 1.70 KiB.
 Offered keys: sending keys 100 B, sending offers 1016 B
 Swapping Output: 1.99 MiB.
 Connection setup: 30.3 MiB output
 Ack-only packets: 31.6 MiB
 Resent bytes: 5.83 MiB (6%)
 Updater Output: 104 KiB
 Announcement output: 6.29 MiB (transferring node refs payload 6.48 MiB)
 Admin bytes: 1.08 MiB initial messages, 1.37 KiB IP change messages,
 2.10 KiB disconnection notifications, 0 B routing status
 Debugging bytes: 0 B network coloring, 0 B ping, 6.12 KiB probe
 requests, 0 B routed test messages.
 Node to node messages: 1.45 MiB
 Load allocation notices: 288 KiB
 Other output: 6.59 MiB (7%)
 Total non-request overhead: 31.4 KiB/sec (96%).


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] [freenet-support] Call for seednodes and explanation of current problems

2011-02-10 Thread Matthew Toseland
On Saturday 05 Feb 2011 19:26:46 Juiceman wrote:
 On Sat, Feb 5, 2011 at 2:24 PM, Juiceman juicema...@gmail.com wrote:
  On Sat, Feb 5, 2011 at 2:17 PM, Juiceman juicema...@gmail.com wrote:
  On Sat, Feb 5, 2011 at 1:39 PM, Matthew Toseland
  t...@amphibian.dyndns.org wrote:
  We need more seednodes. I will explain the broader situation below. If 
  you can run a seednode - which means you need a forwarded port, a 
  reasonably static IP address (or dyndns name), and a reasonable amount of 
  bandwidth (especially upstream), and a reasonably stable node, please 
  send me your opennet noderef (from the strangers page in advanced mode), 
  and enable Be a seednode in the advanced config. Thanks.
 
  Details:
 
  One of the problems Freenet has at the moment is that bootstrapping a new 
  node can take an awfully long time - 20 minutes or more sometimes. It is 
  not clear why; we seem to either get rejected by seednodes (most of the 
  time), or they return nothing, maybe a few not wanted notices, or they 
  return lots of noderefs and we manage to announce.
 
  This might be due to bugs. 1343 fixed a bug that apparently badly 
  affected some seednodes. However it appears most seednodes have upgraded 
  now.
 
  There doesn't seem to be a problem with losing connections - backoff yes 
  but once a node is connected it seems to mostly stay connected.
 
  The most likely answer seems to be that we just don't have enough 
  seednodes to cope with the load.
 
  It is also possible that this is due to an attack. It did come on 
  relatively suddenly a few weeks ago (it was bad before but it got much 
  worse), and it seems to have got significantly worse in the last week. It 
  is not clear how we would identify an attack if that was the problem; 
  there are no obvious signs so far.
 
  It is also possible it is a client-side bug. Testing of the master branch 
  would be useful, it has some small changes.
 
 
  Errors in my log (build 1344)
 
  Also nodestats have been horrible on my seednode for last week or two.
   I am noticing node ping times in the 1500 - 3500 ms range.  Mostly
  during daytime hours here (GMT -5).  Not sure if ATT Uverse has
  quietly started throttling p2p (they say they don't), or if it is an
  attack or bug.  Seeding for 206 is typical and didn't kill my node in
  the past.  Let me know what logger settings to set and I can send you
  my logs if you want.
 
  Peer statistics
 
 * Connected: 15
 * Backed off: 3
 * Too old: 67
 * Disconnected: 13
 * Never connected: 5
 * Clock Problem: 1
 * Seeding for: 206
 * Max peers: 36
 * Max strangers: 36
 
  Bandwidth
 
 * Input Rate: 48.2 KiB/s (of 1.0 MiB/s)
 * Output Rate: 31.6 KiB/s (of 105 KiB/s)
 * Session Total Input: 49.2 MiB (43.7 KiB/s average)
 * Session Total Output: 35.7 MiB (31.7 KiB/s average)
 * Payload Output: 214 KiB (190 B/sec)(0%)
 
 
 Node status overview
 
 * bwlimitDelayTime: 2947ms
 * bwlimitDelayTimeBulk: 2893ms
 * bwlimitDelayTimeRT: 9078ms
 * nodeAveragePingTime: 2315ms
 * darknetSizeEstimateSession: 0 nodes
 * opennetSizeEstimateSession: 833 nodes
 * nodeUptimeSession: 26m34s
 * nodeUptimeTotal: 8w2d
 * routingMissDistanceLocal: 0.0650
 * routingMissDistanceRemote: 0.0141
 * routingMissDistanceOverall: 0.0261
 * backedOffPercent: 22.3%
 * pInstantReject: 95.8%
 * unclaimedFIFOSize: 2663
 * RAMBucketPoolSize: 12.8 MiB / 150 MiB
 * uptimeAverage: 99.3%

Ping times that high mean your node won't accept any requests at all, although 
it might accept some announcements.

Usually this is caused by network or CPU problems. When I've run my seednode 
lately it hasn't had high ping times, nor have any of my other nodes.

The NPE is fixed btw.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Freenet 0.7.5 build 1349

2011-02-10 Thread Matthew Toseland
Freenet 0.7.5 build 1349 is now available. This is another attempt to fix 
announcement/opennet bootstrapping and Update Over Mandatory on old nodes 
(which appear to be flooding the announcement mechanism). The main changes 
include:
- Give nodes more time to connect since it seems announcement can take some 
time to return the announced noderefs.
- Process the announcement noderef transfers in parallel so we can return the 
noderefs as soon as possible, one won't get stuck behind another (note this was 
theoretically exploitable, no evidence it has been though). IMHO this is a 
likely reason for the problems we've been having - when I look at my main node, 
announcements come in and are handled and work fine; when I look at a new node 
trying to announce, or an old node trying to announce, the picture is radically 
different. And on the node handling an announcement, I frequently see that a 
node (especially an old node) gets announced and then never connects - so we 
can't send it the update package.
- Accept TOO OLD peers a bit more often, while still respecting the limits set 
out in 1347: don't worry about the overall limit, related changes.

Please upgrade ASAP! I have set a longish mandatory (the 17th), but IMHO this 
has a good chance of allowing old nodes to successfully announce and update, 
eliminating most of the load from old nodes announcing and failing to update, 
improve announcement generally, and hopefully get bootstrapping back to 
something like the fairly good performance we had for most of last year.

Thanks! Please report any problems.

Sorry for all the difficulties we've been having recently, there have been many 
reasons for these, but I hope we are getting a handle on them; certainly I have 
a better idea what's going on now...


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Wonder why bootstrapping is slow

2011-02-10 Thread Juiceman
On Thu, Feb 10, 2011 at 8:47 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:

 On Tuesday 08 Feb 2011 04:16:25 Juiceman wrote:
  Bootstrapping from seednodes is really slow right now.  I run one of
  those seednodes.  Here are my stats, they have been horrible for about
  2 weeks now.  I don't suppose my node is serving many announcements,
  huh?  Are other seednodes in similarly bad shape?

 I have some theories.

 It looks like most of the announcement load is caused by old nodes trying and 
 repeatedly failing to update using Update Over Mandatory.

 Recent builds should help with this, 1349 should help even more (and might 
 help improve announcement performance generally).


Much better!  1349 stats:

Node status overview

bwlimitDelayTime: 1152ms
bwlimitDelayTimeBulk: 1152ms
bwlimitDelayTimeRT: 444ms
nodeAveragePingTime: 356ms
darknetSizeEstimateSession: 0 nodes
opennetSizeEstimateSession: 1804 nodes
nodeUptimeSession: 1h28m
nodeUptimeTotal: 9w21h
routingMissDistanceLocal: 0.1760
routingMissDistanceRemote: 0.1475
routingMissDistanceOverall: 0.1481
backedOffPercent: 36.7%
pInstantReject: 0.0%
unclaimedFIFOSize: 1398
RAMBucketPoolSize: 66.2 MiB / 150 MiB
uptimeAverage: 90.1%

Peer statistics

Connected: 26
Backed off: 9
Disconnected: 1
Never connected: 1
Seeding for: 32
Max peers: 36
Max strangers: 36

Bandwidth

Input Rate: 31.7 KiB/s (of 1.0 MiB/s)
Output Rate: 33.4 KiB/s (of 105 KiB/s)
Session Total Input: 267 MiB (51.4 KiB/s average)
Session Total Output: 311 MiB (60.0 KiB/s average)
Payload Output: 236 MiB (45.4 KiB/sec)(75%)
Global Total Input: 369 GiB
Global Total Output: 433 GiB
Request output (excluding payload): CHK -6.61 MiB SSK 23.6 MiB.
Insert output (excluding payload): CHK -2.26 MiB SSK 492 KiB.
Offered keys: sending keys -79.8 KiB, sending offers 307 KiB
Swapping Output: 2.75 MiB.
Connection setup: 4.78 MiB output
Ack-only packets: 6.74 MiB
Resent bytes: 6.70 MiB (2%)
Updater Output: 113 KiB
Announcement output: 9.14 MiB (transferring node refs payload 8.26 MiB)
Admin bytes: 313 KiB initial messages, 0 B IP change messages, 917 B
disconnection notifications, 0 B routing status
Debugging bytes: 0 B network coloring, 0 B ping, 0 B probe requests, 0
B routed test messages.
Node to node messages: 564 KiB
Load allocation notices: 1.08 MiB
Other output: 28.0 MiB (8%)
Total non-request overhead: 6.19 KiB/sec (10%).
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] [freenet-support] Call for seednodes and explanation of current problems

2011-02-10 Thread Juiceman
On Thu, Feb 10, 2011 at 8:49 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 On Saturday 05 Feb 2011 19:26:46 Juiceman wrote:
 On Sat, Feb 5, 2011 at 2:24 PM, Juiceman juicema...@gmail.com wrote:
  On Sat, Feb 5, 2011 at 2:17 PM, Juiceman juicema...@gmail.com wrote:
  On Sat, Feb 5, 2011 at 1:39 PM, Matthew Toseland
  t...@amphibian.dyndns.org wrote:
  We need more seednodes. I will explain the broader situation below. If 
  you can run a seednode - which means you need a forwarded port, a 
  reasonably static IP address (or dyndns name), and a reasonable amount 
  of bandwidth (especially upstream), and a reasonably stable node, please 
  send me your opennet noderef (from the strangers page in advanced mode), 
  and enable Be a seednode in the advanced config. Thanks.
 
  Details:
 
  One of the problems Freenet has at the moment is that bootstrapping a 
  new node can take an awfully long time - 20 minutes or more sometimes. 
  It is not clear why; we seem to either get rejected by seednodes (most 
  of the time), or they return nothing, maybe a few not wanted notices, 
  or they return lots of noderefs and we manage to announce.
 
  This might be due to bugs. 1343 fixed a bug that apparently badly 
  affected some seednodes. However it appears most seednodes have upgraded 
  now.
 
  There doesn't seem to be a problem with losing connections - backoff yes 
  but once a node is connected it seems to mostly stay connected.
 
  The most likely answer seems to be that we just don't have enough 
  seednodes to cope with the load.
 
  It is also possible that this is due to an attack. It did come on 
  relatively suddenly a few weeks ago (it was bad before but it got much 
  worse), and it seems to have got significantly worse in the last week. 
  It is not clear how we would identify an attack if that was the problem; 
  there are no obvious signs so far.
 
  It is also possible it is a client-side bug. Testing of the master 
  branch would be useful, it has some small changes.
 
 
  Errors in my log (build 1344)
 
  Also nodestats have been horrible on my seednode for last week or two.
   I am noticing node ping times in the 1500 - 3500 ms range.  Mostly
  during daytime hours here (GMT -5).  Not sure if ATT Uverse has
  quietly started throttling p2p (they say they don't), or if it is an
  attack or bug.  Seeding for 206 is typical and didn't kill my node in
  the past.  Let me know what logger settings to set and I can send you
  my logs if you want.
 
  Peer statistics
 
     * Connected: 15
     * Backed off: 3
     * Too old: 67
     * Disconnected: 13
     * Never connected: 5
     * Clock Problem: 1
     * Seeding for: 206
     * Max peers: 36
     * Max strangers: 36
 
  Bandwidth
 
     * Input Rate: 48.2 KiB/s (of 1.0 MiB/s)
     * Output Rate: 31.6 KiB/s (of 105 KiB/s)
     * Session Total Input: 49.2 MiB (43.7 KiB/s average)
     * Session Total Output: 35.7 MiB (31.7 KiB/s average)
     * Payload Output: 214 KiB (190 B/sec)(0%)
 

 Node status overview

     * bwlimitDelayTime: 2947ms
     * bwlimitDelayTimeBulk: 2893ms
     * bwlimitDelayTimeRT: 9078ms
     * nodeAveragePingTime: 2315ms
     * darknetSizeEstimateSession: 0 nodes
     * opennetSizeEstimateSession: 833 nodes
     * nodeUptimeSession: 26m34s
     * nodeUptimeTotal: 8w2d
     * routingMissDistanceLocal: 0.0650
     * routingMissDistanceRemote: 0.0141
     * routingMissDistanceOverall: 0.0261
     * backedOffPercent: 22.3%
     * pInstantReject: 95.8%
     * unclaimedFIFOSize: 2663
     * RAMBucketPoolSize: 12.8 MiB / 150 MiB
     * uptimeAverage: 99.3%

 Ping times that high mean your node won't accept any requests at all, 
 although it might accept some announcements.

 Usually this is caused by network or CPU problems. When I've run my seednode 
 lately it hasn't had high ping times, nor have any of my other nodes.

 The NPE is fixed btw.

I have a tech coming out to diagnose my DSL issues, but the last
several builds have improved my ping times.
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl