[freenet-dev] Statistics Project Update #1
On Tuesday 01 May 2012 00:27:26 Michael Grube wrote: > On Mon, Apr 30, 2012 at 7:06 PM, Steve Dougherty asksteved.com>wrote: > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA1 > > > > On 04/28/2012 06:56 PM, Zlatin Balevsky wrote: > > > In Gnutella we observed that long-lived nodes tend to be better > > > connected and that they also cluster with other high-uptime nodes. > > > If the same is true for Freenet it's a good idea to keep an eye for > > > side effects as you tweak the behavior. > > > > Good to know - I'll look for that. Are there any particular effects > > you had in mind? The Metropolis-Hastings correction in the new probes > > should produce a fairly uniform distribution of endpoints despite > > clustering and well-connected nodes, but explicitly simulating the > > effects of high uptime could be helpful. > > > > It occurs to me that the probe requests I hope to depreciate allow > > reconstructing the actual network topology - perhaps we could run > > simulations on top of it? The new probe requests are currently planned > > to not report degrees or link length distributions, [1] which as far > > as I can tell would mean no way to reconstruct the network as measured > > in simulation. Does it seem reasonable to omit the ability to gather > > such information? I don't know, in the short run. In the long run we'd prefer not to expose the full topology for security reasons. > > > > > > On 04/28/2012 03:51 PM, Michael Grube wrote: > > > Are you assuming opennet, darknet, a mix? > > > > The simulation generates the graph in a way which effectively sort of > > assumes darknet: it assigns locations, then iterates over the network > > and connects nodes based on link length. [2] [3] I'm working under the > > assumption that when using the same degree distribution as the network > > is measured to have this is an accurate enough approximation. I will > > include these and similar plots in this week's progress report. > > > > Ah, ok. You're making a Kleinberg Small World graph. Nice. > > > > > > My understanding is that a more thorough simulation of darknet would > > randomly assign locations and reassign them with the location-swapping > > algorithm used in Fred. > > > Right. Funnily enough, the swapping algorithm is most concisely described > in the Pitch Black paper. I'd suggest that as a reference for quick > implementation. > > > > I think simulating opennet would be more work > > both to implement and to run, as it would mean implementing > > path-folding and randomly assigning locations and connections. > > > > No doubt about that. Just curious. I did some work involving some very > simple probes, but it was more of a theoretical simulation. We'll see how > close theory matches reality I guess ;) Obviously, opennet will have uptime issues (but hopefully they will settle after a reasonable period), and darknet presumably won't... -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120620/b8f416d5/attachment.pgp>
[freenet-dev] Statistics Project Update #1
On Thursday 10 May 2012 01:08:45 Evan Daniel wrote: > On Tue, May 1, 2012 at 7:48 AM, Zlatin Balevsky wrote: > >> On 04/28/2012 06:56 PM, Zlatin Balevsky wrote: > >>> In Gnutella we observed that long-lived nodes tend to be better > >>> connected and that they also cluster with other high-uptime nodes. > >>> If the same is true for Freenet it's a good idea to keep an eye for > >>> side effects as you tweak the behavior. > >> > >> Good to know - I'll look for that. Are there any particular effects > >> you had in mind? The Metropolis-Hastings correction in the new probes > >> should produce a fairly uniform distribution of endpoints despite > >> clustering and well-connected nodes, but explicitly simulating the > >> effects of high uptime could be helpful. > > > > There was a study that higher uptime correlated with the probability > > of further uptime so if you shift bias towards low-uptime nodes you > > could end will lower overall reliability. It was done on a different > > network with different usage patterns but imho you should definitely > > treat node uptime as a parameter in any simulations. > > MH should produce a good simple random sample from all nodes currently > online, provided that the walk is of sufficient length, regardless of > clustering effects. If there are partitioning effects, those will make > the required walk length to get good dispersion longer, in a way that > might be somewhat difficult to measure, but as long as the network is > not completely partitioned, a sufficient walk length will produce a > good sample. The fact that a large sample must be taken over an > extended period means that low-uptime nodes will have a somewhat > disproportionately lower chance of being in the sample (I think... > need to do math here), but isn't a huge problem. Can we tell whether the network is partitioned? I guess that's one of the key outcomes? I.e. are there barriers between different parts of the network, are there large areas with similar locations but few connections, etc? -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120620/d0a5b618/attachment.pgp>
[freenet-dev] Statistics Project Update #3
On Saturday 12 May 2012 05:23:25 Steve Dougherty wrote: > I'm still working on the initial implementation of MH-corrected probes > in Fred. I'm pretty close to having minimal functionality working, and > next week I plan to finish up the initial implementation. However, I'm > not happy with the current design, so if anyone would like to take a > look at the bleeding edge of development and possibly offer feedback, > the code is available. [1] This branch is quite rough; before submitting > a pull request or a formal request for review I will rewrite the history > onto another branch with each commit making a reasonable unit of change > along with descriptive commit messages. Which is to say this branch does > not have those properties. Linear history isn't really a problem. What is a problem is messy code and repeated reverts; rewriting it for a more sensible history is fine, but for the record, don't feel obliged if the only reason is to linearise the history. BTW having reviewed your code, it looks good, although there are various issues being dealt with. > > The probes do not have a concept of acknowledgement or two-stage > (propagated) timeouts. The possible outcomes of a probe request are > a timeout, a node participating in the probe being disconnected, (also > not propagated - only occurs for local disconnection) or a set of > results. The possible results are: Two stage timeouts probably aren't necessary for probes... > > -Swap identifier - useful for network population estimates > -Statistics on uptime: session, 48-hour percentage, and 7-day percentage > -Output bandwidth limit > -Datastore size limit > -Link lengths > -Freenet build Okay. > -Whether a key is found at the endpoint > -The number of nodes along the trace back on which a key is found I'm not sure that I follow. IMHO re keys all we need is to do a normal request - this will tell us about the key's reachability. Being able to determine its prevalence in datastores per 1000 nodes is probably overkill, and may be (mildly) dangerous (not very dangerous as probing stores is pretty easy even if you just have HTL-1 requests). > > Results like link lengths, bandwidth limit, or store size will have a > small amount of random noise added in an attempt to make them less > identifiable. The originating node requests exactly one type of result. > As the request is routed the HTL is probabilistically decremented, with > a very low probability of decrementing at HTL = 1. This is intended to > make it more difficult to interrogate directly connected nodes by > sending them probe requests with HTL = 1. Each node has a hard limit of > how many probes it can be waiting on at any point in time, and will not > respond to requests beyond that limit. Timeout is proportional to HTL. > > The probes are exposed over FCP. A tentative design for the request and > names for responses is available on the wiki. [2] Nodes will be able to > disable any number of these response types, and refusal to respond to a > probe is (intended to be) indistinguishable from a timeout. > > Comments and suggestions are very welcome! > > Thanks, > operhiem1 > > [1] https://github.com/thynix/fred-staging/tree/newProbesDev > [2] https://wiki.freenetproject.org/User:Operhiem1/MHProbeFCP -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120620/7a15e3e4/attachment.pgp>
[freenet-dev] Transport Plugins Update
On Wednesday 23 May 2012 18:00:40 Chetan Hosmani wrote: > Hello everyone, > > I have been a bit busy last week, and haven't committed much code. > Here are the following refactor changes I have implemented and pushed > to "refactor-1" branch- > > 1. Added a separate list for transport managers. So we basically have > a list which will contain a manager for every mode. Presently it has > two - opennet and darknet. > > 2. NodeCrypto functions in the following way- > a. Transport managers are created in the node irrespective of whether > opennet or darknet is running. > b. NodeCrypto on creation for every mode accesses its respective > transport manager to get a list of transports. > c. It runs the method handleNewTransport to initialise a PacketMangler > and handle each transport plugin. > d. If the list of transports is empty, then it will function using the > inherent UDPSocketHandler transport and run normally. > d. When a plugin is loaded (on registering with TransportManager), the > manager checks if the corresponding mode (opennet/darknet) exists and > notifies the NodeCrypto by calling the handleNewTransport method. > > 3. UDPSocketHandler now extends PacketTransportPlugin, and I have made > changes so that it looks like a plugin. I have built it and checked. > It runs fine. > It also registers in the transport manager. > > 4. Presently working on replacing the Session keys in the PeerNode > object. There are three - current, previous and unverified. I am not > yet well acquainted how these work. > But the idea is to have a separate connection object PeerConnection that - > a. Has a unique PeerNode > b. Has a transport associated > c. Has the corresponding Peer (detected Peer) object associated > (different transports are on different ports, and we can keep it open > to use different addresses itself) This should be fairly similar to the current code. There are some complexities, e.g. : is not necessarily valid externally; this is why we include both that and the address/port that our peers tell us is our address/port number. > d. A list of session keys. It need not be three. The code can become > more generic. > > So PeerNode has a list of PeerConnection objects, for every active > transport. The PeerNode will decide the transport to use using these > objects. The rest looks good. > > > The official coding period has started. So after a few more additions > I will merge them to "Transports" branch and begin working on a new > branch "PacketTransports". > Also awaiting some code review, after nextgens becomes free. > > For the coming week- > > 1. Work and finish handleNewTransport > 2. Finish the PeerConnection object > 3. List the changes needed for multiple transport support in PeerNode > (implementing PeerConnection list) > 4. Figure out PeerNode code to use different transports > > Later on I will mainly work on NewPacketFormat which needs to be > changed so that they are more generic for Packet Transports. Hope to > have toad's and zidel's help then. > > ArneBab - Flog is coming too. Sorry :) > > Regards, > Chetan Hosmani -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120620/382f8a20/attachment.pgp>
[freenet-dev] Some Updates on Winterface
Hey Arne, thanks for the feedback. On Wed, 2012-06-20 at 11:07 +0200, Arne Babenhauserheide wrote: > Hi Pouyan, > > Thanks for getting us up to date! > > I just updated the plugin, and it looks partially functional, now. > > - A nice idea would be to get the patch for jar loader included in freenet as > soon as possible! > > Currently the list of bookmarks looks very hectic and the cathegories are > missing. Is there any of your stuff missing? The yellow rows are supposed the category titles (not yet implemented the localization). > The changing of entry height on hover disturbs the list (makes it harder to > browse). But it already gets the list of my bookmarks. True! I'm gonna change the CSS for that. > > - It would also be nice to make the plugin basically usable as fast as > possible: Make the bookmarks clickable and add a way to retrieve files and > freesites. Next step: Downloads/Uploads and the Menu (for seeing Sone). Very well. I'm gonna implement retrieving files as my next task. > > Best wishes, > Arne Best, pausb > ___ > Devl mailing list > Devl at freenetproject.org > https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120620/a0b285dc/attachment.pgp>
[freenet-dev] Branches
Our current workflow is supposed to be based on this:
http://nvie.com/posts/a-successful-git-branching-model/
Consequently:
- next is where we merge stuff ("development")
- review has been used as a temporary area for stuff which needs reviewing
before merging to next
- We will have release branches for merging and testing towards a new release.
- "testing" is pointless: We use each release branch for generating testing
snapshots. Those who want to be on the bleeding edge can build next.
Unless we want to have a permanent testing branch for testing
somewhat-fully-assembled-but-unstable scary stuff?
Anyway the main point is the model above *DOES* include release branches, and I
propose that whoever is doing release management uses them. That means me or
Bombe.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20120620/370d8e32/attachment.pgp>
[freenet-dev] Some Updates on Winterface
Hi Pouyan, Thanks for getting us up to date! I just updated the plugin, and it looks partially functional, now. - A nice idea would be to get the patch for jar loader included in freenet as soon as possible! Currently the list of bookmarks looks very hectic and the cathegories are missing. The changing of entry height on hover disturbs the list (makes it harder to browse). But it already gets the list of my bookmarks. - It would also be nice to make the plugin basically usable as fast as possible: Make the bookmarks clickable and add a way to retrieve files and freesites. Next step: Downloads/Uploads and the Menu (for seeing Sone). Best wishes, Arne
[freenet-dev] Statistics Project Update #4
> > On 06/19/2012 04:10 PM, Matthew Toseland wrote: > > However, DoS protection should be a little stronger than has been > > discussed: You should limit the average number of probes on a > > given link per unit time, like we do with swapping. This should > > probably be an average, and should be generous enough that it isn't > > going to be violated by accident, but it's preferable to having a > > limit on in-flight probes, as it will quench any flood more or less > > at source, and the attacker will be limited by the number of > > connections he has (at least on darknet, connections are > > expensive). > > The number of probes accepted per peer is limited with a counter which > increments when a request is accepted, decrements 60 seconds later, > and has a maximum (currently 10) above which no more requests are > accepted from that peer. Is my understanding correct that this is an > acceptable way to implement per-link limits? > You can also use probes to detect DoS (or plain old bugs in the probe implementation) if probes report back 1 the average number of outstanding probes across all peer nodes 2 the count or fraction of nodes which are at the maximum 10. That information can be used for all kinds of other research as well. -- next part -- An HTML attachment was scrubbed... URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120620/a03046db/attachment.html>
[freenet-dev] Statistics Project Update #4
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/19/2012 04:10 PM, Matthew Toseland wrote: > However, DoS protection should be a little stronger than has been > discussed: You should limit the average number of probes on a > given link per unit time, like we do with swapping. This should > probably be an average, and should be generous enough that it isn't > going to be violated by accident, but it's preferable to having a > limit on in-flight probes, as it will quench any flood more or less > at source, and the attacker will be limited by the number of > connections he has (at least on darknet, connections are > expensive). The number of probes accepted per peer is limited with a counter which increments when a request is accepted, decrements 60 seconds later, and has a maximum (currently 10) above which no more requests are accepted from that peer. Is my understanding correct that this is an acceptable way to implement per-link limits? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJP4T/9AAoJECLJP19KqmFu0XUP/RE27kkNKT7tIZxQLZYoClOx WMmjNRjuBHx5VPglOiMQs5Bbaa7ZucM91zu0AWW6bJZkUSdX0Y7gXDInbwExR6Cs NxO5t3lkSz2fst6GjsdjVcWgaS2ZGe/RUwll8X9UEwspQeKkm0utRXaWb+3hIJx7 ydU4uQ5/P94znMW2MqwCCaKgcDAXUmY/QxrPdrNGGult/OuhyoDHlxuRgbuJGe6c tLTc+n84A5HLjieDN6vqCrGBU7I5poLIoQ7kuDjXTCPFxOEFJN0khXEF9m+c4COo Qm1l/9Vf+i5bzG1SkAnPQ410fKFG9OgjJUhgYbQX2avWv4xwKpbiTOCK7RW0OfKs gmQsVBl0s3kg3k0qkkvoFIZ22xtwjKdTh5bbmJVGMyr1fsRXbPAuW8fwjEH1nQpQ BWoQpCjAYDkKUr0vqr2o3pQGisDb9NPUBXMiHXZKyYeGJnYFZP84bpdNcrvCs3H+ s7josC7WVYBufRAvC5RkegERcLEz3hfH9YIwp6JLJbT6L0f4ZTvuUw0El2kW6RQU 0gFhbAVsNCm/aJ0S8YIyyknqllg0CpMh7OTC/rbG18NNxnJbD3zbilggoxXMx0td y/ncdusnwAxhP2a+K/bOqjFvmDkT4o9G/4Ror4WCm2uQz93Q/3MfTQlNbsXuCwt3 44rBLUfCklykwf/vYfNB =QUS9 -END PGP SIGNATURE-
Re: [freenet-dev] Some Updates on Winterface
Hi Pouyan, Thanks for getting us up to date! I just updated the plugin, and it looks partially functional, now. - A nice idea would be to get the patch for jar loader included in freenet as soon as possible! Currently the list of bookmarks looks very hectic and the cathegories are missing. The changing of entry height on hover disturbs the list (makes it harder to browse). But it already gets the list of my bookmarks. - It would also be nice to make the plugin basically usable as fast as possible: Make the bookmarks clickable and add a way to retrieve files and freesites. Next step: Downloads/Uploads and the Menu (for seeing Sone). Best wishes, Arne ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Statistics Project Update #4
On 06/19/2012 04:10 PM, Matthew Toseland wrote: However, DoS protection should be a little stronger than has been discussed: You should limit the average number of probes on a given link per unit time, like we do with swapping. This should probably be an average, and should be generous enough that it isn't going to be violated by accident, but it's preferable to having a limit on in-flight probes, as it will quench any flood more or less at source, and the attacker will be limited by the number of connections he has (at least on darknet, connections are expensive). The number of probes accepted per peer is limited with a counter which increments when a request is accepted, decrements 60 seconds later, and has a maximum (currently 10) above which no more requests are accepted from that peer. Is my understanding correct that this is an acceptable way to implement per-link limits? You can also use probes to detect DoS (or plain old bugs in the probe implementation) if probes report back 1 the average number of outstanding probes across all peer nodes 2 the count or fraction of nodes which are at the maximum 10. That information can be used for all kinds of other research as well. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Transport Plugins Update
On Wednesday 23 May 2012 18:00:40 Chetan Hosmani wrote: Hello everyone, I have been a bit busy last week, and haven't committed much code. Here are the following refactor changes I have implemented and pushed to refactor-1 branch- 1. Added a separate list for transport managers. So we basically have a list which will contain a manager for every mode. Presently it has two - opennet and darknet. 2. NodeCrypto functions in the following way- a. Transport managers are created in the node irrespective of whether opennet or darknet is running. b. NodeCrypto on creation for every mode accesses its respective transport manager to get a list of transports. c. It runs the method handleNewTransport to initialise a PacketMangler and handle each transport plugin. d. If the list of transports is empty, then it will function using the inherent UDPSocketHandler transport and run normally. d. When a plugin is loaded (on registering with TransportManager), the manager checks if the corresponding mode (opennet/darknet) exists and notifies the NodeCrypto by calling the handleNewTransport method. 3. UDPSocketHandler now extends PacketTransportPlugin, and I have made changes so that it looks like a plugin. I have built it and checked. It runs fine. It also registers in the transport manager. 4. Presently working on replacing the Session keys in the PeerNode object. There are three - current, previous and unverified. I am not yet well acquainted how these work. But the idea is to have a separate connection object PeerConnection that - a. Has a unique PeerNode b. Has a transport associated c. Has the corresponding Peer (detected Peer) object associated (different transports are on different ports, and we can keep it open to use different addresses itself) This should be fairly similar to the current code. There are some complexities, e.g. our detected IP:our nominal port number is not necessarily valid externally; this is why we include both that and the address/port that our peers tell us is our address/port number. d. A list of session keys. It need not be three. The code can become more generic. So PeerNode has a list of PeerConnection objects, for every active transport. The PeerNode will decide the transport to use using these objects. The rest looks good. The official coding period has started. So after a few more additions I will merge them to Transports branch and begin working on a new branch PacketTransports. Also awaiting some code review, after nextgens becomes free. For the coming week- 1. Work and finish handleNewTransport 2. Finish the PeerConnection object 3. List the changes needed for multiple transport support in PeerNode (implementing PeerConnection list) 4. Figure out PeerNode code to use different transports Later on I will mainly work on NewPacketFormat which needs to be changed so that they are more generic for Packet Transports. Hope to have toad's and zidel's help then. ArneBab - Flog is coming too. Sorry :) Regards, Chetan Hosmani signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Statistics Project Update #3
On Saturday 12 May 2012 05:23:25 Steve Dougherty wrote: I'm still working on the initial implementation of MH-corrected probes in Fred. I'm pretty close to having minimal functionality working, and next week I plan to finish up the initial implementation. However, I'm not happy with the current design, so if anyone would like to take a look at the bleeding edge of development and possibly offer feedback, the code is available. [1] This branch is quite rough; before submitting a pull request or a formal request for review I will rewrite the history onto another branch with each commit making a reasonable unit of change along with descriptive commit messages. Which is to say this branch does not have those properties. Linear history isn't really a problem. What is a problem is messy code and repeated reverts; rewriting it for a more sensible history is fine, but for the record, don't feel obliged if the only reason is to linearise the history. BTW having reviewed your code, it looks good, although there are various issues being dealt with. The probes do not have a concept of acknowledgement or two-stage (propagated) timeouts. The possible outcomes of a probe request are a timeout, a node participating in the probe being disconnected, (also not propagated - only occurs for local disconnection) or a set of results. The possible results are: Two stage timeouts probably aren't necessary for probes... -Swap identifier - useful for network population estimates -Statistics on uptime: session, 48-hour percentage, and 7-day percentage -Output bandwidth limit -Datastore size limit -Link lengths -Freenet build Okay. -Whether a key is found at the endpoint -The number of nodes along the trace back on which a key is found I'm not sure that I follow. IMHO re keys all we need is to do a normal request - this will tell us about the key's reachability. Being able to determine its prevalence in datastores per 1000 nodes is probably overkill, and may be (mildly) dangerous (not very dangerous as probing stores is pretty easy even if you just have HTL-1 requests). Results like link lengths, bandwidth limit, or store size will have a small amount of random noise added in an attempt to make them less identifiable. The originating node requests exactly one type of result. As the request is routed the HTL is probabilistically decremented, with a very low probability of decrementing at HTL = 1. This is intended to make it more difficult to interrogate directly connected nodes by sending them probe requests with HTL = 1. Each node has a hard limit of how many probes it can be waiting on at any point in time, and will not respond to requests beyond that limit. Timeout is proportional to HTL. The probes are exposed over FCP. A tentative design for the request and names for responses is available on the wiki. [2] Nodes will be able to disable any number of these response types, and refusal to respond to a probe is (intended to be) indistinguishable from a timeout. Comments and suggestions are very welcome! Thanks, operhiem1 [1] https://github.com/thynix/fred-staging/tree/newProbesDev [2] https://wiki.freenetproject.org/User:Operhiem1/MHProbeFCP signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Statistics Project Update #1
On Thursday 10 May 2012 01:08:45 Evan Daniel wrote: On Tue, May 1, 2012 at 7:48 AM, Zlatin Balevsky zlat...@gmail.com wrote: On 04/28/2012 06:56 PM, Zlatin Balevsky wrote: In Gnutella we observed that long-lived nodes tend to be better connected and that they also cluster with other high-uptime nodes. If the same is true for Freenet it's a good idea to keep an eye for side effects as you tweak the behavior. Good to know - I'll look for that. Are there any particular effects you had in mind? The Metropolis-Hastings correction in the new probes should produce a fairly uniform distribution of endpoints despite clustering and well-connected nodes, but explicitly simulating the effects of high uptime could be helpful. There was a study that higher uptime correlated with the probability of further uptime so if you shift bias towards low-uptime nodes you could end will lower overall reliability. It was done on a different network with different usage patterns but imho you should definitely treat node uptime as a parameter in any simulations. MH should produce a good simple random sample from all nodes currently online, provided that the walk is of sufficient length, regardless of clustering effects. If there are partitioning effects, those will make the required walk length to get good dispersion longer, in a way that might be somewhat difficult to measure, but as long as the network is not completely partitioned, a sufficient walk length will produce a good sample. The fact that a large sample must be taken over an extended period means that low-uptime nodes will have a somewhat disproportionately lower chance of being in the sample (I think... need to do math here), but isn't a huge problem. Can we tell whether the network is partitioned? I guess that's one of the key outcomes? I.e. are there barriers between different parts of the network, are there large areas with similar locations but few connections, etc? signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Statistics Project Update #1
On Tuesday 01 May 2012 00:27:26 Michael Grube wrote: On Mon, Apr 30, 2012 at 7:06 PM, Steve Dougherty st...@asksteved.comwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/28/2012 06:56 PM, Zlatin Balevsky wrote: In Gnutella we observed that long-lived nodes tend to be better connected and that they also cluster with other high-uptime nodes. If the same is true for Freenet it's a good idea to keep an eye for side effects as you tweak the behavior. Good to know - I'll look for that. Are there any particular effects you had in mind? The Metropolis-Hastings correction in the new probes should produce a fairly uniform distribution of endpoints despite clustering and well-connected nodes, but explicitly simulating the effects of high uptime could be helpful. It occurs to me that the probe requests I hope to depreciate allow reconstructing the actual network topology - perhaps we could run simulations on top of it? The new probe requests are currently planned to not report degrees or link length distributions, [1] which as far as I can tell would mean no way to reconstruct the network as measured in simulation. Does it seem reasonable to omit the ability to gather such information? I don't know, in the short run. In the long run we'd prefer not to expose the full topology for security reasons. On 04/28/2012 03:51 PM, Michael Grube wrote: Are you assuming opennet, darknet, a mix? The simulation generates the graph in a way which effectively sort of assumes darknet: it assigns locations, then iterates over the network and connects nodes based on link length. [2] [3] I'm working under the assumption that when using the same degree distribution as the network is measured to have this is an accurate enough approximation. I will include these and similar plots in this week's progress report. Ah, ok. You're making a Kleinberg Small World graph. Nice. My understanding is that a more thorough simulation of darknet would randomly assign locations and reassign them with the location-swapping algorithm used in Fred. Right. Funnily enough, the swapping algorithm is most concisely described in the Pitch Black paper. I'd suggest that as a reference for quick implementation. I think simulating opennet would be more work both to implement and to run, as it would mean implementing path-folding and randomly assigning locations and connections. No doubt about that. Just curious. I did some work involving some very simple probes, but it was more of a theoretical simulation. We'll see how close theory matches reality I guess ;) Obviously, opennet will have uptime issues (but hopefully they will settle after a reasonable period), and darknet presumably won't... signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
