[freenet-dev] Statistics Project Update #4
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/21/2012 07:46 PM, Zlatin Balevsky wrote: > Are these rate-limited in any way? In what ways could I do effective rate-limiting? Would it work to keep track of when the last request was received from each peer, and send a rate limiting error when the last request was too recent and the last rate error sent to that peer was long enough ago? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJPuu7JAAoJECLJP19KqmFurfgP/1E4S/bdp9gVyUWmH02z6Ccu BTtwoSzR9Iw2c3y1UrvLvQhinCrqmuxzG7dcidU3MmfvQvbGVOUFqZZguTGhoPcj bjoJAZgYkwR/398c4ople2HKcyuAAbISg8HHlQ2Y+MV4XBVqaLSgncrLVbuS3Qg5 Fx+mQZ9L2OrPzTKbNvgdk+P1sc2BCo2sHp6s6gElyAuaPDsyClQKDn64qg4FfpxU R9SweaN4H2qaGEhRjNBy4gm9tdX3Yv0Gvbkr66EkH0bbtcNHg92d6Hn3Xu/wn3Tg gmF/p5XVLcUmgbGUkd0ziOArHGVWHBxybNB5tg3GtBC+EAA4FKgn4sP9j4rEbpKY MQCFbbzpOQPL1uS7YGC1Ut19dfPp03GTQzAxqDrSJ0kPLjwSzHFzaPJSLEh0VJFT ONfQxaLHgrdUyQo5RAekl6ra94TyWExTrRbMBQuvp5gTiSufvxpH3pdZRb3WJGw7 Ok+ZYlxm+K7muixOGBFCSOz1Zdx78xJ9aLqb+lZi2Bj+0WW0kRjvW9UYh4x6pg98 GMydDQosUYxHkBcctLgjVbjEF0cr3soEI1yTOmJ0KRYgn0h+Cip25rhgVzRPMYn0 qxUl4xvPkXhxKcbkRy8Pxc1JHUMyNh+IammM2l5CiZw0k6TFSDT43rMjEwBwsSnJ t3UhSHnew4maBVlL9Bnv =psgK -END PGP SIGNATURE-
[freenet-dev] Statistics Project Update #4
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/21/2012 07:46 PM, Zlatin Balevsky wrote: > Are these rate-limited in any way? Do they obey some general > rate-limiting policies together with other messages? They're limited in that each node will only accept 5 simultaneous probes. For these purposes, probes with equal randomly assigned (in the current code; an adversary could do something different) UIDs are considered to be the same probe. I haven't added any rate limiting myself. If something at the NodeDispatcher level or below has rate limiting for all messages which pass through them then the probes would have it. In checking just now I didn't notice any, but I don't know. > Does fred have any metrics collection system that would allow us to > detect such flooding events? Perhaps I should add a metric like "probe requests accepted in the past hour"? All that occurs to me is logging MHProbe at debug, grepping the logs for "Accepting probe with uid ", counting the lines. > IMHO maximum HTL should be single-digits or you risk flooding. I don't follow. What do you mean? At each hop a probe either stays locally (if a candidate is not accepted) or is routed to another random node. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJPutsVAAoJECLJP19KqmFuf9cQAKPlMhqZ6iYPBORXaYQEAKak Dq8jM4IZVJTv6qpxwgGzEUre6j2SyHU8l6p0ga7EY/A5xLXT8mrc8ykuUWqUnb3d l9ZYqBp++BADrvbL+ETIXXzbD9RjR4uM+HWdVliC8INGoPHUOjZgLuQm7TFMLVI1 Gu4FulIoc1f+9VE+Gai+y+LulrXwplnOSwL8RaKijFJfyDPGCqTNK6JjUV1G7zsw OR7txfcoWv2S5qJkqCW2Kjgfnb9wfvOWZaLz++26OlxWdkQ24GxcBtpH3WEUM1WT HyIwRwEuk/TRsEOkf794LTxpHxtzmbhfkjsogng99u5RL8JfKYeP71Gdj12hiCwr dzO/4fAwsozZaVF7NGqkVGHDJamHfkEHksjv0y90dR5kDD5qi3X/54N3ZvrnlTxi n8DEZi78RlQxdZUngCdC6ZaEF38bJzP5wjqN1deyZ9Lthb1Rx2BfaS5UbTomcN5r EER70mH2Hf6tBxPnT60fZoHw74B02PtL7PrDk7Hpo0MzMWKliI8JEVO9iK9dUNCh bVyLQ9zlWpcJ9uziGsycg7bA4yNUvdWXA3hS2H2LG2x1n6OnQj0IEi+veKT5sJsg uD1yb3Pcxb6MRuRKaz0qPhUzsBpTlQfUot4ihdzQ9kTGy6M1/LIuNAjhezdNjp7M tHiGQsoVbNtr1lHD31Pb =r5oe -END PGP SIGNATURE-
[freenet-dev] Statistics Project Update #4
Are these rate-limited in any way? Do they obey some general rate-limiting policies together with other messages? IMHO maximum HTL should be single-digits or you risk flooding. Does fred have any metrics collection system that would allow us to detect such flooding events? On Sat, May 19, 2012 at 5:46 PM, Steve Dougherty wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > I've finished a proof of concept for Metropolis-Hastings corrected > probes in Fred. If you are so inclined, please offer feedback on the > patches. [1] Metropolis-Hastings correction was described previously, > [2] and the probes are exposed over FCP. [3] An originating node starts > a probe request by specifying the type of result they're looking for, > and optionally a value for hopsToLive, which currently defaults to its > maximum of 50 if not specified. We will in practice initially use > something closer to 30. One can disable responding to any number of > result types on the Core settings page. > > Possible types are: > > BANDWIDTH - outgoing bandwidth limit. > BUILD - Freenet build number. (For instance, currently 1407.) Useful > ? ? ? ?for measuring update propagation. > IDENTIFIER - endpoint identifier. Useful for improving estimates of > ? ? ? ? ? ? network size and churn. > LINK_LENGTHS - differences in location between endpoint and each of > ? ? ? ? ? ? ? endpoint's peers. > STORE_SIZE - size of datastore. > UPTIME - session uptime and the percentage of the past 48 hours during > ? ? ? ? which the endpoint was up. > > +/- up to 1% noise is added to bandwidth, link lengths, store size, and > uptime information in an attempt to make the information less > identifiable but still useful for analysis. > > Bandwidth, link lengths, store size, and uptime are intended to improve > knowledge of the network so that any problems will hopefully become > apparent, or at the very least allow more detailed simulations which > more accurately reflect actual network conditions. This is with the > goal of implementing changes to improve network performance and > reliability. > > Possible outcomes of the probes as implemented are: > ? ?-non-propagated timeout > ? ?-non-propagated disconnection > ? ?-endpoint refusal for the result type > ? ?-result from endpoint > > Based on feedback in #freenet I've built a TODO list: > > 1. Access MHProbe from the callbacks, which are inner classes, via > MHProbe.this. This will mean no need to pass it in to use MHProbe as a > ByteCounter. Also because the callbacks are inner classes there is no > need for the horrible hack that is making pendingProbes public static. > 2. If FOAF data for a peer is not available, ignore that peer. > 3. Implement error messages so that resources can be freed quickly in > the event on an error: > ? ?- next in chain disconnected > ? ?- timeout > ? ?- type unrecognized > ? ?- FOAF disabled > 4. If FOAF is completely disabled on a node, rather than ignore every > peer, leading to HTL decrementing until a local response, respond with > a "FOAF disabled" error message as above. > 5. Add a node configuration option for a random (not based on noderef > like current probe and swap request identifier) identifier which is set > to a random value when at its default of -1. > 6. Use a Gaussian distribution for randomNoise() because many tests > assume Gaussian noise, so why not? > 7. Quantize things: instead of just adding noise, report bandwidth in > as 64-bit integer KiB and store size as 64-bit integer GiB. > 8. Instead of separate identifier and uptime requests, allow requesting: > ? ?- [ identifier, quantized noisy 7-day uptime percentage ] > ? ?- [ noisy 7-day uptime percentage ] > ? ?- [ noisy 48-hour uptime percentage ] > 9. Set timeout as constant1 * HTL + constant2 to account for > probabilistic decrement at HTL = 1. > > Next week I plan to address the TODO list and any more issues that come > up in the patch set, then submit it for merging into Fred once it has > been approved. After that, I will work on FCP library support for the > new probes, and update pyProbe to support gathering and analyzing the > new information. In the event that I finish these things I can work on > making the network simulator more flexible. Our target deployment date > is Friday the 25th May or the day after. > > Thanks, > operhiem1 > > [1] https://github.com/thynix/fred-staging/tree/newProbes > [2] https://emu.freenetproject.org/pipermail/devl/2012-April/036363.html > [3] https://wiki.freenetproject.org/User:Operhiem1/MHProbeFCP > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.11 (GNU/Linux) > > iQIcBAEBAgAGBQJPuBSaAAoJECLJP19KqmFurkUQAJZbvpgctvsZxgppkRIW/ZOY > meNolQJCutEUHl8VFSdut/6ahRwJ3gfdCgOiF/GHcbEkZqm4S+Jna58Sz6E9WBSa > MfhjQ1dULutiE1vmXN/5v/vNwFYXlATAk2XK35IrUqbCorq7NZHQhSyn61j6fLdB > cGIBOV8GCRHd2rhFu4+oP9tECyoBbL7B9OqjI3AHPEuDEyvFVfuxfk6HDcOam+4Z > dzeecRUMU4T1wquOZ9l6sDTqe8m+RRuR++WHObQ/pI/n1s848HWTM0qs3Dzj3pIU >
Re: [freenet-dev] Statistics Project Update #4
Are these rate-limited in any way? Do they obey some general rate-limiting policies together with other messages? IMHO maximum HTL should be single-digits or you risk flooding. Does fred have any metrics collection system that would allow us to detect such flooding events? On Sat, May 19, 2012 at 5:46 PM, Steve Dougherty st...@asksteved.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I've finished a proof of concept for Metropolis-Hastings corrected probes in Fred. If you are so inclined, please offer feedback on the patches. [1] Metropolis-Hastings correction was described previously, [2] and the probes are exposed over FCP. [3] An originating node starts a probe request by specifying the type of result they're looking for, and optionally a value for hopsToLive, which currently defaults to its maximum of 50 if not specified. We will in practice initially use something closer to 30. One can disable responding to any number of result types on the Core settings page. Possible types are: BANDWIDTH - outgoing bandwidth limit. BUILD - Freenet build number. (For instance, currently 1407.) Useful for measuring update propagation. IDENTIFIER - endpoint identifier. Useful for improving estimates of network size and churn. LINK_LENGTHS - differences in location between endpoint and each of endpoint's peers. STORE_SIZE - size of datastore. UPTIME - session uptime and the percentage of the past 48 hours during which the endpoint was up. +/- up to 1% noise is added to bandwidth, link lengths, store size, and uptime information in an attempt to make the information less identifiable but still useful for analysis. Bandwidth, link lengths, store size, and uptime are intended to improve knowledge of the network so that any problems will hopefully become apparent, or at the very least allow more detailed simulations which more accurately reflect actual network conditions. This is with the goal of implementing changes to improve network performance and reliability. Possible outcomes of the probes as implemented are: -non-propagated timeout -non-propagated disconnection -endpoint refusal for the result type -result from endpoint Based on feedback in #freenet I've built a TODO list: 1. Access MHProbe from the callbacks, which are inner classes, via MHProbe.this. This will mean no need to pass it in to use MHProbe as a ByteCounter. Also because the callbacks are inner classes there is no need for the horrible hack that is making pendingProbes public static. 2. If FOAF data for a peer is not available, ignore that peer. 3. Implement error messages so that resources can be freed quickly in the event on an error: - next in chain disconnected - timeout - type unrecognized - FOAF disabled 4. If FOAF is completely disabled on a node, rather than ignore every peer, leading to HTL decrementing until a local response, respond with a FOAF disabled error message as above. 5. Add a node configuration option for a random (not based on noderef like current probe and swap request identifier) identifier which is set to a random value when at its default of -1. 6. Use a Gaussian distribution for randomNoise() because many tests assume Gaussian noise, so why not? 7. Quantize things: instead of just adding noise, report bandwidth in as 64-bit integer KiB and store size as 64-bit integer GiB. 8. Instead of separate identifier and uptime requests, allow requesting: - [ identifier, quantized noisy 7-day uptime percentage ] - [ noisy 7-day uptime percentage ] - [ noisy 48-hour uptime percentage ] 9. Set timeout as constant1 * HTL + constant2 to account for probabilistic decrement at HTL = 1. Next week I plan to address the TODO list and any more issues that come up in the patch set, then submit it for merging into Fred once it has been approved. After that, I will work on FCP library support for the new probes, and update pyProbe to support gathering and analyzing the new information. In the event that I finish these things I can work on making the network simulator more flexible. Our target deployment date is Friday the 25th May or the day after. Thanks, operhiem1 [1] https://github.com/thynix/fred-staging/tree/newProbes [2] https://emu.freenetproject.org/pipermail/devl/2012-April/036363.html [3] https://wiki.freenetproject.org/User:Operhiem1/MHProbeFCP -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJPuBSaAAoJECLJP19KqmFurkUQAJZbvpgctvsZxgppkRIW/ZOY meNolQJCutEUHl8VFSdut/6ahRwJ3gfdCgOiF/GHcbEkZqm4S+Jna58Sz6E9WBSa MfhjQ1dULutiE1vmXN/5v/vNwFYXlATAk2XK35IrUqbCorq7NZHQhSyn61j6fLdB cGIBOV8GCRHd2rhFu4+oP9tECyoBbL7B9OqjI3AHPEuDEyvFVfuxfk6HDcOam+4Z dzeecRUMU4T1wquOZ9l6sDTqe8m+RRuR++WHObQ/pI/n1s848HWTM0qs3Dzj3pIU GNcgX4dbgH8yz3veInur2i6FSkM41IdwEbV3OHro+peOIxVmCJ3c6cEbkEYLQW/R 1mICM7+6CLNkzjrxSwiPNrhVYGGKrYy7F4YSDj2GWwwfwd5aQaxzCM5CO9+VaHWj
Re: [freenet-dev] Statistics Project Update #4
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/21/2012 07:46 PM, Zlatin Balevsky wrote: Are these rate-limited in any way? In what ways could I do effective rate-limiting? Would it work to keep track of when the last request was received from each peer, and send a rate limiting error when the last request was too recent and the last rate error sent to that peer was long enough ago? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJPuu7JAAoJECLJP19KqmFurfgP/1E4S/bdp9gVyUWmH02z6Ccu BTtwoSzR9Iw2c3y1UrvLvQhinCrqmuxzG7dcidU3MmfvQvbGVOUFqZZguTGhoPcj bjoJAZgYkwR/398c4ople2HKcyuAAbISg8HHlQ2Y+MV4XBVqaLSgncrLVbuS3Qg5 Fx+mQZ9L2OrPzTKbNvgdk+P1sc2BCo2sHp6s6gElyAuaPDsyClQKDn64qg4FfpxU R9SweaN4H2qaGEhRjNBy4gm9tdX3Yv0Gvbkr66EkH0bbtcNHg92d6Hn3Xu/wn3Tg gmF/p5XVLcUmgbGUkd0ziOArHGVWHBxybNB5tg3GtBC+EAA4FKgn4sP9j4rEbpKY MQCFbbzpOQPL1uS7YGC1Ut19dfPp03GTQzAxqDrSJ0kPLjwSzHFzaPJSLEh0VJFT ONfQxaLHgrdUyQo5RAekl6ra94TyWExTrRbMBQuvp5gTiSufvxpH3pdZRb3WJGw7 Ok+ZYlxm+K7muixOGBFCSOz1Zdx78xJ9aLqb+lZi2Bj+0WW0kRjvW9UYh4x6pg98 GMydDQosUYxHkBcctLgjVbjEF0cr3soEI1yTOmJ0KRYgn0h+Cip25rhgVzRPMYn0 qxUl4xvPkXhxKcbkRy8Pxc1JHUMyNh+IammM2l5CiZw0k6TFSDT43rMjEwBwsSnJ t3UhSHnew4maBVlL9Bnv =psgK -END PGP SIGNATURE- ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
