[freenet-dev] Re: A really easy way to defeat datastore probing timing attacks?
Tom Kaitchuck wrote: On Wednesday 05 November 2003 10:23 pm, Toad wrote: On Thu, Nov 06, 2003 at 03:58:44AM +, Toad wrote: Thus the requesting node would not have a convincing case that we had the data in our store before they asked. Now, the question is, how important is our plausible deniability with respect to whether content was in our store before the request came in? Parts of freenet are designed to help this - for example the probabilistically not decrementing hops to live when HTL is low; but there are several ways to get around all these protections - requests at HTL 0 with timing information, announcements at HTL 0, and so on. If it is important, it may be possible to do something about it; if it is not, that may also make things easier. So, is it important? Wouldn't sending such a signal to an upstream node, be the same as telling it "I have this data, and am about to return it." If so, that would be far more incriminating than a timing attack. None the less I would think it would be possible to make it harder by somewhat deprioritizing the return of data on your node. It might be easier to just say, my next trailer will be done in less than Xms. I'll return the data when that completes. Although admittedly, sometimes it still might return too fast to be from another node. What about sending an insert request instead, followed by a data request when the insert is finished? Then, it's no more incriminating than an insert + data-request to double-check that it's there. And/Or, what about just using our estimate of how long it will take the next node to return the key to decide how long we should delay before returning the key? This time would also be randomized to hinder a variance analysis (where the attacker notices that tTransferRate varies more or less than would be expected if we weren't actually the datasource). Of course, for this to work, we need to have realistic estimates of tTransferRate and tSearchSuccess, instead of the overly-optimistic ones we've seen recently (part of this fix would be by using an estimate that doesn't include the initial optimistic one). Then, we don't waste bandwidth between our node and the next one by sending/receiving the key in question. We just make sure to be doing some sort of transfer on another key at the same time. Would that work? I'm guessing here a bit since I don't fully understand. :( On Thu, Nov 06, 2003 at 03:58:44AM +, Toad wrote: I suggest that with premix routing and a transient client-level cache, determining whether something is in the local datastore is not a big deal, because the store is only doing it's job as part of the cache. Is this reasonable? Ya, fwiw, that's what I was thinking. In which case although it doesn't have direct consequences for routing, it could have other consequences, for example, once a node is announced, it could ask other nodes for data to fill its store with near its initial announcement key - an extension of the initial request system we have now, where an announcement includes a few keys from each node in the announcement chain, near to the announcement point, to get the new node started, but we would extend it so that instead of starting with a few keys it starts with as many as it wants. Of course there are load issues and so on to be resolved in this (it could be a low priority request) - but the most obvious thing is that it would make datastore probing easy, even if implemented with a request terminate probability rather than an HTL; but if datastore probing doesn't matter, then this is fine. I'm not sure what you are trying to say here. Are you saying that this additional layer of protection allows us to safetly send more keys to an announcincg node? What is your point here? Premix routing is good. indeed... someday... ya'll better start contributing some code or ca$h. :) -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: A really easy way to defeat datastore probing timing attacks?
Toad wrote: Ok, I had another crazy idea. This one is an idea to defeat the timing attacks we all know and hate. The attack goes like this: Request a file at HTL 0 from a node. The node has a 30% chance of forwarding the request to another node, if it doesn't have the file. The node returns the file quickly, and we conclude that the node must have the file in the store. Solutions to this basically involve making it look like the node didn't have the file. One avenue of approach would be to try to emulate the slowness expected if we got the file from the next node on the chain. My suggestion: use the next node on the chain to emulate the latency of the next node on the chain! Send a special kind of request to the next node on the chain that requires it to route the request, return the datasource it would have routed to, and return the data we are sending to it. We would then send it back to the requestor in the usual way. So the additional time spent would be: tSendKeyToNextNode + tNextNodeRoutesRequest + tNextNodeSendsKeyBackToUs? ...where I guess tNextNodeRoutesRequest=0 if the next node actually already has the key. Please clarify. -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] A really easy way to defeat datastore probing timing attacks?
On Wednesday 05 November 2003 10:23 pm, Toad wrote: > On Thu, Nov 06, 2003 at 03:58:44AM +, Toad wrote: > > Thus the requesting node would not have a convincing case that we had > > the data in our store before they asked. Now, the question is, how > > important is our plausible deniability with respect to whether content > > was in our store before the request came in? Parts of freenet are > > designed to help this - for example the probabilistically not > > decrementing hops to live when HTL is low; but there are several ways to > > get around all these protections - requests at HTL 0 with timing > > information, announcements at HTL 0, and so on. If it is important, it > > may be possible to do something about it; if it is not, that may also > > make things easier. So, is it important? Wouldn't sending such a signal to an upstream node, be the same as telling it "I have this data, and am about to return it." If so, that would be far more incriminating than a timing attack. None the less I would think it would be possible to make it harder by somewhat deprioritizing the return of data on your node. It might be easier to just say, my next trailer will be done in less than Xms. I'll return the data when that completes. Although admittedly, sometimes it still might return too fast to be from another node. > I suggest that with premix routing and a transient client-level cache, > determining whether something is in the local datastore is not a big > deal, because the store is only doing it's job as part of the cache. Is > this reasonable? In which case although it doesn't have direct > consequences for routing, it could have other consequences, for example, > once a node is announced, it could ask other nodes for data to fill its > store with near its initial announcement key - an extension of the > initial request system we have now, where an announcement includes a few > keys from each node in the announcement chain, near to the announcement > point, to get the new node started, but we would extend it so that > instead of starting with a few keys it starts with as many as it wants. > Of course there are load issues and so on to be resolved in this (it > could be a low priority request) - but the most obvious thing is that it > would make datastore probing easy, even if implemented with a request > terminate probability rather than an HTL; but if datastore probing > doesn't matter, then this is fine. Premix routing is good. ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] A really easy way to defeat datastore probing timing attacks?
On Thu, Nov 06, 2003 at 03:58:44AM +, Toad wrote: > Thus the requesting node would not have a convincing case that we had > the data in our store before they asked. Now, the question is, how > important is our plausible deniability with respect to whether content > was in our store before the request came in? Parts of freenet are > designed to help this - for example the probabilistically not > decrementing hops to live when HTL is low; but there are several ways to > get around all these protections - requests at HTL 0 with timing > information, announcements at HTL 0, and so on. If it is important, it > may be possible to do something about it; if it is not, that may also > make things easier. So, is it important? I suggest that with premix routing and a transient client-level cache, determining whether something is in the local datastore is not a big deal, because the store is only doing it's job as part of the cache. Is this reasonable? In which case although it doesn't have direct consequences for routing, it could have other consequences, for example, once a node is announced, it could ask other nodes for data to fill its store with near its initial announcement key - an extension of the initial request system we have now, where an announcement includes a few keys from each node in the announcement chain, near to the announcement point, to get the new node started, but we would extend it so that instead of starting with a few keys it starts with as many as it wants. Of course there are load issues and so on to be resolved in this (it could be a low priority request) - but the most obvious thing is that it would make datastore probing easy, even if implemented with a request terminate probability rather than an HTL; but if datastore probing doesn't matter, then this is fine. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] New mailFredStatus sends outputBytesTrailingAttempted and outputBytes
Here is a new version of mailFredStatus. The main change is it
sends the last 5 hourly accumulations of outputBytesTrailingAttempted
(a new statistic) and outputBytes. Toad would like to see some of
this data.
The diagnostics sent are configurable from the command line and
are remembered if you use --writedefaults.
64cebb4d7a27cf40008b003d81c5c9bf mailFredStatus
9f6bf2ca856f2e0ec61cf3d083a98414 mailFredStatus.asc
--
Edward J. Huff <[EMAIL PROTECTED]>
#!/bin/bash
# if you edit your copy of this script, change the version.
VERSION='$Id: mailFredStatus,v 1.4 2003/11/06 03:51:13 ejhuff Exp $'
printMailHeaders() {
echo "To: $TO"
echo "From: $FROM"
echo "Errors-To: $FROM"
echo "Bcc: $BCC"
echo "Subject: Success probabilities $(date --utc)"
echo
}
printAssortedData() {
echo $VERSION
echo SERVLET=$SERVLET
date --utc
wget -O - "http://$SERVLET/servlet/nodeinfo/performance/general"; 2>/dev/null \
| tr -d "\n" \
| sed 's//@/g' \
| sed 's||@|' \
| sed 's/<[^>]*>/ /g; s/ * @/@/g; s/@ */@/g; s/ */ /g' \
| sed 's/Uptime[@ ]* /Uptime /g' \
| tr '@' "\n" \
| egrep 'Version|Build|Uptime|Current|Active' \
| grep -v 'README'
wget -O - "http://$SERVLET/servlet/nodeinfo/networking/ocm?setLevel=1&setMode=new"; 2>/dev/null \
| egrep '^[^<]' \
| sed 's/<[^>]*>/ /g; s/^ *//g; s/ */ /g; s/ / /g'
echo
echo "Network Load:"
wget -O - "http://$SERVLET/servlet/nodeinfo/networking/loadstats"; 2>/dev/null \
| sed 's/<[^>]*>/ /g; s/^ *//g; s/ */ /g; s/ / /g' \
| grep '^[A-Za-z() ]*:'
echo
echo "Frost, etc. status:"
echo "$FROST_ETC_STATUS"
echo
echo "Traffic shaping:"
echo $TC '-s qdisc show dev eth0 | head -3'
$TC -s qdisc show dev eth0 | head -3
echo Non-default non-bookmark freenet.conf lines
grep '^[a-zA-Z]' freenet.conf | grep -v bookmark
echo
echo Parameters to last startup:
ls -l --full-time freenet.last-startup
cat freenet.last-startup
echo
echo "$NETSTAT -n --inet | egrep -v :6000\|:\|\^tcp\ \*0\ \*0\ | grep ESTAB | sort -k 2nr,2 -k 3nr,3"
$NETSTAT -n --inet | head -2
$NETSTAT -n --inet | egrep -v :6000\|:\|\^tcp\ \*0\ \*0\ | grep ESTAB | sort -k 2nr,2 -k 3nr,3
echo
wget -O - http://$SERVLET/servlet/nodestatus/psuccess_data.txt 2>/dev/null
echo
wget -O - http://$SERVLET/servlet/nodeinfo/internal/env 2>/dev/null \
| grep '^' \
| sed 's/<[^>]*>/ /g; s/^ *//g; s/ */ /g' \
| egrep 'Architecture|processors|Operating|Version|JVM|Maximum|Percent|Total'
echo
wget -O - http://$SERVLET/servlet/nodestatus/nodestatus.html 2>/dev/null \
| grep '^' \
| grep -v '' \
| sed 's/<[^>]*>/ /g; s/^ *//g; s/ */ /g'
echo
for diag in $DIAGNOSTICS ; do
echo
echo "$diag"
wget -O - \
http://$SERVLET/servlet/nodestatus/diagnostics/$diag/raw \
2>/dev/null | grep '#'
wget -O - \
http://$SERVLET/servlet/nodestatus/diagnostics/$diag/raw \
2>/dev/null | grep hour | tail -6
done
date --utc
}
[ -z "$TO" ] && TO="Toad <[EMAIL PROTECTED]>"
[ -z "$FROM" ] && FROM="[EMAIL PROTECTED]:-localhost}"
[ -z "$BCC" ] && BCC="[EMAIL PROTECTED]:-localhost}"
[ -z "$INTERVAL" ] && INTERVAL=3600
[ -z "$SERVLET" ] && SERVLET=127.0.0.1:
[ -z "$FREENETHOME" ] && FREENETHOME=$HOME/freenet/
[ -z "$VERBOSE" ] && VERBOSE=
[ -z "$GOLIVE" ] && GOLIVE=
[ -z "$SENDMAIL" ] && SENDMAIL=/usr/sbin/sendmail
[ -z "$TC" ] && TC=/sbin/tc
[ -z "$NETSTAT" ] && NETSTAT=/bin/netstat
[ -z "$FROST_ETC_STATUS" ] && FROST_ETC_STATUS="Frost, etc. status unknown."
[ -z "$DIAGNOSTICS" ] && DIAGNOSTICS="requestSuccessRatio routingSuccessRatio localQueryTraffic outputBytes outputBytesTrailingAttempted"
HELP=
WRITEDEFAULTS=
if [ -f ~/.mailFredStatus ] ; then
source ~/.mailFredStatus
fi
while [[ $# > 0 ]] ; do
case "$1" in
--to)
TO=$2
shift;;
--from)
FROM=$2
shift;;
--BCC)
BCC=$2
shift;;
--interval)
INTERVAL=$2
shift;;
--servlet)
SERVLET=$2
shift;;
--FreenetHOME|--freenetHOME|--FreeNetHOME|--freenethome)
FREENETHOME=$2
shift;;
--verbose)
VERBOSE=-v
;;
--sendmail)
SENDMAIL=$2
shift;;
--tc)
TC=$2
shift;;
--netstat)
NETSTAT=$2
shift;;
--frost*)
FROST_ETC_STATUS="$2"
shift;;
--add-diag*)
DIAGNOSTICS="$DIAGNOSTICS $2"
shift;;
--set-diag*)
DIAGNOSTICS="$2"
shift;;
--dryrun)
GOLIVE=
;;
--golive)
GOLIVE=TRUE
;;
--v*|--V*|-v|-V)
echo $VERSION
echo mailFredStatus --help for help.
exit 0
;;
--write*|-w*)
WRITEDEFAULTS=TRUE
;;
--h*|-h*)
HELP=TRUE
;;
*)
HELP=TRUE
echo "$1 interpreted as --help"
;;
esac
shift
done
if [ ! -z "$WRITEDEFAULTS" ] ; then
echo "Making current options the defaults in ~/.mailFredStatus"
(cat < ~/.mailFredStatus
echo New contents of ~/.mailFredStatus:
cat ~/.mailFredStatus
echo
echo Don
[freenet-dev] A really easy way to defeat datastore probing timing attacks?
Ok, I had another crazy idea. This one is an idea to defeat the timing attacks we all know and hate. The attack goes like this: Request a file at HTL 0 from a node. The node has a 30% chance of forwarding the request to another node, if it doesn't have the file. The node returns the file quickly, and we conclude that the node must have the file in the store. Solutions to this basically involve making it look like the node didn't have the file. One avenue of approach would be to try to emulate the slowness expected if we got the file from the next node on the chain. My suggestion: use the next node on the chain to emulate the latency of the next node on the chain! Send a special kind of request to the next node on the chain that requires it to route the request, return the datasource it would have routed to, and return the data we are sending to it. We would then send it back to the requestor in the usual way. An attacker who could surveil the network at an IP layer might be able to deduce we were sending it the data and it was returning us it through traffic analysis, but that is made harder by multiplexing, a reasonable traffic level, perhaps dividing the payload into several pieces and redirecting it through other nodes before it reaches the next node in the chain (in the hope of concealing it amidst regular traffic), or well known anti-traffic-analysis measures such as traffic padding. In any case it would be significantly better than we have now, at a moderate performance cost (it's only one extra hop in the base proposal). Thus the requesting node would not have a convincing case that we had the data in our store before they asked. Now, the question is, how important is our plausible deniability with respect to whether content was in our store before the request came in? Parts of freenet are designed to help this - for example the probabilistically not decrementing hops to live when HTL is low; but there are several ways to get around all these protections - requests at HTL 0 with timing information, announcements at HTL 0, and so on. If it is important, it may be possible to do something about it; if it is not, that may also make things easier. So, is it important? Now back to your regularly scheduled debugging. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] polling & sharing estimator data
On Wed, Nov 05, 2003 at 09:09:21PM -0600, Tom Kaitchuck wrote: > On Wednesday 05 November 2003 08:36 pm, Zlatin Balevsky wrote: > > This is an extention to the idea of keeping estimator data in the .refs > > which aims to bring more consistent "worldview" between the nodes. It > > works best if there is trust implemented between nodes, but will also > > bring some limited results without trust. > > > > At random interval the node asks a random subset of the nodes in the rt > > for the estimator data they have of another node/subset of nodes (the > > two subsets can overlap). After that the node "merges" their results > > with its own estimator in a manner that would be as cancer-proof as > > possible in a network without trust. (that means removing outliers, > > averaging, etc.) > > > > Drawbacks: > > * increased network traffic. The timing will have to be chosen not to > > create a cascade effect. > > * powerful attacker with many nodes totally screws up our estimates, or > > even worse - provides such data which would make the requesting node > > favor his cancer nodes. > > > > Benefits: > > * Nodes have a more consistent picture of each other's strengths and > > weaknesses. > > * Newer nodes find their place faster > > * The network does not become any more static > > * Since topological neighbors know similar things about each other, each > > request from within the "neighborhood" will go directly to the best node. > > > > "Neighborhood" is not rigidly defined; if a network map is drawn, there > > will be no rigid boundaries between shared estimator information; rather > > it will flow gradually, providing optimal routing paths. (ok this > > sounds psycho but its much easier to visualise than explain - will try > > and get a pic/graph soon) > > Rather than inventing a new protocol for this, I would propose that when a > node tells you about some other node. IE: it's noderef comes back as the > source for some data, it should tack on it's own estimate. That way, we don't > create much more overhead. Then if we don't know the node, we can use that as > an initializer, and if we already know about it, merge that in with what we > already know. If the data does not seem to fit, then don't trust the > reporter, and preferably get several nodes oppinion on a new node before > connecting to it. That is exactly what Ian proposed. :) Well, without the merging in bit. It will get implemented soon. Details: If we don't reset the datasource, and we have an estimator for that node, we substitute our own estimator. We should perhaps not do this if our estimator doesn't have much information to go on. Hopefully this can't be abused... the other side of the question is how to prevent it being abused - the former mechanism is a start, another means is to set all the ageing data really low so that although we have an initial estimator, when we try requests through it it is as sensitive to the results as if it were initialized from scratch i.e. a flat line or a mountain. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] A couple bugs and an idea
On Fri, Oct 31, 2003 at 02:54:43PM +, Toad wrote: > On Fri, Oct 31, 2003 at 02:47:34PM +, Toad wrote: > > On Fri, Oct 31, 2003 at 02:41:35PM +, Toad wrote: > > > On Thu, Oct 30, 2003 at 04:39:22PM -0800, Tracy R Reed wrote: > > > > When a splitfile download finishes it often has a tendency to hang on "FEC > > > > decoding missing data blocks... this may take a while; please be patient." > > > > seemingly forever. I know this is a known bug to some but I'm not sure if > > > > everyone knows about it. > > > > > > > > When I try to cancel the download (usually while it is supposedly FEC > > > > decoding) so I can try again it takes forever and I have to shutdown the > > > > node to get it to stop so I can restart the download. > > > > > > > > An idea: I have 40G of ds only half of which is used after months of > > > > operation. I am on a DS-3 so my node doesn't lack for bandwidth and I > > > > always run the latest rev. This isn't good. Empty space in the ds is > > > > wasted space and hurts freenet performance. It would be helpful if there > > > > was something in fnp which allowed a node with a partially full ds to ask > > > > one of its peers to give it a key, preferably one in the requesting nodes > > > > area of specialization (isn't some specialiation info sent out in > > > > announcement requests?) but any data at all would be better than an empty > > > > datastore and having to poll keys and hope to get lucky in order to get > > > > the datastore to fill up. I don't think I have ever had pcaching kick in > > > > because I have never been able to fill my store. If everyone would fill > > > > their datastores when their node started up I bet our psuccess would > > > > increase significantly. > > > > > > You could spider Fredweb... There are security issues with a give me a > > > random file command. They were discussed some time ago... > > > > Actually, the obvious security issue is a hopsToLive problem, which is > > easily solved - just make the request drop randomly, with a fixed > > probability (say 0.05), and don't have a hopsToLive value. This makes > > the time taken by the request extremely variable, but this isn't a > > problem if it's only used to fill the store. However there may be more > > serious issues. Such a command would have been useful for certain > > whacky schemes involving XORing, that's why it was discussed last time. > > Anyway, the more serious issue: would it dilute the node's > > specialization? It probably would... Also, we may be trying to treat the > > symptoms here - what is your outgoing bandwidth? What is you maximum > > threads? What is your local and global query traffic and your psuccess? > > Maybe your node isn't getting much traffic? > > Another security issue: we probably don't want people to download all of > Freenet with sufficient bandwidth - it makes it far too easy to make > ubernodes. Actually, we already have a mechanism to fetch keys near to a given key that could be used by an attacker to eventually download all of Freenet... announcement initial requests. An attacker would set up a node and repeatedly announce. Each time he gets a random keyspace area, and a bunch of keys in that area, and a bunch of references in other nodes' routing tables. Of course he'd have to constantly change his ID to make it work - but there are 65000 odd ports available on one IP connection, and he could recycle them; he might even be able to have multiple identities on the same port. So perhaps it would be useful to have a FindClosestKeys command, and not too dangerous? Of course then you need to figure out which keys would be useful - if you are trying to fill a node's datastore with keys useful to your perceived specialization you'd need to somehow figure out what it was. Announcement provides an initial specialization key, and a bunch of keys close to it to request; you could just request keys near that if you haven't had many requests; otherwise you could request keys near the peak on your global psuccess estimator... also could be used for wierd stuff like XOR encoding. Anyway, just a thought. Probably not a very practical one - but my earlier response to some suggestions as to how to fill the DS quickly on a new node not serving many requests with an empty DS made some false assumptions, I am clarifying them. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] polling & sharing estimator data
On Wednesday 05 November 2003 08:36 pm, Zlatin Balevsky wrote: > This is an extention to the idea of keeping estimator data in the .refs > which aims to bring more consistent "worldview" between the nodes. It > works best if there is trust implemented between nodes, but will also > bring some limited results without trust. > > At random interval the node asks a random subset of the nodes in the rt > for the estimator data they have of another node/subset of nodes (the > two subsets can overlap). After that the node "merges" their results > with its own estimator in a manner that would be as cancer-proof as > possible in a network without trust. (that means removing outliers, > averaging, etc.) > > Drawbacks: > * increased network traffic. The timing will have to be chosen not to > create a cascade effect. > * powerful attacker with many nodes totally screws up our estimates, or > even worse - provides such data which would make the requesting node > favor his cancer nodes. > > Benefits: > * Nodes have a more consistent picture of each other's strengths and > weaknesses. > * Newer nodes find their place faster > * The network does not become any more static > * Since topological neighbors know similar things about each other, each > request from within the "neighborhood" will go directly to the best node. > > "Neighborhood" is not rigidly defined; if a network map is drawn, there > will be no rigid boundaries between shared estimator information; rather > it will flow gradually, providing optimal routing paths. (ok this > sounds psycho but its much easier to visualise than explain - will try > and get a pic/graph soon) Rather than inventing a new protocol for this, I would propose that when a node tells you about some other node. IE: it's noderef comes back as the source for some data, it should tack on it's own estimate. That way, we don't create much more overhead. Then if we don't know the node, we can use that as an initializer, and if we already know about it, merge that in with what we already know. If the data does not seem to fit, then don't trust the reporter, and preferably get several nodes oppinion on a new node before connecting to it. ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: Trying to eliminate a theory - request for stats
outputBytesTrailingAttempted 11/5/03 8:00:00 PM EST 1604 2773308 765024.189124798 302.23877805486285 11/5/03 9:00:00 PM EST 11313 32099592 2.03990867695729E7 318.20586935384074 11/5/03 10:00:00 PM EST 8607 27344048 4.8612403961292885E7 418.280353200883 outputBytes 11/5/03 8:00:00 PM EST 1924 4080467 1306757.6556167796 252.03846153846155 11/5/03 9:00:00 PM EST 15187 41573424 2.523062630160542E7 237.05695660762495 11/5/03 10:00:00 PM EST 7680 43798986 6.77748026180006E7 468.592578125 ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Announcements and load
One key question as we move towards 0.6 is announcements. When the network is overloaded, is precisely when we want announcements to succeed - but announcements use significant resources on the nodes on the chain, so could perhaps be used for DoS, and could easily contribute to nodes being overloaded. Now, the question is, when exactly should the node accept announcements? Another perversity: if an announcement does find a successful path, it'll probably be a chain of lightly loaded nodes that don't need the load taking off them, and won't send the new node that wants more traffic much anyway! Any suggestions? Second issue: currently we route an announcement to a maximum of 5 nodes before giving up on it (assuming we are not the first node on the chain). The corresponding number for normal requests is 40, i.e. they will traverse most of the routing table. Is there any reason for the announcement maximum routes not to be increased significantly? And the maximum number of restarts too perhaps? -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
On Wednesday 05 November 2003 07:39 pm, Martin Stone Davis wrote: > Tom Kaitchuck wrote: > > Wait. Shouldn't that be > > pSuccess*tSuccess + > > pFailure*(tFailure+tGlobalSuccess+tTimeItTakesForTheOriginalNodeToFindOut > >ItFailed). if it it's not, it should be. (note to self: need a better > > varable name) > > Since it's the original node (our node) that is measuring all of this, > tFailure includes tTimeItTakesForTheOriginalNodeToFindOutItFailed. How do the intermediate nodes route then? ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] polling & sharing estimator data
This is an extention to the idea of keeping estimator data in the .refs which aims to bring more consistent "worldview" between the nodes. It works best if there is trust implemented between nodes, but will also bring some limited results without trust. At random interval the node asks a random subset of the nodes in the rt for the estimator data they have of another node/subset of nodes (the two subsets can overlap). After that the node "merges" their results with its own estimator in a manner that would be as cancer-proof as possible in a network without trust. (that means removing outliers, averaging, etc.) Drawbacks: * increased network traffic. The timing will have to be chosen not to create a cascade effect. * powerful attacker with many nodes totally screws up our estimates, or even worse - provides such data which would make the requesting node favor his cancer nodes. Benefits: * Nodes have a more consistent picture of each other's strengths and weaknesses. * Newer nodes find their place faster * The network does not become any more static * Since topological neighbors know similar things about each other, each request from within the "neighborhood" will go directly to the best node. "Neighborhood" is not rigidly defined; if a network map is drawn, there will be no rigid boundaries between shared estimator information; rather it will flow gradually, providing optimal routing paths. (ok this sounds psycho but its much easier to visualise than explain - will try and get a pic/graph soon) pgp0.pgp Description: PGP signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Trying to eliminate a theory - request for stats
Please could I have some more stats: outputBytesAttemptedTrailing versus outputBytes Hour averages are preferred. Most nodes QueryReject a LOT, and most of those seem to be because of bandwidth usage. If as is suspected, outputBytesAttemptedTrailing is only a small part of outputBytes, the solution is to implement a binary presentation for messages to reduce the bandwidth used by e.g. QueryRejected messages. If however outputBytesAttemptedTrailing accounts for most of the total outputBytes, we have more subtle problems. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Some good news for a change - announcement HTLs
My node has had 3 announcements at HTL 2... On Thu, Nov 06, 2003 at 01:59:16AM +, Toad wrote: > hobbs on IRC's node announced successfully at HTL 6. Mine announced at > HTL 3, but only because it was set to only try HTL 3 - I have wiped the > RT and node file and left it to try again at HTL 15. The significance of > this is that since late last year at least, an announcement at HTL 3 has > been rare, HTL 2 being more common, and HTL of as high as 6 completely > unheard of. Please go through your logs - did you see a message like > this?: > > Nov 5, 2003 8:22:39 PM (freenet.node.states.announcing.Announcing, > YThread-49, NORMAL): Announced node successfully to d20a 39dc 71be 5c7d > e332 dff4 08ab 20a0 45e4 431c at depth 6. > > Successful announcements are noted at NORMAL priority, so they should be > in your logs if your node has announced recently. I would like to see a > few successful announcement HTLs - this is evidence that NGRouting is > improving load balancing, or some other factor is. > -- > Matthew J Toseland - [EMAIL PROTECTED] > Freenet Project Official Codemonkey - http://freenetproject.org/ > ICTHUS - Nothing is impossible. Our Boss says so. > ___ > Devl mailing list > [EMAIL PROTECTED] > http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Path to 0.6
On Wed, Nov 05, 2003 at 08:46:20PM -0500, Juiceman wrote: > from the freenet.ini file: > > # set yes to not bandwidth throttle connections to LocalInterfaces i.e. FCP and > mainport > dontLimitClients=false > > > Set this as true by default. Why? Accesses from localhost are unlimited anyway. This setting only affects runners of public nodes. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Results of new stats in 6313+
Martin Stone Davis wrote: Here's a new thread for all interested to post results of the new stats Okay, Martin. Here is another post. You -are- cool to have started this thread ! I also have seen questionable numbers reported (NaN, mean less than min...) searchFailedCount / hour: 4:00:00 PM EST 3.1773 3.3368 2.0 10.01274 5:00:00 PM EST 2.7857 3.3019 0.0 12.0532 6:00:00 PM EST 3.5265 3.2908 0.0 13.01132 7:00:00 PM EST 4.0681 3.5423 0.0 14.0484 8:00:00 PM EST 1.4351 2.3373 NaN NaN 1156 9:00:00 PM EST 3.2159 2.9681 0.0 13.01241 diffSearchSuccessTime / hour: 5:00:00 PM EST 16455.7058 31434.6445 NaN NaN 34 6:00:00 PM EST 10255.0 26624.2763 NaN NaN 45 7:00:00 PM EST 16000.3157 36345.5399 -3035.0 81864.0 19 8:00:00 PM EST 33819.9354 62016.4550 NaN NaN 31 9:00:00 PM EST 13128.6428 37031.2233 NaN NaN 28 diffTransferRate / hour (this one looks very wiggy) : 6:00:00 PM EST -13.683512.3816 NaN NaN 36 7:00:00 PM EST -15.77597.5716 -22.625954660161508 -7.754801998616891 15 8:00:00 PM EST -22.00196.2480 NaN NaN 27 9:00:00 PM EST -5.2779 18.2735 NaN NaN 20 ken ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] routing symmetry, or "trust is a 2 way street"
On Wed, Nov 05, 2003 at 09:00:10PM -0500, Ken Corson wrote: > Currently routing in Freenet is not reflexive. Query requests > can only be sent to peers in a node's routing table (RT). If > node G is in node F's RT, node F is not necessarily in G's RT. > > What benefits, drawbacks, security risks, performance metrics, > protocol conformance metrics (if rules were imposed), etc. > might be realized if symmetrical routing were implemented ? > (connection multiplexing could have strong relations below) It can be implemented now in a rather crude manner by firewalling your node's incoming port. The "hostile environment"/trusted-links topology would almost certainly have a similar model. > > A node functions in several roles: > + insertor (and those who forward an insertion) > + requestor (and those who forward requests) > + provider (and those who forward provided data) > + consumer (one who downloads, regardless of destination) > > Symmetric routing would allow one node to better 'profile' > another node for each of the above roles. Transient nodes > are not addressed for now. This profiling would measure another > node's level of benefit to the Freenet network as a whole, and > give preferential treatment to 'model citizens.' It would be > expected that the vast majority of nodes were equally good > citizens (adjusted for bandwidth), but the idea is to identify > those who aren't. This doesn't appear to be at odds with the > goals of NGR. For a given path, a reverse path would be almost > guaranteed - of interest to those people persuing forms of > bidirectional unicast communications through Freenet. A compromise solution would be to preferentially accept requests from nodes in our routing table when under moderate load. Nodes in our routing table are trusted to some degree because if they don't perform for our requests they will eventually be dropped. I suggested this a week or two ago. > > It seems that currently, a denial of service attack against > a single node would be easy to accomplish. Just use 'big' > bandwidth (perhaps multiple nodes) to assault the target with > useless queries and inserts. And there is no penalty! One > potential penalty would be to close the connection and ignore > Mr. NastyNode for X hours/days/years, depending on the > severity of the offense. At present, there is no mechanism to > specify the desired parameters for communications with another > node; query and insertion rates (minimum and maximum) could > be specified by count or in bytes. Well, even if we could, there is no such thing as negative trust in cyberspace - he'd just make another node. For any fair play mechanism to work, you need revocable positive trust i.e. a web of trust. This is a pain in the ass, and has MAJOR introduction problems. > > Insertions between a pair of nodes could be rate-limited. The > data may be "good" (popular), or it may be flooding garbage. > Is there a difference between data inserted maliciously and > data that simply isn't popular ? So we can't reward a node > solely for publishing... and we can only measure a publisher's > popularity to a small degree, but there is -some- information > available. We are supposed to favor the most popular content- > so why not favor the most popular publishers (as defined > above)? If everything is working as designed, all nodes would > be expected to publish at the same popularity level. Maybe the > "price" of using Freenet to publish could be equated to the > time spent inserting. Should lower bandwidth nodes "ask for" > and prefer small inserts ? Small requests ? I don't know. I think requests in general would be regarded as load (negative), and answering requests would be regarded as success (positive)... > > Requests between a pair of nodes could be rate-limited. A node > only has so much outbound bandwidth. Queries, whether > forwarded or original, should not be allowed to occupy more > than a certain amount of it. At least, not in a healthy network > that is effective at delivering content. Perhaps. I don't think any of this is urgent, it is worth discussing. > > Nodes should be able to negotiate an agreed upon rate (or pair > of rates) for sending stuff between them. A node can earn > the resources of the other node, and vice versa. And there > are many ways to profile a node - what is the average size of > inserts from node X ? And average throughput rate ? How about > requests - at what rate are they received ? What rate of > success do requests from a particular node experience ? > > I confess this is all sort of hand-wavy and vague stuff on > my part. But does it click with anyone ? Could there be a > net gain for Freenet via symmetric routing ? Did I trigger > any similar ideas ? > > Thanks for taking the time to read this. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signatu
[freenet-dev] routing symmetry, or "trust is a 2 way street"
Currently routing in Freenet is not reflexive. Query requests can only be sent to peers in a node's routing table (RT). If node G is in node F's RT, node F is not necessarily in G's RT. What benefits, drawbacks, security risks, performance metrics, protocol conformance metrics (if rules were imposed), etc. might be realized if symmetrical routing were implemented ? (connection multiplexing could have strong relations below) A node functions in several roles: + insertor (and those who forward an insertion) + requestor (and those who forward requests) + provider (and those who forward provided data) + consumer (one who downloads, regardless of destination) Symmetric routing would allow one node to better 'profile' another node for each of the above roles. Transient nodes are not addressed for now. This profiling would measure another node's level of benefit to the Freenet network as a whole, and give preferential treatment to 'model citizens.' It would be expected that the vast majority of nodes were equally good citizens (adjusted for bandwidth), but the idea is to identify those who aren't. This doesn't appear to be at odds with the goals of NGR. For a given path, a reverse path would be almost guaranteed - of interest to those people persuing forms of bidirectional unicast communications through Freenet. It seems that currently, a denial of service attack against a single node would be easy to accomplish. Just use 'big' bandwidth (perhaps multiple nodes) to assault the target with useless queries and inserts. And there is no penalty! One potential penalty would be to close the connection and ignore Mr. NastyNode for X hours/days/years, depending on the severity of the offense. At present, there is no mechanism to specify the desired parameters for communications with another node; query and insertion rates (minimum and maximum) could be specified by count or in bytes. Insertions between a pair of nodes could be rate-limited. The data may be "good" (popular), or it may be flooding garbage. Is there a difference between data inserted maliciously and data that simply isn't popular ? So we can't reward a node solely for publishing... and we can only measure a publisher's popularity to a small degree, but there is -some- information available. We are supposed to favor the most popular content- so why not favor the most popular publishers (as defined above)? If everything is working as designed, all nodes would be expected to publish at the same popularity level. Maybe the "price" of using Freenet to publish could be equated to the time spent inserting. Should lower bandwidth nodes "ask for" and prefer small inserts ? Small requests ? Requests between a pair of nodes could be rate-limited. A node only has so much outbound bandwidth. Queries, whether forwarded or original, should not be allowed to occupy more than a certain amount of it. At least, not in a healthy network that is effective at delivering content. Nodes should be able to negotiate an agreed upon rate (or pair of rates) for sending stuff between them. A node can earn the resources of the other node, and vice versa. And there are many ways to profile a node - what is the average size of inserts from node X ? And average throughput rate ? How about requests - at what rate are they received ? What rate of success do requests from a particular node experience ? I confess this is all sort of hand-wavy and vague stuff on my part. But does it click with anyone ? Could there be a net gain for Freenet via symmetric routing ? Did I trigger any similar ideas ? Thanks for taking the time to read this. ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Some good news for a change - announcement HTLs
hobbs on IRC's node announced successfully at HTL 6. Mine announced at HTL 3, but only because it was set to only try HTL 3 - I have wiped the RT and node file and left it to try again at HTL 15. The significance of this is that since late last year at least, an announcement at HTL 3 has been rare, HTL 2 being more common, and HTL of as high as 6 completely unheard of. Please go through your logs - did you see a message like this?: Nov 5, 2003 8:22:39 PM (freenet.node.states.announcing.Announcing, YThread-49, NORMAL): Announced node successfully to d20a 39dc 71be 5c7d e332 dff4 08ab 20a0 45e4 431c at depth 6. Successful announcements are noted at NORMAL priority, so they should be in your logs if your node has announced recently. I would like to see a few successful announcement HTLs - this is evidence that NGRouting is improving load balancing, or some other factor is. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Unstable build 6316 and stats
Unstable build 6316 main change is to not accept transfer rates unless the file is multi-segment i.e. over 16kB (below this the transfer time is probably meaningless), and the time taken to transfer is more than 10 milliseconds. The reason for this: I looked into the ludicrous transfer rate differences seen recently, apparently there were unrealistic transfer rates being estimated - megabytes per second on an ordinary cable modem here - for nodes that hadn't had many transfers. Several nodes that had had zero transfers showed: tTransferRate Minimum: 4,700,361 bytes/second Maximum: 4,700,361 bytes/second On their stats pages. This could be caused by the general breakage in routing tables around 6302 (which has been fixed for a while now), or it could have been caused by occasionally a small file being transferred in 1ms because of it being in the same packet as the header. It seems to have been caused mainly by the former though, as when I wiped my RT, it went back to the default initial transfer rate estimate, 4 times the bandwidth limit, 80kB/sec (new nodes get a strong bias). -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Path to 0.6
from the freenet.ini file: # set yes to not bandwidth throttle connections to LocalInterfaces i.e. FCP and mainportdontLimitClients=false Set this as true by default. ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
On Wednesday 05 November 2003 01:06 pm, Martin Stone Davis wrote: > Altruism: > - > I think the problem with any altruistic strategy is that it then lets > non-altruistic nodes get the better of us, and hordes of people would > choose a non-altruistic hack of freenet. Also, if we take your > altruistic strategy, couldn't it be argued that we would be > unnecessarily slowing down the tortoises even more? NGR takes care of > load balancing naturally. Altruism, non-altruism, it's all the same. If Fred does not treat the packets any differently, then all routing methods are altruistic.(It is advantageous to your own anonymity not to route others packets differently.) If you want to be as selfish as possible, then route all your packets in the way that will get you the data fastest. Just doing the same to all the packets is the optimal solution. I suppose someone could mod freenet to drop everyone else's packets to save bandwidth for their own. But then again they could much more easily just be transient. ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-CVS] freenet/src/freenet/node/rt NGRoutingTable.java, 1.29, 1.30 StandardNodeEstimator.java, 1.25, 1.26 StandardNodeEstimatorFactory.java, 1.3, 1.4
Update of /cvsroot/freenet/freenet/src/freenet/node/rt
In directory sc8-pr-cvs1:/tmp/cvs-serv25695/src/freenet/node/rt
Modified Files:
NGRoutingTable.java StandardNodeEstimator.java
StandardNodeEstimatorFactory.java
Log Message:
6316:
Don't accept a transfer rate unless the file is multi-segment and the time is more
than 10ms.
Logging.
Index: NGRoutingTable.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/rt/NGRoutingTable.java,v
retrieving revision 1.29
retrieving revision 1.30
diff -u -w -r1.29 -r1.30
--- NGRoutingTable.java 5 Nov 2003 22:36:40 - 1.29
+++ NGRoutingTable.java 6 Nov 2003 01:41:16 - 1.30
@@ -62,14 +62,26 @@
}
public void reportRate(double rate) {
- if(rate > fastestTransferSeen) fastestTransferSeen = rate;
+ if(rate > fastestTransferSeen) {
+ fastestTransferSeen = rate;
+ if(Core.logger.shouldLog(Logger.DEBUG, this))
+ Core.logger.log(this,"New fastest rate seen: "+rate,
+ Logger.NORMAL);
+ }
}
/** Return the fastest transfer rate seen for files over 16k */
public double getFastestRateSeen() {
+ boolean logDEBUG = Core.logger.shouldLog(Logger.DEBUG,this);
if(fastestTransferSeen == 0.0) {
+ if(logDEBUG)
+ Core.logger.log(this, "Returning default rate
"+defaultFastestTransferSeen,
+ Logger.DEBUG);
return defaultFastestTransferSeen;
}
+ if(logDEBUG)
+ Core.logger.log(this, "Returning fastest rate seen:
"+fastestTransferSeen,
+ Logger.DEBUG);
return fastestTransferSeen;
}
Index: StandardNodeEstimator.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/rt/StandardNodeEstimator.java,v
retrieving revision 1.25
retrieving revision 1.26
diff -u -w -r1.25 -r1.26
--- StandardNodeEstimator.java 5 Nov 2003 00:06:51 - 1.25
+++ StandardNodeEstimator.java 6 Nov 2003 01:41:16 - 1.26
@@ -286,6 +286,9 @@
public void transferSucceeded(Key key, long searchTime, int htl,
long size, long
transferTime) {
boolean logDEBUG = Core.logger.shouldLog(Logger.DEBUG,this);
+ if(logDEBUG)
+ Core.logger.log(this, "transferSucceeded("+key+","+searchTime+
+ ","+htl+","+size+","+transferTime,
Logger.DEBUG);
synchronized(this) {
successes++;
}
@@ -296,9 +299,12 @@
etSuccessSearch.reportTime(key, searchTime);
rpSearchFailed.report(0); // the search succeeded
rpTransferFailed.report(0); // the transfer succeeded
- if(size > 0 && transferTime > 0) {
- etTransferRate.reportTransferRate(key, ((double)size) /
-
(double)transferTime);
+ if(size > 16384 /* must be multi-segment */
+ && transferTime > 10 /* sanity check */) {
+ double rate = ((double)size) / (double)transferTime;
+ if(logDEBUG) Core.logger.log(this, "Logging transfer rate of
"+rate+
+ " bytes per millisecond", rate > 100 ?
Logger.NORMAL : Logger.DEBUG);
+ etTransferRate.reportTransferRate(key, rate);
} else {
Core.logger.log(this, "Not logging transfer rate because
size="+
size+",
transferTime="+transferTime,
Index: StandardNodeEstimatorFactory.java
===
RCS file:
/cvsroot/freenet/freenet/src/freenet/node/rt/StandardNodeEstimatorFactory.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -w -r1.3 -r1.4
--- StandardNodeEstimatorFactory.java 30 Oct 2003 01:34:06 - 1.3
+++ StandardNodeEstimatorFactory.java 6 Nov 2003 01:41:16 - 1.4
@@ -3,9 +3,11 @@
import java.io.IOException;
+import freenet.Core;
import freenet.Key;
import freenet.node.NodeReference;
import freenet.support.DataObjectPending;
+import freenet.support.Logger;
/**
* Factory interface for StandardNodeEstimators
@@ -40,8 +42,11 @@
NodeReference ref,
double rate,
[freenet-CVS] freenet/src/freenet Version.java,1.512,1.513
Update of /cvsroot/freenet/freenet/src/freenet In directory sc8-pr-cvs1:/tmp/cvs-serv25695/src/freenet Modified Files: Version.java Log Message: 6316: Don't accept a transfer rate unless the file is multi-segment and the time is more than 10ms. Logging. Index: Version.java === RCS file: /cvsroot/freenet/freenet/src/freenet/Version.java,v retrieving revision 1.512 retrieving revision 1.513 diff -u -w -r1.512 -r1.513 --- Version.java5 Nov 2003 23:00:56 - 1.512 +++ Version.java6 Nov 2003 01:41:16 - 1.513 @@ -20,7 +20,7 @@ public static String protocolVersion = "1.47"; /** The build number of the current revision */ -public static final int buildNumber = 6315; +public static final int buildNumber = 6316; public static final int ignoreBuildsAfter = 6500; ___ cvs mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs
Re: [freenet-dev] Path to 0.6
Make Y-threads the default. ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
Tom Kaitchuck wrote: On Wednesday 05 November 2003 11:57 am, Martin Stone Davis wrote: Roger Hayter wrote: As I understand it, the present routing method tries to predict the node to contact which will give it the quickest response overall to its request. Almost. It chooses the node for which estimate() is the least. As I understand it (please someone correct me if I'm wrong) estimate() is *roughly* defined as: pSuccess*tSuccess + pFailure*(tFailure+tGlobalSuccess) where all of the below are a function of location in the keyspace: pSuccess=chance of the node returning data tSuccess=time it will take the node to return data if successful pFailure=1-pSuccess tFailure=time it will take the node to fail tGlodalSuccess=time it will take to retrieve data successfully if we went to another node. I believe this is estimated by averaging all the past tSuccess:es *starting from our node*. So if, in the past, we had to try 10 different nodes to get a success for a key in this area, all the time we spent trying the 10 nodes is added-in to tGlobalSuccess. Wait. Shouldn't that be pSuccess*tSuccess + pFailure*(tFailure+tGlobalSuccess+tTimeItTakesForTheOriginalNodeToFindOutItFailed). if it it's not, it should be. (note to self: need a better varable name) Since it's the original node (our node) that is measuring all of this, tFailure includes tTimeItTakesForTheOriginalNodeToFindOutItFailed. -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
On Wednesday 05 November 2003 11:57 am, Martin Stone Davis wrote: > Roger Hayter wrote: > > As I understand it, the present routing method tries to predict the node > > to contact which will give it the quickest response overall to its > > request. > > Almost. It chooses the node for which estimate() is the least. As I > understand it (please someone correct me if I'm wrong) estimate() is > *roughly* defined as: > > pSuccess*tSuccess + pFailure*(tFailure+tGlobalSuccess) > > where all of the below are a function of location in the keyspace: > > pSuccess=chance of the node returning data > tSuccess=time it will take the node to return data if successful > pFailure=1-pSuccess > tFailure=time it will take the node to fail > tGlodalSuccess=time it will take to retrieve data successfully if we > went to another node. I believe this is estimated by averaging all the > past tSuccess:es *starting from our node*. So if, in the past, we had > to try 10 different nodes to get a success for a key in this area, all > the time we spent trying the 10 nodes is added-in to tGlobalSuccess. Wait. Shouldn't that be pSuccess*tSuccess + pFailure*(tFailure+tGlobalSuccess+tTimeItTakesForTheOriginalNodeToFindOutItFailed). if it it's not, it should be. (note to self: need a better varable name) ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: I/O error replying
On Tuesday 04 November 2003 01:04 pm, Tom Kaitchuck wrote: > On Tuesday 04 November 2003 11:55 am, Toad wrote: > > On Tue, Nov 04, 2003 at 10:43:36AM -0600, Tom Kaitchuck wrote: > > > On Monday 03 November 2003 11:45 pm, Martin Stone Davis wrote: > > > > I have no idea if is will actually help, but: try threadFactory=Y > > > > instead of Q; and also try increasing maximumThreads to at least 300. > > > > Make sure to remove the % (comment), and restart freenet. Let us > > > > know what happens either way. > > > > > > > > -Martin > > > > > > WOW! I just tried this on 5031, and what a deferance. Before I was > > > getting 4+QPH and accepting 20% of those. Now I'm getting close to > > > 45000QPH and it is still accpting 100%. I am now pumping data out more > > > than twice as fast as I was before. The weird thing is my CPU useage is > > > not much higher. Maybe we should up the default thread limit. > > > > When we last tried it we got a lot of OOMs. It's also very user hostile. > > If the main reason for rejecting is threads, we can do more NIO. But for > > a lot of users it's bandwidth. > > Well bare in mind that those results may not be typical. This is comming > from my firewalled node. Now that it has been running for several hours my > connections are up, and I am doing some QRing. But still much better than > it was, although the routing time is longer. My memory useage is around 96 > megs and my CPU load is now higher than it was. > > In any event I would advise others to play around with settings and report > what works and what doesn't. > > Also would it be possible to somehow test my suggestion of making a QR the > default, just by letting the connection time out and not sending an accept, > so that no bandwidth is wasted on them, without subjecting the whole > network to it? Doug Bostrom writes: FYI I reset the node here to limit at 256 connections and 300 threads. That improved the performance of the node, a lot, though CPU load went way up. I then followed the advice of somebody else and changed threadFactory=Y. This resulted in a further improvement. The node is now consistently running at maximum bandwidth, so is QRing but not for any reason other than it's moving data as fast as possible over the link. This is new behavior. ... and ... Some hours later, I see what appears to my untrained eye as a fairly encouraging 0.44 legitimate DNF and even more encouraging 0.70 current proportion of requests being accepted. 340MB of data have been transferred during the past 4 hours. PS. Sorry for the late responce Doug. ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Memory induced CPU overload.
On Wednesday 05 November 2003 03:00 pm, Tom Kaitchuck wrote: > I'm seeing a not enough Memory induced CPU overload on build 5031. It is > realy slowing down my node. This has been going on for several hours: Humm, this doesn't seem to be a memory problem after all. It seems most likely caused by this: " read interface thread" daemon prio=5 tid=0x825a168 nid=9777 runnable [0xbe1ff000..0xbe1ff83c] at sun.nio.ch.PollArrayWrapper.poll0(Native Method) at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100) at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:64) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:62) - locked <0x44e88300> (a sun.nio.ch.SelectorImpl$Lock) at freenet.transport.AbstractSelectorLoop.mySelect(AbstractSelectorLoop.java:527) at freenet.transport.AbstractSelectorLoop.loop(AbstractSelectorLoop.java:675) at freenet.transport.ReadSelectorLoop.run(ReadSelectorLoop.java:506) at java.lang.Thread.run(Thread.java:536) This was the thread that was causing the problem, and this exact printout recered many times when SIGQUITing it. So can someone look into what's going on AbstractSelectorLoop.java:527. BTW why is NIO locked? ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] [mailman-bounces@freenetproject.org: Uncaught bounce notification]
- Forwarded message from [EMAIL PROTECTED] - Subject: Uncaught bounce notification From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] X-BeenThere: [EMAIL PROTECTED] X-Mailman-Version: 2.1.2 List-Id: Discussion of development issues X-List-Administrivia: yes The attached message was received as a bounce, but either the bounce format was not recognized, or no member addresses could be extracted from it. This mailing list has been configured to send all unrecognized bounce messages to the list administrator(s). For more information see: http://dodo.freenetproject.org/cgi-bin/mailman/admin/devl/bounce From: Stephen Tidey <[EMAIL PROTECTED]> To: Devl-Bounces <[EMAIL PROTECTED]> Subject: Some fun I'm having with 5031 Importance: Normal Thought you might like to know that for an experiment I took off bandwidth limiting altogether last night and left my node running, looked at the screen this morning to find a Blue Screen of Death (running on Win 2000). Restarted, did a consistency check on the drives and restarted node earlier this evening, already it has blown up with another BSOD... At the moment I can't even run chkdsk in safe mode... Looks like a re-install might be on the cards :-( Up to the point where I removed limiting I was having the best throughput ever! Still not conversant with all the stats/logging data you have got in freenet now, so rather than paste stuff in this msg, thought you could tell me what would be best info to give. --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.532 / Virus Database: 326 - Release Date: 27/10/2003 - End forwarded message - -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-CVS] freenet/src/freenet Version.java, 1.511, 1.512 ConnectionHandler.java, 1.205, 1.206
Update of /cvsroot/freenet/freenet/src/freenet
In directory sc8-pr-cvs1:/tmp/cvs-serv27087/src/freenet
Modified Files:
Version.java ConnectionHandler.java
Log Message:
6315: New diag var outputBytesTrailingAttempted. (outputBytesLow included some
messages).
Index: Version.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/Version.java,v
retrieving revision 1.511
retrieving revision 1.512
diff -u -w -r1.511 -r1.512
--- Version.java5 Nov 2003 22:36:40 - 1.511
+++ Version.java5 Nov 2003 23:00:56 - 1.512
@@ -20,7 +20,7 @@
public static String protocolVersion = "1.47";
/** The build number of the current revision */
-public static final int buildNumber = 6314;
+public static final int buildNumber = 6315;
public static final int ignoreBuildsAfter = 6500;
Index: ConnectionHandler.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/ConnectionHandler.java,v
retrieving revision 1.205
retrieving revision 1.206
diff -u -w -r1.205 -r1.206
--- ConnectionHandler.java 4 Nov 2003 16:53:23 - 1.205
+++ ConnectionHandler.java 5 Nov 2003 23:01:00 - 1.206
@@ -1340,6 +1340,7 @@
}
twcb = cb;
// Send the data
+
Core.diagnostics.occurrenceCounting("outputBytesTrailingAttempted", length);
sendBytes(block, offset, length, WriteSelectorLoop.TRAILER);
}
}
___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs
[freenet-CVS] freenet/src/freenet/node Main.java,1.297,1.298
Update of /cvsroot/freenet/freenet/src/freenet/node
In directory sc8-pr-cvs1:/tmp/cvs-serv27087/src/freenet/node
Modified Files:
Main.java
Log Message:
6315: New diag var outputBytesTrailingAttempted. (outputBytesLow included some
messages).
Index: Main.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/Main.java,v
retrieving revision 1.297
retrieving revision 1.298
diff -u -w -r1.297 -r1.298
--- Main.java 5 Nov 2003 22:36:40 - 1.297
+++ Main.java 5 Nov 2003 23:01:04 - 1.298
@@ -1693,7 +1693,9 @@
d.registerCounting("outputBytesLow", Diagnostics.MINUTE,
"Low priority external bytes written via TCP.",
transport);
-
+d.registerCounting("outputBytesTrailingAttempted", Diagnostics.MINUTE,
+ "Bytes attempted to write on trailing field sends over the
network.",
+ transport);
tcpConnection.logBytes = true;
}
___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs
Re: [freenet-dev] Sun Fucks Up Again
On November 05, 2003 03:05 pm, Toad wrote: > Actually it may not be using green threads, it may be using NPTL. But > the point remains, two separate stack dumps showed it hung waiting for a > lock on a var that wasn't even locked. I lost the logs, sorry. What NPTL verson? getconf GNU_LIBPTHREAD_VERSION Its upto atleast version 0.57 - versions in the 0.3x are apt to be quite buggy according to a thread on LKML. If its old try updating libc6 to the latest for your dist. Ed > On Wed, Nov 05, 2003 at 08:03:36PM +, Toad wrote: > > It appears that the Sun JDK really sucks in debug mode. I was running it > > with -Xdebug -Xnoagent -Xrunjdwp transport=dt_socket,server=y,address=7 > > 777,suspend=n -mx400m freenet.node.Main , so that I could connect to it > > with eclipse. It is only using one process, so presumably it is using > > green threads in order to make pausing the whole JVM possible. However > > every time I try to go to any freesite, it eventually returns 0 data, > > the logfile for the last hour (I haven't bothered to unzip previous > > hours) contains only about 5 different messages, repeated over an over > > (attached, gzipped), and when I get a stack dump, twice, I find many > > threads waiting for the Ticker and the Ticker waiting for a lock on > > "<0x460d05e8> (a freenet.support.BufferLoggerHook)", which is apparently > > not locked by ANY thread. I would have just written this down to stack > > dumps not being atomic except a) it's running in debug mode, and using > > green threads, apparently, so they probably are atomic, and b) I got two > > separate stack traces several seconds apart that said exactly the same > > thing. > > > > Debugging Fred in Eclipse is therefore much harder than originally > > thought. I will revert to my usual shotgun debugging approach. > > -- > > Matthew J Toseland - [EMAIL PROTECTED] > > Freenet Project Official Codemonkey - http://freenetproject.org/ > > ICTHUS - Nothing is impossible. Our Boss says so. > > > > > > > > ___ > > Devl mailing list > > [EMAIL PROTECTED] > > http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-CVS] freenet/src/freenet/node/states/announcing Announcing.java, 1.38, 1.39 SendAnnouncement.java, 1.19, 1.20 ExecuteAnnouncement.java, 1.11, 1.12 CompleteAnnouncement.java, 1.12, 1.13 AnnouncingState.java, 1.4, 1.5
Update of /cvsroot/freenet/freenet/src/freenet/node/states/announcing
In directory sc8-pr-cvs1:/tmp/cvs-serv21851/src/freenet/node/states/announcing
Modified Files:
Announcing.java SendAnnouncement.java ExecuteAnnouncement.java
CompleteAnnouncement.java AnnouncingState.java
Log Message:
6314:
Fix announcements to nodes not already known.
Fix descriptions for diff diagnostics vars.
Report diffTransferRate etc in bytes per second.
Logging
Index: Announcing.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/announcing/Announcing.java,v
retrieving revision 1.38
retrieving revision 1.39
diff -u -w -r1.38 -r1.39
--- Announcing.java 31 Oct 2003 19:36:15 - 1.38
+++ Announcing.java 5 Nov 2003 22:36:39 - 1.39
@@ -265,7 +265,7 @@
" announcement targets for this node.",
Logger.NORMAL);
for (int i = 0 ; i < m ; i++) {
-SendAnnouncement.makeTry(n, id, n.getPeer(now[i]),
+SendAnnouncement.makeTry(n, id, now[i],
hopsToLive);
}
} else {
Index: SendAnnouncement.java
===
RCS file:
/cvsroot/freenet/freenet/src/freenet/node/states/announcing/SendAnnouncement.java,v
retrieving revision 1.19
retrieving revision 1.20
diff -u -w -r1.19 -r1.20
--- SendAnnouncement.java 31 Oct 2003 19:21:21 - 1.19
+++ SendAnnouncement.java 5 Nov 2003 22:36:39 - 1.20
@@ -8,6 +8,7 @@
import freenet.node.BadStateException;
import freenet.node.Node;
import freenet.node.NodeMessageObject;
+import freenet.node.NodeReference;
import freenet.node.State;
import freenet.node.states.announcement.NoReply;
import freenet.support.Logger;
@@ -28,7 +29,7 @@
*make sure it is not more than what is tolerated by the
*network.
*/
-static State makeTry(Node n, long id, Peer target, int htl) {
+static State makeTry(Node n, long id, NodeReference target, int htl) {
if (htl > Node.maxHopsToLive) {
Core.logger.log(SendAnnouncement.class,
@@ -66,7 +67,7 @@
-private SendAnnouncement(long id, int htl, Peer target) {
+private SendAnnouncement(long id, int htl, NodeReference target) {
super(id, htl, target);
}
@@ -109,7 +110,7 @@
// Count outbound requests.
Core.diagnostics.occurrenceBinomial("outboundAggregateRequests", 1, 1);
if (Core.outboundRequests != null) {
-Core.outboundRequests.incTotal(target.getAddress().toString());
+Core.outboundRequests.incTotal(target.firstPhysicalToString());
}
// Keep track of outbound requests for rate limiting.
Index: ExecuteAnnouncement.java
===
RCS file:
/cvsroot/freenet/freenet/src/freenet/node/states/announcing/ExecuteAnnouncement.java,v
retrieving revision 1.11
retrieving revision 1.12
diff -u -w -r1.11 -r1.12
--- ExecuteAnnouncement.java31 Oct 2003 18:25:12 - 1.11
+++ ExecuteAnnouncement.java5 Nov 2003 22:36:39 - 1.12
@@ -48,7 +48,7 @@
if (mo instanceof NoReply)
return nr == mo;
else if (mo instanceof Message)
-return target.equalsIdent(((Message) mo).peerIdentity());
+return target.getIdentity().equals(((Message) mo).peerIdentity());
else
return false;
}
Index: CompleteAnnouncement.java
===
RCS file:
/cvsroot/freenet/freenet/src/freenet/node/states/announcing/CompleteAnnouncement.java,v
retrieving revision 1.12
retrieving revision 1.13
diff -u -w -r1.12 -r1.13
--- CompleteAnnouncement.java 31 Oct 2003 19:36:14 - 1.12
+++ CompleteAnnouncement.java 5 Nov 2003 22:36:39 - 1.13
@@ -40,7 +40,7 @@
if (mo instanceof NoComplete)
return nc == mo;
else if (mo instanceof Message)
-return target.equalsIdent(((Message) mo).peerIdentity());
+return target.getIdentity().equals(((Message) mo).peerIdentity());
else
return false;
}
Index: AnnouncingState.java
===
RCS file:
/cvsroot/freenet/freenet/src/freenet/node/states/announcing/AnnouncingState.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -u -w -r1.4 -r1.5
--- AnnouncingState.java30 Jun 2003 21:00:55 - 1.4
+++ AnnouncingState.java5 Nov 2003 22:36:39 - 1.5
@@ -10,12 +10,12 @@
static final Digest ctx = SHA1.getInstance();
final int hopsToLive;
-final Pe
[freenet-CVS] freenet/src/freenet/node Main.java,1.296,1.297
Update of /cvsroot/freenet/freenet/src/freenet/node
In directory sc8-pr-cvs1:/tmp/cvs-serv21851/src/freenet/node
Modified Files:
Main.java
Log Message:
6314:
Fix announcements to nodes not already known.
Fix descriptions for diff diagnostics vars.
Report diffTransferRate etc in bytes per second.
Logging
Index: Main.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/Main.java,v
retrieving revision 1.296
retrieving revision 1.297
diff -u -w -r1.296 -r1.297
--- Main.java 5 Nov 2003 00:21:41 - 1.296
+++ Main.java 5 Nov 2003 22:36:40 - 1.297
@@ -2021,20 +2021,22 @@
"Search time for successful requests",
messages);
d.registerContinuous("diffSearchSuccessTime", Diagnostics.MINUTE,
- "Predicted search time for a successful request minus "+
- "the actual search time. Should average at 0.", messages);
+ "Search time for a successful request minus "+
+ "the predicted search time. Should average at 0.", messages);
d.registerContinuous("absDiffSearchSuccessTime", Diagnostics.MINUTE,
"Absolute difference between predicted search time for a "+
"successful request and the actual search time.", messages);
d.registerContinuous("successTransferRate", Diagnostics.MINUTE,
- "Transfer rate for successful requests",
+ "Transfer rate for successful requests in bytes per
second",
messages);
d.registerContinuous("diffTransferRate", Diagnostics.MINUTE,
- "Predicted transfer rate minus the actual transfer rate "+
- "for a successful request. Should average to 0.", messages);
+ "Actual transfer rate minus the predicted transfer rate "+
+ "for a successful request. Should average to 0. In bytes per
second.",
+ messages);
d.registerContinuous("absDiffTransferRate", Diagnostics.MINUTE,
"Absolute difference between predicted transfer rate and "+
- "actual transfer rate for a successful request.", messages);
+ "actual transfer rate for a successful request, in bytes per
second.",
+ messages);
d.registerBinomial("requestSuccessRatio", Diagnostics.MINUTE,
"The number of requests started, and the "+
"number that successfully returned data.",
___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs
[freenet-CVS] freenet/src/freenet Version.java,1.510,1.511
Update of /cvsroot/freenet/freenet/src/freenet In directory sc8-pr-cvs1:/tmp/cvs-serv21851/src/freenet Modified Files: Version.java Log Message: 6314: Fix announcements to nodes not already known. Fix descriptions for diff diagnostics vars. Report diffTransferRate etc in bytes per second. Logging Index: Version.java === RCS file: /cvsroot/freenet/freenet/src/freenet/Version.java,v retrieving revision 1.510 retrieving revision 1.511 diff -u -w -r1.510 -r1.511 --- Version.java5 Nov 2003 00:06:51 - 1.510 +++ Version.java5 Nov 2003 22:36:40 - 1.511 @@ -20,7 +20,7 @@ public static String protocolVersion = "1.47"; /** The build number of the current revision */ -public static final int buildNumber = 6313; +public static final int buildNumber = 6314; public static final int ignoreBuildsAfter = 6500; ___ cvs mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs
[freenet-CVS] freenet/src/freenet/node/rt NGRouting.java, 1.21, 1.22 NGRoutingTable.java, 1.28, 1.29
Update of /cvsroot/freenet/freenet/src/freenet/node/rt
In directory sc8-pr-cvs1:/tmp/cvs-serv21851/src/freenet/node/rt
Modified Files:
NGRouting.java NGRoutingTable.java
Log Message:
6314:
Fix announcements to nodes not already known.
Fix descriptions for diff diagnostics vars.
Report diffTransferRate etc in bytes per second.
Logging
Index: NGRouting.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/rt/NGRouting.java,v
retrieving revision 1.21
retrieving revision 1.22
diff -u -w -r1.21 -r1.22
--- NGRouting.java 4 Nov 2003 20:29:18 - 1.21
+++ NGRouting.java 5 Nov 2003 22:36:39 - 1.22
@@ -202,15 +202,13 @@
Core.diagnostics.occurrenceContinuous("absDiffSearchSuccessTime",
Math.abs(diffSearchTime));
if(size > 16384) { // more than one segment
- double rate = ((double)size) / ((double)transferTime);
- double diffRate = rate - lastEstimate.transferRate;
+ double rate = ((double)size * 1000) / ((double)transferTime);
+ double diffRate = rate - (lastEstimate.transferRate*1000);
Core.diagnostics.occurrenceContinuous("diffTransferRate", diffRate);
Core.diagnostics.occurrenceContinuous("absDiffTransferRate",
Math.abs(diffRate));
ngrt.reportRate(rate);
- Core.diagnostics.occurrenceContinuous("successTransferRate",
- ((double)size*1000)/
- ((double)transferTime));
+ Core.diagnostics.occurrenceContinuous("successTransferRate", rate);
}
terminate(true, true);
}
Index: NGRoutingTable.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/rt/NGRoutingTable.java,v
retrieving revision 1.28
retrieving revision 1.29
diff -u -w -r1.28 -r1.29
--- NGRoutingTable.java 4 Nov 2003 23:00:43 - 1.28
+++ NGRoutingTable.java 5 Nov 2003 22:36:40 - 1.29
@@ -116,7 +116,7 @@
}
ne = factory.create(mem, ref, e, false);
} catch (IOException ex) {
- Core.logger.log(this, "Caught "+ex+"
deserializing a NodeEstimator for "+mem, Logger.ERROR);
+ Core.logger.log(this, "Caught "+ex+"
deserializing a NodeEstimator for "+mem, ex, Logger.ERROR);
ne = null;
}
}
___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs
[freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
Roger Hayter wrote: In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes snip Fine, but whenever you altruistically send a request to a node (A) where another node (B) would have given you a faster success, you also prevent another node (C) from sending a request to its best candidate, A. So, wouldn't your so-called altruism would actually hurt the network? For shame! -Martin I freely admit you are more likely to be right than me! However, implicit in my proposal is that everyone would be sending a *lot* fewer requests, because they would only be sending them to nodes empirically unlikely to QR (for certain, not yet defined, values of unlikely). Ah, but sending requests is not, *in and of itself*, such a problem, as QR:s apparently (this has not been 100% verified) take up a pretty small portion of bw. It can probably be made even smaller with compression et al. (As far as too many QR:s degrading routing, that's what the rest of the discussion was about.) -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes snip Fine, but whenever you altruistically send a request to a node (A) where another node (B) would have given you a faster success, you also prevent another node (C) from sending a request to its best candidate, A. So, wouldn't your so-called altruism would actually hurt the network? For shame! -Martin I freely admit you are more likely to be right than me! However, implicit in my proposal is that everyone would be sending a *lot* fewer requests, because they would only be sending them to nodes empirically unlikely to QR (for certain, not yet defined, values of unlikely). -- Roger Hayter ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
On Wed, Nov 05, 2003 at 08:03:43PM +, Roger Hayter wrote: > In message <[EMAIL PROTECTED]>, Martin Stone Davis > <[EMAIL PROTECTED]> writes > >Roger Hayter wrote: > > > >>In message <[EMAIL PROTECTED]>, Martin Stone Davis > >><[EMAIL PROTECTED]> writes > >> > > > >>>Roger Hayter wrote: > >>> > >>>Specialization: > >>>--- > >>>I guess the idea is that if there are 10 areas of specialization, > >>>and we know 5 that specialize in the first area, but all 5 regularly > >>>QR, then we begin diluting the specialization of the other nodes > >>>which aren't specialized in the first area. Did I state the problem > >>> > >> Not entirely, suppose we start with 10 nodes which are only 0.1% > >>specialised due to random variations in what they have cached, and are > >>never likely to get more specialised. > > > >Why aren't they likely to get more specialized? Can you draw out your > >example a bit more? > > Well, if speed of rejection favours a node many times more than what it > is good at retrieving, then there is little or no selection pressure in It doesn't. Rate of rejection perhaps. But it has been explained that the probability of a QueryRejection is multiplied by the sum of the rejection time AND the time for a successful request from another node. > favour of a given specialisation. I am suggesting there is a threshold > of selection pressure below which it has no effect, because random > events in the node have a bigger effect on specialisation than any net > specialisation of requests, over relevant time periods. Well unfortunately at the moment the probability of the node QueryRejected'ing may well be a dominating factor in NGRouting - but I don't think the solution is to radically alter NGRouting, the solution is to make the nodes QR less. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Memory induced CPU overload.
I'm seeing a not enough Memory induced CPU overload on build 5031. It is realy slowing down my node. This has been going on for several hours: Maximum memory the JVM will allocate 192 MiB Memory currently allocated by the JVM 130,112 KiB Memory in use 130,154,432 Bytes Estimated memory used by logger None Unused allocated memory 3,078,240 Bytes ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
Roger Hayter wrote: In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: Specialization: --- I guess the idea is that if there are 10 areas of specialization, and we know 5 that specialize in the first area, but all 5 regularly QR, then we begin diluting the specialization of the other nodes which aren't specialized in the first area. Did I state the problem Not entirely, suppose we start with 10 nodes which are only 0.1% specialised due to random variations in what they have cached, and are never likely to get more specialised. Why aren't they likely to get more specialized? Can you draw out your example a bit more? Well, if speed of rejection favours a node many times more than what it is good at retrieving, then there is little or no selection pressure in favour of a given specialisation. I am suggesting there is a threshold of selection pressure below which it has no effect, because random events in the node have a bigger effect on specialisation than any net specialisation of requests, over relevant time periods. I see. It would be nice to have some sort of measure of "specialization pressure", both theoretically (somehow in terms of number of nodes, number of keys being transferred, etc.) and practically (in terms of things that we could measure at our own node). That reminds me of my eSpecializationRelevance (see the thread: "Measuring node/network health"), which is supposedly a measure of how much our specialization should matter to other nodes. If so, I would say the problem here is that all 5 nodes specialized near this key are hares. If the hares become burdened with QR:s, shouldn't NGR lead to us eventually favoring a sixth (tortoise) which also becomes specialized? Yes, but the tortoise would be favoured for the whole keyspace, and not be very good at that, so soon go out of use again. In my hypothetical, the tortoise we find only needs to be good at giving a success for the keyspace near the target key. Why would that one tortoise be favored by us for the whole keyspace? If it had been previously neglected so much it could QR everything it received quicker than the (overwhelmed) "fast" nodes could at that time. This might be a bigger effect than any specialisation present. Yes, but read the rest of what I said: Also, how could any one node be *that* wonderful at returning sucesses for the entire keyspace? If it were, yes, it would eventually go out of use (due to being overloaded) for *some* of the nodes which use it, but as most nodes back-off from it, the few that continued to use it would do so for whatever keys they had found it to be best for, leading to it becoming specialized. So even though the tortoise would start out not very specialized, it might become specialized due to the few nodes that end up using it heavily. Or at least that is the hope. I think that certainly NGR will give rise to specialization in an active network *in the long run* (as time goes to infintity). The real question are: how long? and how does the amount of time depend on the activity and size of the network. Those are probably best answered by theory/simulations, which I am not equipped to do. Another key question is: how much is specialization even needed in any given network? Certainly, some networks need it more than others. My eHealth et al (again, see the thread: "Measuring node/network health") were intended to start getting answers to some of these questions. More and better measures are possible, too, I am sure. Got some? :) Altruism: - I think the problem with any altruistic strategy is that it then lets non-altruistic nodes get the better of us, and hordes of people would choose a non-altruistic hack of freenet. Also, if we take your altruistic strategy, couldn't it be argued that we would be unnecessarily slowing down the tortoises even more? NGR takes care of load balancing naturally. This, I have no answer to (but there may be one - there is some information revealed by the non-altruistic strategy). not sure sure what you mean by "some information revealed" I am assuming an altruistic node would send a set of requests (to the given node) with a more specialised bias than a non-altruistic node. Ah, I think I see. You note that a given node has a potential to become good in a certain area, so you give it even more requests in that area than you normally would, hoping that it will then *really* specialize and be extra good for you to use later in that same area, thus paying off your altruism. Fine, but whenever you altruistically send a request to a node (A) where another node (B) would have given you a faster success, you also prevent another node (C) from sending a request to its best candidate, A. So, wouldn't y
[freenet-dev] 6313 YThread null/null instead of tcp/connection
Anyone know what this means? Turning up with increasing frequency in my logs; I've * for privacy. It's the bit which says null,null, which in similar messages in my log usually reads something like tcp/connection: a.b.c.d:e>local, [EMAIL PROTECTED], [EMAIL PROTECTED] Also concerned about the sending null:-1 which I'm getting a lot of (even for messages that say tcp/connection:stuff instead of null,null) Should I just stop worrying? --- These messages all have the same DSA(***): 05-Nov-2003 13:41:16 (freenet.message.DataRequest, YThread-13965): Long messageInitialStateTime 1732 : freenet.Message: DataRequest @[EMAIL PROTECTED] for null,null, sending null:-1 for [EMAIL PROTECTED] (DSA(***),sessions=1, presentations=1, ID=DSA(***)): outbound attempts=0:0/0 @ 35d767378409f536 05-Nov-2003 13:41:16 (freenet.message.DataRequest, YThread-13963): Long messageInitialStateTime 1742 : freenet.Message: DataRequest @[EMAIL PROTECTED] for null,null, sending null:-1 for [EMAIL PROTECTED] (DSA(***),sessions=1, presentations=1, ID=DSA(***)): outbound attempts=0:0/0 @ 595c0485a8519bdd 05-Nov-2003 13:41:16 (freenet.message.DataRequest, YThread-13649): Long messageInitialStateTime 1742 : freenet.Message: DataRequest @[EMAIL PROTECTED] for null,null, sending null:-1 for [EMAIL PROTECTED] (DSA(***),sessions=1, presentations=1, ID=DSA(***)): outbound attempts=0:0/0 @ 5475ab0ffdbd859b 05-Nov-2003 13:41:16 (freenet.message.DataRequest, YThread-14097): Long messageInitialStateTime 1752 : freenet.Message: DataRequest @[EMAIL PROTECTED] for null,null, sending null:-1 for [EMAIL PROTECTED] (DSA(***),sessions=1, presentations=1, ID=DSA(***)): outbound attempts=0:0/0 @ ceefe18cae67ac95 These messages also share the same DSA(***) tho different to above, and also share the same tcp/***.*** mentioned toward the end of the message 05-Nov-2003 13:42:26 (freenet.message.DataRequest, YThread-13923): Long messageInitialStateTime 1121 : freenet.Message: DataRequest @[EMAIL PROTECTED] for null,null, sending null:-1 for [EMAIL PROTECTED] (DSA(***),tcp/***.***, sessions=1, presentations=1, ID=DSA(***)): outbound attempts=0:0/0 @ b1301936f1f76852 05-Nov-2003 13:42:27 (freenet.message.DataRequest, YThread-14102): Long messageInitialStateTime 1132 : freenet.Message: DataRequest @[EMAIL PROTECTED] for null,null, sending null:-1 for [EMAIL PROTECTED] (DSA(***),tcp/***:***, sessions=1, presentations=1, ID=DSA(***)): outbound attempts=0:0/0 @ 91964eced2d18220 ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Sun Fucks Up Again
Actually it may not be using green threads, it may be using NPTL. But the point remains, two separate stack dumps showed it hung waiting for a lock on a var that wasn't even locked. I lost the logs, sorry. On Wed, Nov 05, 2003 at 08:03:36PM +, Toad wrote: > It appears that the Sun JDK really sucks in debug mode. I was running it > with -Xdebug -Xnoagent -Xrunjdwp transport=dt_socket,server=y,address=7 > 777,suspend=n -mx400m freenet.node.Main , so that I could connect to it > with eclipse. It is only using one process, so presumably it is using > green threads in order to make pausing the whole JVM possible. However > every time I try to go to any freesite, it eventually returns 0 data, > the logfile for the last hour (I haven't bothered to unzip previous > hours) contains only about 5 different messages, repeated over an over > (attached, gzipped), and when I get a stack dump, twice, I find many > threads waiting for the Ticker and the Ticker waiting for a lock on > "<0x460d05e8> (a freenet.support.BufferLoggerHook)", which is apparently > not locked by ANY thread. I would have just written this down to stack > dumps not being atomic except a) it's running in debug mode, and using > green threads, apparently, so they probably are atomic, and b) I got two > separate stack traces several seconds apart that said exactly the same > thing. > > Debugging Fred in Eclipse is therefore much harder than originally > thought. I will revert to my usual shotgun debugging approach. > -- > Matthew J Toseland - [EMAIL PROTECTED] > Freenet Project Official Codemonkey - http://freenetproject.org/ > ICTHUS - Nothing is impossible. Our Boss says so. > ___ > Devl mailing list > [EMAIL PROTECTED] > http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: Specialization: --- I guess the idea is that if there are 10 areas of specialization, and we know 5 that specialize in the first area, but all 5 regularly QR, then we begin diluting the specialization of the other nodes which aren't specialized in the first area. Did I state the problem Not entirely, suppose we start with 10 nodes which are only 0.1% specialised due to random variations in what they have cached, and are never likely to get more specialised. Why aren't they likely to get more specialized? Can you draw out your example a bit more? Well, if speed of rejection favours a node many times more than what it is good at retrieving, then there is little or no selection pressure in favour of a given specialisation. I am suggesting there is a threshold of selection pressure below which it has no effect, because random events in the node have a bigger effect on specialisation than any net specialisation of requests, over relevant time periods. If so, I would say the problem here is that all 5 nodes specialized near this key are hares. If the hares become burdened with QR:s, shouldn't NGR lead to us eventually favoring a sixth (tortoise) which also becomes specialized? Yes, but the tortoise would be favoured for the whole keyspace, and not be very good at that, so soon go out of use again. In my hypothetical, the tortoise we find only needs to be good at giving a success for the keyspace near the target key. Why would that one tortoise be favored by us for the whole keyspace? If it had been previously neglected so much it could QR everything it received quicker than the (overwhelmed) "fast" nodes could at that time. This might be a bigger effect than any specialisation present. Also, how could any one node be *that* wonderful at returning sucesses for the entire keyspace? If it were, yes, it would eventually go out of use (due to being overloaded) for *some* of the nodes which use it, but as most nodes back-off from it, the few that continued to use it would do so for whatever keys they had found it to be best for, leading to it becoming specialized. Altruism: - I think the problem with any altruistic strategy is that it then lets non-altruistic nodes get the better of us, and hordes of people would choose a non-altruistic hack of freenet. Also, if we take your altruistic strategy, couldn't it be argued that we would be unnecessarily slowing down the tortoises even more? NGR takes care of load balancing naturally. This, I have no answer to (but there may be one - there is some information revealed by the non-altruistic strategy). not sure sure what you mean by "some information revealed" I am assuming an altruistic node would send a set of requests (to the given node) with a more specialised bias than a non-altruistic node. -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl -- Roger Hayter ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Sun Fucks Up Again
It appears that the Sun JDK really sucks in debug mode. I was running it with -Xdebug -Xnoagent -Xrunjdwp transport=dt_socket,server=y,address=7 777,suspend=n -mx400m freenet.node.Main , so that I could connect to it with eclipse. It is only using one process, so presumably it is using green threads in order to make pausing the whole JVM possible. However every time I try to go to any freesite, it eventually returns 0 data, the logfile for the last hour (I haven't bothered to unzip previous hours) contains only about 5 different messages, repeated over an over (attached, gzipped), and when I get a stack dump, twice, I find many threads waiting for the Ticker and the Ticker waiting for a lock on "<0x460d05e8> (a freenet.support.BufferLoggerHook)", which is apparently not locked by ANY thread. I would have just written this down to stack dumps not being atomic except a) it's running in debug mode, and using green threads, apparently, so they probably are atomic, and b) I got two separate stack traces several seconds apart that said exactly the same thing. Debugging Fred in Eclipse is therefore much harder than originally thought. I will revert to my usual shotgun debugging approach. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: 6281 broken
Toad wrote: On Sun, Oct 26, 2003 at 11:25:08AM -0800, Mike Stump wrote: Usenet is an interesting comparison. Usenet goes for around $1/gig, with a 18 to 180 day retention, depending upon how large the avereage post size is. If some large operator offered freenet access at $1/gig, would we have a design that was less secure? Fred would migrate connections to them by itself, and if they were required by law to log access, would freenet be any different than usenet? Whatever. I'm not going to bother even arguing with you today. What he means is: usenet ain't anonymous. It just hasn't been busted big-time *yet*. -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
Roger Hayter wrote: In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: Specialization: --- I guess the idea is that if there are 10 areas of specialization, and we know 5 that specialize in the first area, but all 5 regularly QR, then we begin diluting the specialization of the other nodes which aren't specialized in the first area. Did I state the problem correctly? Not entirely, suppose we start with 10 nodes which are only 0.1% specialised due to random variations in what they have cached, and are never likely to get more specialised. Why aren't they likely to get more specialized? Can you draw out your example a bit more? If so, I would say the problem here is that all 5 nodes specialized near this key are hares. If the hares become burdened with QR:s, shouldn't NGR lead to us eventually favoring a sixth (tortoise) which also becomes specialized? Yes, but the tortoise would be favoured for the whole keyspace, and not be very good at that, so soon go out of use again. In my hypothetical, the tortoise we find only needs to be good at giving a success for the keyspace near the target key. Why would that one tortoise be favored by us for the whole keyspace? Also, how could any one node be *that* wonderful at returning sucesses for the entire keyspace? If it were, yes, it would eventually go out of use (due to being overloaded) for *some* of the nodes which use it, but as most nodes back-off from it, the few that continued to use it would do so for whatever keys they had found it to be best for, leading to it becoming specialized. Altruism: - I think the problem with any altruistic strategy is that it then lets non-altruistic nodes get the better of us, and hordes of people would choose a non-altruistic hack of freenet. Also, if we take your altruistic strategy, couldn't it be argued that we would be unnecessarily slowing down the tortoises even more? NGR takes care of load balancing naturally. This, I have no answer to (but there may be one - there is some information revealed by the non-altruistic strategy). not sure sure what you mean by "some information revealed" -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: Wi^H^H Redhat installer
On Wed, Nov 05, 2003 at 10:20:24AM -0800, Todd Walton wrote: > Mika Hirvonen wrote: > > > At the moment, this is a moot point, because the only JRE that will run > > the latest versions of Freenet adequately is Sun's, but if (or when) the > > Kaffe guys implement NIO and we want to support them, we might have a > > problem. > > Blackdown works. Blackdown is not even in debian non-free. Guess why. > > -todd -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] A paradoxical effect of NG routing: c.f. Prisoner paradox
On Wed, Nov 05, 2003 at 01:41:17PM +, Roger Hayter wrote: > As I understand it, the present routing method tries to predict the node > to contact which will give it the quickest response overall to its > request. This is (I will take as given) the individual node's best > strategy. However, inherent in this strategy, is a high likelihood of > any given request being QRed (because a quick QR is better than a very > slow response from a node which might ultimately be somewhat more likely > to reach the needed key than the first choice node, but which is known > to be less likely to QR quickly). But, a large number of QRs on the > network adds to network load, and, arguably, may greatly reduce the > chances of a fast node ever specialising, as the requests coming to it > are likely to be from all areas, "hoping" for a quick QR. Yes. The quickest response overall, INCLUDING RETRIES IF WE ARE REJECTED. > > Perhaps, an altruistic node would route to avoid QRs for the benefit of > the network, even if it meant it was likely to take it much longer to > get the data it wants, by trying to predict where it would be found > without QRs. > > The paradox (of course) would be apparent if a network of altruistic > nodes in fact achieved better routing, and much less pointless traffic > (I know they are not *necessarily* the same thing), and worked better. > > > Have I got a point here, or am I just arguing for bringing the original > routing algorithm back - is there a possible variant of NGRouting that > looks for quickest route but with time to QR weighted at X1000 seconds > or some such? No, QR is already heavily weighted. > -- > Roger Hayter -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] packet-level compression
On Tue, Nov 04, 2003 at 01:14:44PM -0600, Edgar Friendly wrote: > Toad <[EMAIL PROTECTED]> writes: > > > On Tue, Nov 04, 2003 at 11:48:13AM -0600, Edgar Friendly wrote: > > > Every routed request already has a UniqueID; why would we make up a > > > new ID system just for QRs? Currently, QRs are between 80 and 100 > > > bytes, I plan on having them down to 12 bytes. Dropping to a 2 byte > > > QueryID value would make QRs 6 bytes, but at these sizes, packet > > > overhead is going to dominate anyway, so there's no real benefit in a > > > smaller QueryID. I'm not sure that packet overhead won't make my > > > changes useless, but I'm willing to try and find out. If you want to > > > pitch in, feel free to start hacking on fred. > > > > I make it 17... you are missing out any field specifiers? Typical QR: > > QueryRejected > > HopsToLive=8 > > Reason=Node overloaded > > Attenuation=1 > > UniqueID=5059d54c3cd2b593 > > EndMessage > > > You're probably right; I was doing all the counting in my head, and > forgot about Reason. What kind of values does Attenuation have? I'm not sure that it's used at present, but it might be in future. > > Thelema > -- > E-mail: [EMAIL PROTECTED]Raabu and Piisu > GPG 1024D/36352AAB fpr:756D F615 B4F3 BFFC 02C7 84B7 D8D7 6ECE 3635 2AAB -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] 6281 broken
On Sun, Oct 26, 2003 at 11:25:08AM -0800, Mike Stump wrote:
> This isn't working:
>
> Oct 26, 2003 5:53:46 PM (freenet.PeerPacketMessage, write interface thread,
> NORMAL): Took 1143 seconds to send [EMAIL PROTECTED]:freenet.Message: QueryRejected
> @null @ 29166ecc7cf191: htl=1, reason=Node
> overloaded:QueryRejected{Close=false,Sustain=false,DataLength=0,{HopsToLive=1,Reason=Node
> overloaded,Attenuation=1,UniqueID=29166ecc7cf191,}}:null:true,
> prio=1(notifySuccess(null)! (last connection registered 7778 seconds ago on [EMAIL
> PROTECTED] (DSA(64a4 40e5 401b b3af 9d19 47be 0436 189c 3407
> 5eab),tcp/12.247.157.104:26694, sessions=1, presentations=1, ID=DSA(64a4 40e5 401b
> b3af 9d19 47be 0436 189c 3407 5eab)): outbound attempts=2:13/15
> Oct 26, 2003 5:53:46 PM (freenet.PeerPacketMessage, write interface thread,
> NORMAL): Took 2210 seconds to send [EMAIL PROTECTED]:freenet.Message: QueryRejected
> @null @ cbb4370c3201e6f4: htl=5, reason=Required protocol version is
> 1.47:QueryRejected{Close=false,Sustain=false,DataLength=0,{HopsToLive=5,Reason=Required
> protocol version is 1.47,Attenuation=0,UniqueID=cbb4370c3201e6f4,}}:null:true,
> prio=1(notifySuccess(null)! (last connection registered 81 seconds ago on [EMAIL
> PROTECTED] (DSA(d723 922d 6e7a 7226 6203 498d d358 e88c 03dc
> b14d),tcp/24.208.128.31:30721, sessions=1, presentations=1, ID=DSA(d723 922d 6e7a
> 7226 6203 498d d358 e88c 03dc b14d)): outbound attempts=0:0/0
> Oct 26, 2003 5:53:49 PM (freenet.PeerPacketMessage, write interface thread,
> NORMAL): Took 317 seconds to send [EMAIL PROTECTED]:freenet.Message: QueryRejected
> @null @ 8c6d4c8c4f4cbf86: htl=18, reason=Node
> overloaded:QueryRejected{Close=false,Sustain=false,DataLength=0,{HopsToLive=12,Reason=Node
> overloaded,Attenuation=1,UniqueID=8c6d4c8c4f4cbf86,}}:null:true,
> prio=1(notifySuccess(null)! (last connection registered 553 seconds ago on [EMAIL
> PROTECTED] (DSA(66d7 6419 5569 c1b1 469d 0492 702a 00a8 303e
> 5950),tcp/12.235.108.80:17704, sessions=1, presentations=1, ID=DSA(66d7 6419 5569
> c1b1 469d 0492 702a 00a8 303e 5950)): outbound attempts=1:0/1
> Oct 26, 2003 5:53:51 PM (freenet.PeerPacketMessage, write interface thread,
> NORMAL): Took 2045 seconds to send [EMAIL PROTECTED]:freenet.Message: QueryRejected
> @null @ 8498fa9f302fb500: htl=7, reason=Required protocol version is
> 1.47:QueryRejected{Close=false,Sustain=false,DataLength=0,{HopsToLive=7,Reason=Required
> protocol version is 1.47,Attenuation=0,UniqueID=8498fa9f302fb500,}}:null:true,
> prio=1(notifySuccess(null)! (last connection registered 86 seconds ago on [EMAIL
> PROTECTED] (DSA(d723 922d 6e7a 7226 6203 498d d358 e88c 03dc
> b14d),tcp/24.208.128.31:30721, sessions=1, presentations=1, ID=DSA(d723 922d 6e7a
> 7226 6203 498d d358 e88c 03dc b14d)): outbound attempts=0:0/0
One possibility for the above is that your actual internet connection is
clogged up despite the node not thinking you are using much bandwidth.
Please check this.
>
>
> I have 304 transmitting (from an 8KB/s upstream) and if my CPU was
> faster it feels like it would want to devolve into a 1500 transmitting
> to achieve the 5 bytes/sec transmit rate it wants to be. The problem
> is that general says I am using 30% of my upstream. 304 transmitting
> using 30%? Does this mean that 70% of my upstream is free or that 70%
> is used for other things? If for other things, what things?
Please check that this is sustained long term using outputBytes. Please
check how much of your output bandwidth is *actually* used using some
bandwidth monitor software or something. It could well be that the nodes
that are supplying the data are simply sending it slow, and your node is
performing reasonably.
>
> Instead, lets queue up the transmits and do the very last one queued
> first, before all others. If the upstream isn't 100% pegged, open up
> another, keep doing that until we peg the upstream. Once it is
> pegged, we don't transmit any others until it unpegs.
The next person who suggests this... never mind. Seriously, we can't
just queue sends BECAUSE WE DONT USUALLY HAVE THE DATA TO QUEUE. And if
the upstream is so much as 80% pegged, WE DO ALREADY REJECT ALL QUERIES!
>
> What we have now feels like a collapsed network that is grinding under
> its own request load. Queing up a QR for 2000 seconds just isn't
> right. Is this the design that someone wanted?
That's possible. I don't know why the QRs are queued for that time, they
should have timed out before that, could be a bug, will debug if/when I
reproduce it locally.
>
> And an indication that something is wrong:
>
> Lowest global time estimate 233249ms
> Highest global time estimate 1106205ms
>
> that's over 18 minutes. My browser times things out after 60 seconds.
> Reading web pages that take 18 minutes per link, isn't useful. Start
> with the idea that if they don't read it in the next 10 seconds, they
> never want to read it. Gear the
Re: [freenet-dev] rejections
On Wed, Nov 05, 2003 at 03:05:24AM -0500, Ken Corson wrote: > On Mon, Nov 03, 2003 at 07:10:10PM -0500, Ken Corson wrote: > >I've been thinking more about this concern. What I really want to > >observe is - for any given request that I forward, _how_many_ rejections > >occur before my request is able to hop (accepted). If this number is > >averaging something over about 5, this could be SEVERELY damaging to > >our NGR "specialized" routing intentions ... > > Toad responded: > >I doubt it's less than that... I'll trace a few high HTL requests > >through my log files... I wonder if there is a stat... if there isn't > >there should be... > > Toad- > Thank you so much for implementing searchFailedCount! My numbers > by hour (trimmed to reduce width) are this so far : > > 9:00:00 PM EST0.95897 2.10480 0.0 17.0390 > 10:00:00 PM EST 2.44145 4.04589 0.0 18.02041 > 11:00:00 PM EST 2.93011 3.45867 NaN NaN 1989 > 12:00:00 AM EST 3.23421 3.76093 0.0 19.02297 > 1:00:00 AM EST4.21296 4.11976 NaN NaN 1681 > 2:00:00 AM EST3.68234 3.65358 0.0 20.01807 > 3:00:00 AM EST2.23778 3.04152 0.0 15.03049 > > What can I say ? Probably it's time for me to shut up about the > impact of QR !! I mentioned a strange phenomenon in how my node > (many nodes?) seems to "surge" it's rate of rejecting. What I > have gleaned from looking at individual searchFailedCount > events, then inboundQueryRejecteds, and finally > outboundAggregateRequests is that my outbound requests are also > fluctuating to a high degree (unsure how much this matters). > They certainly are not smoothly distributed in time, but I'll > have to monitor it some more. > > Thank you again for adding this :) > > I don't think the implications of the numbers above are serious - > at least when using 50 peers. Fewer peers, greater impact. Does > my (bad) theory about poor routing maybe suggest that > maxRoutingSteps should not exceed a small fraction of rtMaxNodes ? > > Looks like the only way for me to earn some developer points > is to start writing some code :) I sure thought I had something > here, but I guessed my extrapolated math wouldn't hold water. I'm not sure. I think it could still be a problem. > > -Ken -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: As I understand it, the present routing method tries to predict the node to contact which will give it the quickest response overall to its request. Almost. It chooses the node for which estimate() is the least. As I understand it (please someone correct me if I'm wrong) estimate() is *roughly* defined as: pSuccess*tSuccess + pFailure*(tFailure+tGlobalSuccess) where all of the below are a function of location in the keyspace: pSuccess=chance of the node returning data tSuccess=time it will take the node to return data if successful pFailure=1-pSuccess tFailure=time it will take the node to fail tGlodalSuccess=time it will take to retrieve data successfully if we went to another node. I believe this is estimated by averaging all the past tSuccess:es *starting from our node*. So if, in the past, we had to try 10 different nodes to get a success for a key in this area, all the time we spent trying the 10 nodes is added-in to tGlobalSuccess. This is (I will take as given) the individual node's best strategy. However, inherent in this strategy, is a high likelihood of any given request being QRed (because a quick QR is better than a very slow response from a node which might ultimately be somewhat more likely to reach the needed key than the first choice node, but which is known to be less likely to QR quickly). The speed of QR affects tFailure in the above formula. In your example, let's also assume that the node which retrieves a quick QR (let's call it the "hare") also will be less likely to return a success, but if successful will return data more quickly than the other node (let's call that the "tortoise"), which will be more likely to succeed but slower to respond either successfully or unsuccessfully. The best option will not always be the hare. The higher tGlobalSuccess is, the better the tortoise becomes. Lower it, and the hare is a better option. The system should balance itself out. Suppose we started off trying only hares. We might eventually see that tGlobalSuccess was getting large because of all the failed initial attemps which many times resulted in us finally succeeding with a tortoise. Since tGlobalSuccess would then be larger, we would tend to start favoring tortoises more. The reverse is also true, and we should achieve a balance. But, a large number of QRs on the network adds to network load, and, arguably, may greatly reduce the chances of a fast node ever specialising, as the requests coming to it are likely to be from all areas, "hoping" for a quick QR. Perhaps, an altruistic node would route to avoid QRs for the benefit of the network, even if it meant it was likely to take it much longer to get the data it wants, by trying to predict where it would be found without QRs. Or maybe it doesn't need to be altruisitic. If the hares get overloaded, their pSuccess will get even worse, tGlobalSuccess will then start to rise, and we will start favoring tortoises. The paradox (of course) would be apparent if a network of altruistic nodes in fact achieved better routing, and much less pointless traffic (I know they are not *necessarily* the same thing), and worked better. Have I got a point here, or am I just arguing for bringing the original routing algorithm back - is there a possible variant of NGRouting that looks for quickest route but with time to QR weighted at X1000 seconds or some such? See if the above answers your concerns. -Martin Potentially, yes it does: but I gathered that some people were concerned that the noise from QRs and the requests eliciting them was so great as to drown out any stimulus to specialisation. Presumably if the specialisation stimulus is so small that it is less than the random variation in effectiveness resulting from, say, caching a particular file, then there will never be any net effect from routing on a node's specialisation: the granularity of a node's randomly varying "specialisation" will be greater than the size of any net stimulus to alter it. Specialization: --- I guess the idea is that if there are 10 areas of specialization, and we know 5 that specialize in the first area, but all 5 regularly QR, then we begin diluting the specialization of the other nodes which aren't specialized in the first area. Did I state the problem correctly? Not entirely, suppose we start with 10 nodes which are only 0.1% specialised due to random variations in what they have cached, and are never likely to get more specialised. If so, I would say the problem here is that all 5 nodes specialized near this key are hares. If the hares become burdened with QR:s, shouldn't NGR lead to us eventually favoring a sixth (tortoise) which also becomes speci
[freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
Roger Hayter wrote: In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: As I understand it, the present routing method tries to predict the node to contact which will give it the quickest response overall to its request. Almost. It chooses the node for which estimate() is the least. As I understand it (please someone correct me if I'm wrong) estimate() is *roughly* defined as: pSuccess*tSuccess + pFailure*(tFailure+tGlobalSuccess) where all of the below are a function of location in the keyspace: pSuccess=chance of the node returning data tSuccess=time it will take the node to return data if successful pFailure=1-pSuccess tFailure=time it will take the node to fail tGlodalSuccess=time it will take to retrieve data successfully if we went to another node. I believe this is estimated by averaging all the past tSuccess:es *starting from our node*. So if, in the past, we had to try 10 different nodes to get a success for a key in this area, all the time we spent trying the 10 nodes is added-in to tGlobalSuccess. This is (I will take as given) the individual node's best strategy. However, inherent in this strategy, is a high likelihood of any given request being QRed (because a quick QR is better than a very slow response from a node which might ultimately be somewhat more likely to reach the needed key than the first choice node, but which is known to be less likely to QR quickly). The speed of QR affects tFailure in the above formula. In your example, let's also assume that the node which retrieves a quick QR (let's call it the "hare") also will be less likely to return a success, but if successful will return data more quickly than the other node (let's call that the "tortoise"), which will be more likely to succeed but slower to respond either successfully or unsuccessfully. The best option will not always be the hare. The higher tGlobalSuccess is, the better the tortoise becomes. Lower it, and the hare is a better option. The system should balance itself out. Suppose we started off trying only hares. We might eventually see that tGlobalSuccess was getting large because of all the failed initial attemps which many times resulted in us finally succeeding with a tortoise. Since tGlobalSuccess would then be larger, we would tend to start favoring tortoises more. The reverse is also true, and we should achieve a balance. But, a large number of QRs on the network adds to network load, and, arguably, may greatly reduce the chances of a fast node ever specialising, as the requests coming to it are likely to be from all areas, "hoping" for a quick QR. Perhaps, an altruistic node would route to avoid QRs for the benefit of the network, even if it meant it was likely to take it much longer to get the data it wants, by trying to predict where it would be found without QRs. Or maybe it doesn't need to be altruisitic. If the hares get overloaded, their pSuccess will get even worse, tGlobalSuccess will then start to rise, and we will start favoring tortoises. The paradox (of course) would be apparent if a network of altruistic nodes in fact achieved better routing, and much less pointless traffic (I know they are not *necessarily* the same thing), and worked better. Have I got a point here, or am I just arguing for bringing the original routing algorithm back - is there a possible variant of NGRouting that looks for quickest route but with time to QR weighted at X1000 seconds or some such? See if the above answers your concerns. -Martin Potentially, yes it does: but I gathered that some people were concerned that the noise from QRs and the requests eliciting them was so great as to drown out any stimulus to specialisation. Presumably if the specialisation stimulus is so small that it is less than the random variation in effectiveness resulting from, say, caching a particular file, then there will never be any net effect from routing on a node's specialisation: the granularity of a node's randomly varying "specialisation" will be greater than the size of any net stimulus to alter it. Specialization: --- I guess the idea is that if there are 10 areas of specialization, and we know 5 that specialize in the first area, but all 5 regularly QR, then we begin diluting the specialization of the other nodes which aren't specialized in the first area. Did I state the problem correctly? If so, I would say the problem here is that all 5 nodes specialized near this key are hares. If the hares become burdened with QR:s, shouldn't NGR lead to us eventually favoring a sixth (tortoise) which also becomes specialized? Altruism: - I think the problem with any altruistic strategy is that it then lets non-altruistic nodes get the better of us, and hordes of people would choose a non-altruistic hack of freenet. Also, if we take your altruistic strategy, couldn't it be argued
Re: [freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Roger Hayter wrote: As I understand it, the present routing method tries to predict the node to contact which will give it the quickest response overall to its request. Almost. It chooses the node for which estimate() is the least. As I understand it (please someone correct me if I'm wrong) estimate() is *roughly* defined as: pSuccess*tSuccess + pFailure*(tFailure+tGlobalSuccess) where all of the below are a function of location in the keyspace: pSuccess=chance of the node returning data tSuccess=time it will take the node to return data if successful pFailure=1-pSuccess tFailure=time it will take the node to fail tGlodalSuccess=time it will take to retrieve data successfully if we went to another node. I believe this is estimated by averaging all the past tSuccess:es *starting from our node*. So if, in the past, we had to try 10 different nodes to get a success for a key in this area, all the time we spent trying the 10 nodes is added-in to tGlobalSuccess. This is (I will take as given) the individual node's best strategy. However, inherent in this strategy, is a high likelihood of any given request being QRed (because a quick QR is better than a very slow response from a node which might ultimately be somewhat more likely to reach the needed key than the first choice node, but which is known to be less likely to QR quickly). The speed of QR affects tFailure in the above formula. In your example, let's also assume that the node which retrieves a quick QR (let's call it the "hare") also will be less likely to return a success, but if successful will return data more quickly than the other node (let's call that the "tortoise"), which will be more likely to succeed but slower to respond either successfully or unsuccessfully. The best option will not always be the hare. The higher tGlobalSuccess is, the better the tortoise becomes. Lower it, and the hare is a better option. The system should balance itself out. Suppose we started off trying only hares. We might eventually see that tGlobalSuccess was getting large because of all the failed initial attemps which many times resulted in us finally succeeding with a tortoise. Since tGlobalSuccess would then be larger, we would tend to start favoring tortoises more. The reverse is also true, and we should achieve a balance. But, a large number of QRs on the network adds to network load, and, arguably, may greatly reduce the chances of a fast node ever specialising, as the requests coming to it are likely to be from all areas, "hoping" for a quick QR. Perhaps, an altruistic node would route to avoid QRs for the benefit of the network, even if it meant it was likely to take it much longer to get the data it wants, by trying to predict where it would be found without QRs. Or maybe it doesn't need to be altruisitic. If the hares get overloaded, their pSuccess will get even worse, tGlobalSuccess will then start to rise, and we will start favoring tortoises. The paradox (of course) would be apparent if a network of altruistic nodes in fact achieved better routing, and much less pointless traffic (I know they are not *necessarily* the same thing), and worked better. Have I got a point here, or am I just arguing for bringing the original routing algorithm back - is there a possible variant of NGRouting that looks for quickest route but with time to QR weighted at X1000 seconds or some such? See if the above answers your concerns. -Martin Potentially, yes it does: but I gathered that some people were concerned that the noise from QRs and the requests eliciting them was so great as to drown out any stimulus to specialisation. Presumably if the specialisation stimulus is so small that it is less than the random variation in effectiveness resulting from, say, caching a particular file, then there will never be any net effect from routing on a node's specialisation: the granularity of a node's randomly varying "specialisation" will be greater than the size of any net stimulus to alter it. -- Roger Hayter ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: Wi^H^H Redhat installer
Mika Hirvonen wrote: > At the moment, this is a moot point, because the only JRE that will run > the latest versions of Freenet adequately is Sun's, but if (or when) the > Kaffe guys implement NIO and we want to support them, we might have a > problem. Blackdown works. -todd ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: A paradoxical effect of NG routing: c.f. Prisoner paradox
Roger Hayter wrote: As I understand it, the present routing method tries to predict the node to contact which will give it the quickest response overall to its request. Almost. It chooses the node for which estimate() is the least. As I understand it (please someone correct me if I'm wrong) estimate() is *roughly* defined as: pSuccess*tSuccess + pFailure*(tFailure+tGlobalSuccess) where all of the below are a function of location in the keyspace: pSuccess=chance of the node returning data tSuccess=time it will take the node to return data if successful pFailure=1-pSuccess tFailure=time it will take the node to fail tGlodalSuccess=time it will take to retrieve data successfully if we went to another node. I believe this is estimated by averaging all the past tSuccess:es *starting from our node*. So if, in the past, we had to try 10 different nodes to get a success for a key in this area, all the time we spent trying the 10 nodes is added-in to tGlobalSuccess. > This is (I will take as given) the individual node's best strategy. However, inherent in this strategy, is a high likelihood of any given request being QRed (because a quick QR is better than a very slow response from a node which might ultimately be somewhat more likely to reach the needed key than the first choice node, but which is known to be less likely to QR quickly). The speed of QR affects tFailure in the above formula. In your example, let's also assume that the node which retrieves a quick QR (let's call it the "hare") also will be less likely to return a success, but if successful will return data more quickly than the other node (let's call that the "tortoise"), which will be more likely to succeed but slower to respond either successfully or unsuccessfully. The best option will not always be the hare. The higher tGlobalSuccess is, the better the tortoise becomes. Lower it, and the hare is a better option. The system should balance itself out. Suppose we started off trying only hares. We might eventually see that tGlobalSuccess was getting large because of all the failed initial attemps which many times resulted in us finally succeeding with a tortoise. Since tGlobalSuccess would then be larger, we would tend to start favoring tortoises more. The reverse is also true, and we should achieve a balance. > But, a large number of QRs on the network adds to network load, and, arguably, may greatly reduce the chances of a fast node ever specialising, as the requests coming to it are likely to be from all areas, "hoping" for a quick QR. Perhaps, an altruistic node would route to avoid QRs for the benefit of the network, even if it meant it was likely to take it much longer to get the data it wants, by trying to predict where it would be found without QRs. Or maybe it doesn't need to be altruisitic. If the hares get overloaded, their pSuccess will get even worse, tGlobalSuccess will then start to rise, and we will start favoring tortoises. The paradox (of course) would be apparent if a network of altruistic nodes in fact achieved better routing, and much less pointless traffic (I know they are not *necessarily* the same thing), and worked better. Have I got a point here, or am I just arguing for bringing the original routing algorithm back - is there a possible variant of NGRouting that looks for quickest route but with time to QR weighted at X1000 seconds or some such? See if the above answers your concerns. -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Targeted DoS in TCP/IP
Arg, I just realised another censorship threat. The more niehbors a node has that know his specialization, the higher the chance that cancer nodes will know about him. Example say we have an N=1 node network where nodes on average know about m=500 nieghbors, and f=1% of the nodes are cancer nodes (100 nodes). Then the chance that a particular nodes doesn't have a cancer node that knows about is: (1-f)^m = 0.00657 That means more than 99% of those "target nodes", where a particular piece of data is likely to be stored, have thier IP:ports known to the adversary. If he knows thier specialization, he can blast the crap out of them with a SYN flood. That just sucks! I thought I'd bring it up. I don't think there's much we can do about it except reduce the neighbor count, which may increase routing times. Maybe an option would be the "hostile enviroment" topology, where you only connect to people you can trust. Anyone else got an idea? Boy it would be sweet if we all had something better than TCP/IP like ATM where you can build connections with QoS guaranties, maybe in 20 years. DoS floods would be a thing of the past. Maybe we should just stay happy with a network that is broadly censorship resistant: You can destroy all of the data some of the time(by DNFing) and some of the data some of the time, (with this targetted attack) but you can't destroy all of the data all of the time. (YoYo is everywhere) This is good enough to piss of the RIAA, or anyone out to censor everything in freenet. I wish we could do better though. __ Gesendet von Yahoo! Mail - http://mail.yahoo.de Logos und Klingeltöne fürs Handy bei http://sms.yahoo.de ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Re: Results of new stats in 6313+
Martin Stone Davis wrote: Here's a new thread for all interested to post results of the new stats Okay, Martin. Here is the first post. You are so cool to have started this thread. I hope others post their results by replying to your thread and not to this one. Some notes on the results below: 1. There are many Minimum/Maximum values that make no sense. For example on 11/4/03 11PM of searchFailedCount, mean is 2.9 but min/max is NaN. What's going on here? 2. diffSearchSuccessTime is not averaging 0. Instead it looks like actual>predicted. 3. diffTransferRate is not averaging 0. Instead it looks like actual -Martin Title: searchFailedCount(hour) Continuous: searchFailedCount Counts the number of QueryRejected's or timeouts before a given data request DNFed, succeeded or had a transfer fail. Aggregated over every minute . Recorded aggregates over hour: TimeMeanStd. Dev.MinimumMaximumObservations 11/4/03 12:00:00 PM PSTNaNNaNNaNNaN0 11/4/03 1:00:00 PM PST6.4329643296432975.7699745749887580.024.0813 11/4/03 2:00:00 PM PST4.5390286209887255.3837945727696110.024.02306 11/4/03 3:00:00 PM PST9.4582441113490376.004988237313270.024.0934 11/4/03 4:00:00 PM PST7.9739130434782615.0378697329233270.024.0345 11/4/03 5:00:00 PM PST6.377919320594486.20489765286215440.024.01884 11/4/03 6:00:00 PM PST4.2560706401766014.83231707560.024.01812 11/4/03 7:00:00 PM PST4.108628181253884.9935355769172450.024.01611 11/4/03 8:00:00 PM PST4.9282700421940935.3376615677158460.024.02133 11/4/03 9:00:00 PM PST2.11051546391752563.47031148769762240.024.02425 11/4/03 10:00:00 PM PST2.17430441898527033.2372162040944080.024.03666 11/4/03 11:00:00 PM PST2.90670170827858074.057162347165283NaNNaN3044 11/5/03 12:00:00 AM PST3.46953937592867774.9618058766123090.022.01346 11/5/03 1:00:00 AM PST5.1133550488599355.2531902656460550.022.01535 11/5/03 2:00:00 AM PST3.9337386018237084.1233842598057550.022.01645 11/5/03 3:00:00 AM PST3.03260146373918853.2332125614566554NaNNaN1503 11/5/03 4:00:00 AM PST4.4719020172910666.16581174278733NaNNaN1388 11/5/03 5:00:00 AM PST4.7712230215827345.1568102357636710.05.01390 11/5/03 6:00:00 AM PST7.3253697383390216.41124725419273660.033.01758 11/5/03 7:00:00 AM PST4.1715089034676664.0324218243947380.025.01067 11/5/03 8:00:00 AM PST2.2292243767313023.2195184717736027NaNNaN1444 11/5/03 9:00:00 AM PST3.669731258840173.48251770719793670.011.01414 Title: diffSearchSuccessTime(hour) Continuous: diffSearchSuccessTime Difference between predicted search time for a successful request and the actual search time. Should average at 0. Aggregated over every minute . Recorded aggregates over hour: TimeMeanStd. Dev.MinimumMaximumObservations 11/4/03 3:00:00 PM PST-20302.42857142857238493.16766586482-106195.0-106195.07 11/4/03 4:00:00 PM PST-6762.08999.67377186529-26169.0-2141.09 11/4/03 5:00:00 PM PST8554.88571428571425160.35643950668NaNNaN35 11/4/03 6:00:00 PM PST15145.36585365853751139.22483610604NaNNaN41 11/4/03 7:00:00 PM PST24816.88985819.02393468807NaNNaN27 11/4/03 8:00:00 PM PST11151.7435897435921248.8489418503851712.01712.039 11/4/03 9:00:00 PM PST27655.0188679245356170.55845660216-3562.02063.053 11/4/03 10:00:00 PM PST31973.34020618556772788.68438398793121.0181312.097 11/4/03 11:00:00 PM PST19419.12280701754269741.85509359042NaNNaN57 11/5/03 12:00:00 AM PST48340.64285714285579861.71735097644NaNNaN14 11/5/03 1:00:00 AM PST2278.1555612237.100646200985-1158.011240.045 11/5/03 2:00:00 AM PST25754.0851063829834752.8749538483NaNNaN47 11/5/03 3:00:00 AM PST21280.666837617.758578798304NaNNaN24 11/5/03 4:00:00 AM PST31146.45454545454449530.96297865683NaNNaN22 11/5/03 5:00:00 AM PST31238.277791946.57473014614NaNNaN36 11/5/03 6:00:00 AM PST7750.28846153846265630.38524782483NaNNaN52 11/5/03 7:00:00 AM PST11598.791640948.725346174691344.0137406.024 11/5/03 8:00:00 AM PST30156.666855385.72670824136NaNNaN27 11/5/03 9:00:00 AM PST2753.476190476190470946.97368430809NaNNaN21 Title: diffTransferRate(hour) Continuous: diffTransferRate Difference between predicted transfer rate and actual transfer rate for a successful request. Should average to 0. Aggregated over every minute . Recorded aggregates over hour: TimeMeanStd. Dev.MinimumMaximumObservations 11/4/03 1:00:00 PM PST-14.93959708711083110.007745415928815-28.181878279883378-7.34697332652091710 11/4/03 2:00:00 PM PST-10.22645846637224312.167204952361052NaNNaN59 11/4/03 3:00:00 PM PST-4.4465025358818663.893083690461956-10.6441356526928-10.64413565269285 11/4/03 4:00:00 PM PST-1.12606283913425152.675672457378163-4.4740230113595362.02592074296262059 11/4/03 5:00:00 PM PST-7.5556222740942297.2315722597158345NaNNaN28 11/4/03 6:00:00 PM PST-11.0567451764417489.976631804590657NaNNaN38 11/4/03 7:00:00 PM PST-13.78398317801290912.302885314189258NaNNaN23 11/4/03 8:00:00 PM PST-13.5521231072037327.746280844791631-19.536427621865577-19.53642762186557731 11/4/03 9:00:00 PM PST-13.016867283201487.332234602199145-21.7512334183
[freenet-CVS] freenet/src/freenet PeerHandler.java, 1.43, 1.44 OpenConnectionManager.java, 1.162, 1.163
Update of /cvsroot/freenet/freenet/src/freenet
In directory sc8-pr-cvs1:/tmp/cvs-serv16370/src/freenet
Modified Files:
PeerHandler.java OpenConnectionManager.java
Log Message:
Display message failed-to-sent count in OCM PeerHandler mode (max details).
Index: PeerHandler.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/PeerHandler.java,v
retrieving revision 1.43
retrieving revision 1.44
diff -u -w -r1.43 -r1.44
--- PeerHandler.java4 Nov 2003 16:53:23 - 1.43
+++ PeerHandler.java5 Nov 2003 14:02:54 - 1.44
@@ -142,9 +142,11 @@
{
String messageType;
int sent=0;
+ int sendFailed=0;
int received=0;
+ int receiveFailed=0;
}
- public String toHTML(){
+ public String toHTML(boolean renderFailCount){
String retval = "";
retval += "";
Hashtable h = new Hashtable();
@@ -155,7 +157,6 @@
String key = (String) eSent.nextElement();
String messageType = key.substring(0,
key.indexOf(":"));
String messageStatus =
key.substring(key.indexOf(":")+1, key.length());
- if
(messageStatus.equalsIgnoreCase(String.valueOf(true))) {
int sentCount = ((myInt)
messagesSentByTypeAndStatus.get(key)).intValue;
MessageCountByType val =
(MessageCountByType) h.get(messageType);
if (val == null){
@@ -163,9 +164,11 @@
val.messageType = messageType;
h.put(val.messageType, val);
}
+ if
(messageStatus.equalsIgnoreCase(String.valueOf(true)))
val.sent += sentCount;
- h.put(messageType, val); //dont care
about performance;
- } //What to do else
+ else
+ val.sendFailed += sentCount;
+
}
}
synchronized (messagesReceivedByTypeAndStatus) {
@@ -174,7 +177,7 @@
String key = (String) eSent.nextElement();
String messageType = key.substring(0,
key.indexOf(":"));
String messageStatus =
key.substring(key.indexOf(":")+1, key.length());
- if
(messageStatus.equalsIgnoreCase(String.valueOf(true))) {
+
int receivedCount = ((myInt)
messagesReceivedByTypeAndStatus.get(key)).intValue;
MessageCountByType val =
(MessageCountByType) h.get(messageType);
if (val == null){
@@ -182,16 +185,23 @@
val.messageType = messageType;
h.put(val.messageType, val);
}
+ if
(messageStatus.equalsIgnoreCase(String.valueOf(true)))
val.received += receivedCount;
-
- } //What to do else
+ else
+ val.receiveFailed += receivedCount;
}
}
if (h.size() > 0) {
+ if(renderFailCount)
+ retval += "TypeSent (failed)/Received (failed)";
+ else
retval += "TypeSent/Received";
Enumeration e = h.keys();
while (e.hasMoreElements()) {
MessageCountByType val = (MessageCountByType)
h.get(e.nextElement());
+ if(renderFailCount)
+ retval += "" +
val.messageType + "" + val.sent +
" ("+val.sendFailed+")/" + val.received +
" ("+val.receiveFailed+")";
+ else
retval
[freenet-dev] A paradoxical effect of NG routing: c.f. Prisoner paradox
As I understand it, the present routing method tries to predict the node to contact which will give it the quickest response overall to its request. This is (I will take as given) the individual node's best strategy. However, inherent in this strategy, is a high likelihood of any given request being QRed (because a quick QR is better than a very slow response from a node which might ultimately be somewhat more likely to reach the needed key than the first choice node, but which is known to be less likely to QR quickly). But, a large number of QRs on the network adds to network load, and, arguably, may greatly reduce the chances of a fast node ever specialising, as the requests coming to it are likely to be from all areas, "hoping" for a quick QR. Perhaps, an altruistic node would route to avoid QRs for the benefit of the network, even if it meant it was likely to take it much longer to get the data it wants, by trying to predict where it would be found without QRs. The paradox (of course) would be apparent if a network of altruistic nodes in fact achieved better routing, and much less pointless traffic (I know they are not *necessarily* the same thing), and worked better. Have I got a point here, or am I just arguing for bringing the original routing algorithm back - is there a possible variant of NGRouting that looks for quickest route but with time to QR weighted at X1000 seconds or some such? -- Roger Hayter ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
RE: [freenet-dev] (To Iakin) Clarification of tTransferRate, etc. graphs
I would prefer to keep the (0,0) coordinate at bottom-left all the time. This will probably avoid some confusion amongst users.. Would you say that the transfer rate graphs, both at /servlet/nodeinfo/networking/ocm?setLevel=2 and /servlet/nodestatus/nodestatus.html are upside down? I haven't really checked into that, I just used the code that was already present. If this is an ok solution and I get a list of which of the graphs that are upside down from someone I'll fix it immediately. /N > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Martin > Stone Davis > Sent: den 5 november 2003 01:53 > To: [EMAIL PROTECTED] > Subject: [freenet-dev] (To Iakin) Clarification of > tTransferRate, etc. graphs > > > Could someone (probably Iakin) clarify what is meant by the > tTransferRate graphs under /servlet/nodestatus/nodestatus.html ? > > The graph should indicate whether it's bytes per second or > seconds per > byte. Also, indicate (maybe with an L-shaped corner) where the 0,0 > point is. The same could go for all the other graphs. > > Thanks, > -Martin > > > ___ > Devl mailing list > [EMAIL PROTECTED] > http://dodo.freenetproject.org/cgi-> bin/mailman/listinfo/devl > ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Re: Usability improvement ideas
In message <[EMAIL PROTECTED]>, Martin Stone Davis <[EMAIL PROTECTED]> writes Toad wrote: On Tue, Nov 04, 2003 at 09:10:08AM -0500, Daves Lists wrote: Always active jobs. Here's what I generally look like once the node has been up and running a few hours. I tried bringing the threads up to support the connections but that always seemed to cause the node to slow down and then I would reject on routing time or messagesendtime. Total pooled threads 455 Available pooled threads 189 Pooled threads in use 266 Pooled Thread Consumers Class Threads used Checkpoint: Connection opener 28 freenet.interfaces.LocalNIOInterface$ConnectionShell 1 freenet.interfaces.PublicNIOInterface$ConnectionShell 173 freenet.node.states.data.DataStateInitiator 55 freenet.node.states.data.TrailerWriteCallbackMessage:true:true 9 Oooh, interesting. If the reason for QRing is too many active jobs, and the reason for that is connections... then rather than NIO'ing connections, the solution is to implement multiplexing. But I need to know whether this is characteristic. It's not typical for me. 99% of the time, I QR due to bw limit. Current routingTime41ms Curent messageSendTimeRequest 227ms Pooled threads running jobs72 (30%) Pooled threads which are idle 51 Current upstream bandwidth usage 10991 bytes/second (89.4%) Reason for QueryRejecting requests:Estimated load (100%) > overloadLow (85%) Total pooled threads 123 Available pooled threads 53 Pooled threads in use 70 Class Threads used Checkpoint: Connection opener 4 Checkpoint: Native Filesystem Directory checkpoint 1 freenet.interfaces.LocalNIOInterface$ConnectionShell 3 freenet.interfaces.PublicNIOInterface$ConnectionShell 3 freenet.node.states.data.DataStateInitiator43 freenet.node.states.data.TrailerWriteCallbackMessage:true:true 16 -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl Also bandwidth here, but with messageSendTimeRequest usually hovering around the cut off, and occasionally exceeding it. Cutting max. connections to 256 makes no discernible difference to this. -- Roger Hayter ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] rejections
On November 05, 2003 03:05 am, Ken Corson wrote: > On Mon, Nov 03, 2003 at 07:10:10PM -0500, Ken Corson wrote: > >I've been thinking more about this concern. What I really want to > >observe is - for any given request that I forward, _how_many_ rejections > >occur before my request is able to hop (accepted). If this number is > >averaging something over about 5, this could be SEVERELY damaging to > >our NGR "specialized" routing intentions ... > > Toad responded: > > I doubt it's less than that... I'll trace a few high HTL requests > > through my log files... I wonder if there is a stat... if there isn't > > there should be... > > Toad- >Thank you so much for implementing searchFailedCount! My numbers > by hour (trimmed to reduce width) are this so far : On the other hand I see: 11/4/03 6:00:00 PM EST 4.530973451327434 5.028147499730006 3.0 7.0 791 11/4/03 7:00:00 PM EST 5.724611161939616 4.807284989900680.0 24.0 1093 11/4/03 8:00:00 PM EST 6.183456183456183 5.174567837732911 0.0 24.0 1221 11/4/03 9:00:00 PM EST 5.308146399055494.622201253996054 0.0 23.0 847 11/4/03 10:00:00 PM EST 5.148215919487649 4.404874612170076 0.0 26.0 1093 11/4/03 11:00:00 PM EST 5.554545454545455 4.757610984187850.0 24.0 770 11/5/03 12:00:00 AM EST 5.704977375565611 4.683634650207609 0.0 20.0 1105 11/5/03 1:00:00 AM EST 5.5396370582617 5.262630025433891 0.0 23.02094 11/5/03 2:00:00 AM EST 6.813329992483087 6.203522542955120.0 27.0 3991 11/5/03 3:00:00 AM EST 6.256900673766572 5.930472952098081 0.0 27.0 4601 11/5/03 4:00:00 AM EST 5.618973561430793 5.430659256230049 0.0 25.0 1286 11/5/03 5:00:00 AM EST 5.934179222839017 5.043582821662076 0.0 24.0 1261 11/5/03 6:00:00 AM EST 5.119897959183674 5.300830475902776 0.0 26.0 1176 11/5/03 7:00:00 AM EST 5.624183006535947 5.198050155142540.0 25.0 1224 So maybe the jury should not decide just yet. Ed ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
RE: [freenet-dev] Re: Re: Wi^H^H Redhat installer
There's some rpm stuff in build.xml and a basic .spec file in the
SPEC dir on cvs I did a while ago.
Matt
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Mika Hirvonen
Sent: Wednesday, 5 November 2003 8:59 AM
To: Discussion of development issues
Subject: Re: [freenet-dev] Re: Re: Wi^H^H Redhat installer
[EMAIL PROTECTED] wrote:
>>I have tinkered with Freenet rpms, but the problem is how to detect
>>the
>>various JREs (Kaffe, Sun, Blackdown, IBM etc) so that other package
>>management tools (like apt-rpm) wouldn't unnecessarily install a JRE.
>>
>>
>
>Ideally we'll just look for >= jre-1.4.1 and if the user has installed
>the jre
>
>
It's not that easy, even with just Sun's packages. What if the user
already has the SDK installed (which contains a working JRE)? One
solution might be wrapper packages. Each package would require a
specific version of the JRE or SDK, satisfy the java requirement
(Provides: java) and contain some symlinks that would point from $bindir
to the JRE/SDK's bin directory.
>The real problem is how to set up the installation in an uniform way.
>Putting everything in one directory as under windoze doesn't work; how
>about the following default placement for *nix:
>
>freenet.conf -> /etc/
>
>
>start-freenet.sh, stop-freenet.sh -> $bindir
>
>
How about a sysV-style init script?
>freenet.jar, freenet-ext.jar -> $libdir
>seednodes.ref, store, all routing-related files -> $opt/freenet
>
>
How about %{_var]/spool/fred?
--
Mika Hirvonen <[EMAIL PROTECTED]>
http://nightwatch.mine.nu/
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Results of new stats in 6313+
Here's a new thread for all interested to post results of the new stats added in 6312. Interesting stats can be found at: http://localhost:/servlet/nodestatus/diagnostics/searchFailedCount/hour http://localhost:/servlet/nodestatus/diagnostics/diffSearchSuccessTime/hour http://localhost:/servlet/nodestatus/diagnostics/diffTransferRate/hour and possibly also: http://localhost:/servlet/nodestatus/diagnostics/absDiffSearchSuccessTime/hour http://localhost:/servlet/nodestatus/diagnostics/absDiffTransferRate/hour and there may be more... I don't know. -Martin ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Open Question
Andrew Rodland wrote: I run a transient (99% of the time) node. Should I delete my routing table when I move from home (modem) to friend's (fast cable)? Since the downstream is about 20 times faster, I can see potential for routing to get really screwed when I change to and from -- but how long would it probably last? Ideally you would have two separate Freenet installations for each but failing that I don't think deleting your RT would really help. Your node has valuable knowledge which will still be valuable even on a different connection. Ian. ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] rejections
On Mon, Nov 03, 2003 at 07:10:10PM -0500, Ken Corson wrote: I've been thinking more about this concern. What I really want to observe is - for any given request that I forward, _how_many_ rejections occur before my request is able to hop (accepted). If this number is averaging something over about 5, this could be SEVERELY damaging to our NGR "specialized" routing intentions ... Toad responded: I doubt it's less than that... I'll trace a few high HTL requests through my log files... I wonder if there is a stat... if there isn't there should be... Toad- Thank you so much for implementing searchFailedCount! My numbers by hour (trimmed to reduce width) are this so far : 9:00:00 PM EST 0.95897 2.10480 0.0 17.0390 10:00:00 PM EST 2.44145 4.04589 0.0 18.02041 11:00:00 PM EST 2.93011 3.45867 NaN NaN 1989 12:00:00 AM EST 3.23421 3.76093 0.0 19.02297 1:00:00 AM EST 4.21296 4.11976 NaN NaN 1681 2:00:00 AM EST 3.68234 3.65358 0.0 20.01807 3:00:00 AM EST 2.23778 3.04152 0.0 15.03049 What can I say ? Probably it's time for me to shut up about the impact of QR !! I mentioned a strange phenomenon in how my node (many nodes?) seems to "surge" it's rate of rejecting. What I have gleaned from looking at individual searchFailedCount events, then inboundQueryRejecteds, and finally outboundAggregateRequests is that my outbound requests are also fluctuating to a high degree (unsure how much this matters). They certainly are not smoothly distributed in time, but I'll have to monitor it some more. Thank you again for adding this :) I don't think the implications of the numbers above are serious - at least when using 50 peers. Fewer peers, greater impact. Does my (bad) theory about poor routing maybe suggest that maxRoutingSteps should not exceed a small fraction of rtMaxNodes ? Looks like the only way for me to earn some developer points is to start writing some code :) I sure thought I had something here, but I guessed my extrapolated math wouldn't hold water. -Ken ___ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
