On Dec 10, 2007, at 6:00 PM, Matthew Toseland wrote:
> [...]
>> If we are leaving our stores behind, I would predict that for many
>> nodes (running suffecient to have meaningful stats) that the
>> 'storeDist' value may become arbitrary. And if it approaches the
>> 'furthestSuccess' value the store is "left-behind"; whereas the
>> cacheDist will maintain approx. the same (once reliable).
>
> IMHO we need the histogram, or at least a measure of spread, not
> just an
> average key.
>> Do you want this in the trunk? I don't think that I have the time to
>> do much data collection.
>
> Your diff is empty. Might be a problem on my end though.
It came up alright in the message coming back to me through the list,
but I've pasted it on the end of the message to avoid any attachment
trouble.
On Dec 9, 2007, at 11:47 AM, David Sowder (Zothar) wrote:
> As I see it currently, stuff being kicked out of the datastore isn't
> happening except when the datastore is being shrunk since AFAIK,
> nobody's datastore is filling, only their datacache is.
On Dec 10, 2007, at 11:27 AM, Robert Hailey wrote:
>> Not true. We have mechanisms to get popular data back into the
>> datastore. The
>> main one is 1 in every 100 successful requests gets turned into an
>> insert
>> post-completion.
>
>
> That is intriguing. I wonder how much of my 0.3% store-hit-successes
> (my present value) are actually blocks recently-inserted from the 1%
> healing (i.e. bleed-over from the cache).
I've got another idea from the code I saw while forming this stats
diff...
Whereas:
(1) presently we always check the datastore before the datacache,
(2) the datacache has a 8-9x better hit rate,
(3) I suspect that the hits from the store may actually be in the
cache as well
I think that we should reverse the fetch()es to try the cache first
(since it is more likely anyway). Certainly we are running on caches
alone, no? If my suspicion is correct, the false-hits for the data
store will be removed, and the store-hit-rate may tank to 0.000%.
Arguably it would not promote the data in the store, but presently it
doesn't promote it in the cache.
--
Robert Hailey
<Inserted here: freenet-datastore-location-stats.diff>
Index: src/freenet/node/NodeStats.java
===================================================================
--- src/freenet/node/NodeStats.java (revision 16462)
+++ src/freenet/node/NodeStats.java (working copy)
@@ -22,6 +22,7 @@
import freenet.support.api.IntCallback;
import freenet.support.api.LongCallback;
import freenet.support.math.RunningAverage;
+import freenet.support.math.SimpleRunningAverage;
import freenet.support.math.TimeDecayingRunningAverage;
import freenet.support.math.TrivialRunningAverage;
@@ -144,8 +145,17 @@
// various metrics
public final RunningAverage routingMissDistance;
public final RunningAverage backedOffPercent;
+ public final RunningAverage avgCacheLocation;
+ public final RunningAverage avgStoreLocation;
+ public final RunningAverage avgCacheSuccess;
+ public final RunningAverage avgStoreSuccess;
+ // FIXME: does furthest{Store,Cache}Success need to be synchronized?
+ public double furthestCacheSuccess=0.0;
+ public double furthestStoreSuccess=0.0;
protected final Persister persister;
+ protected final RunningAverage avgRequestLocation;
+
// ThreadCounting stuffs
public final ThreadGroup rootThreadGroup;
private int threadLimit;
@@ -176,6 +186,14 @@
this.hardRandom = node.random;
this.routingMissDistance = new TimeDecayingRunningAverage(0.0,
180000, 0.0, 1.0, node);
this.backedOffPercent = new TimeDecayingRunningAverage(0.0,
180000, 0.0, 1.0, node);
+ // FIXME PLEASE remove (int) casts
+ double nodeLoc=node.lm.getLocation();
+ this.avgCacheLocation=new
SimpleRunningAverage((int)node.maxCacheKeys, nodeLoc);
+ this.avgStoreLocation=new
SimpleRunningAverage((int)node.maxStoreKeys, nodeLoc);
+ // FIXME average for success-location may not need to be so
large
as the store.
+ this.avgCacheSuccess=new SimpleRunningAverage(10000, nodeLoc);
+ this.avgStoreSuccess=new SimpleRunningAverage(10000, nodeLoc);
+ this.avgRequestLocation=new SimpleRunningAverage(10000,
nodeLoc);
preemptiveRejectReasons = new StringCounter();
localPreemptiveRejectReasons = new StringCounter();
pInstantRejectIncoming = new TimeDecayingRunningAverage(0,
60000,
0.0, 1.0, node);
@@ -186,6 +204,18 @@
new TimeDecayingRunningAverage(1, 10*60*1000 /* should
be
significantly longer than a typical transfer */, 0, Long.MAX_VALUE,
node);
nodePinger = new NodePinger(node);
+ // FIXME: data-store/cache averages need to be persisted to be
valuable (or scanned at every launch).
+ /*
+ if (node.isAdvancedModeEnabled()) {
+ //Uggghh....
+ System.err.println("Scanning datastore/cache for
location values");
+ chkDatastore.kludgeScan(avgStoreLocation);
+ sskDatastore.kludgeScan(avgStoreLocation);
+ chkDatacache.kludgeScan(avgCacheLocation);
+ sskDatacache.kludgeScan(avgCacheLocation);
+ }
+ */
+
previous_input_stat = 0;
previous_output_stat = 0;
previous_io_stat_time = 1;
Index: src/freenet/node/Node.java
===================================================================
--- src/freenet/node/Node.java (revision 16462)
+++ src/freenet/node/Node.java (working copy)
@@ -247,8 +247,8 @@
/** The maximum number of keys stored in each of the datastores,
cache and store combined. */
private long maxTotalKeys;
- private long maxCacheKeys;
- private long maxStoreKeys;
+ long maxCacheKeys;
+ long maxStoreKeys;
/** The maximum size of the datastore. Kept to avoid rounding
turning 5G into 5368698672 */
private long maxTotalDatastoreSize;
/** If true, store shrinks occur immediately even if they are over
10% of the store size. If false,
@@ -1767,11 +1767,23 @@
public SSKBlock fetch(NodeSSK key, boolean dontPromote) {
if(logMINOR) dumpStoreHits();
try {
+ double loc=key.toNormalizedDouble();
+ double dist=Location.distance(lm.getLocation(), loc);
+ nodeStats.avgRequestLocation.report(loc);
SSKBlock block = sskDatastore.fetch(key, dontPromote);
if(block != null) {
+ nodeStats.avgStoreSuccess.report(loc);
+ if (dist > nodeStats.furthestStoreSuccess)
+ nodeStats.furthestStoreSuccess=dist;
return block;
}
- return sskDatacache.fetch(key, dontPromote);
+ block=sskDatacache.fetch(key, dontPromote);
+ if (block != null) {
+ nodeStats.avgCacheSuccess.report(loc);
+ if (dist > nodeStats.furthestCacheSuccess)
+ nodeStats.furthestCacheSuccess=dist;
+ }
+ return block;
} catch (IOException e) {
Logger.error(this, "Cannot fetch data: "+e, e);
return null;
@@ -1781,9 +1793,23 @@
public CHKBlock fetch(NodeCHK key, boolean dontPromote) {
if(logMINOR) dumpStoreHits();
try {
+ double loc=key.toNormalizedDouble();
+ double dist=Location.distance(lm.getLocation(), loc);
+ nodeStats.avgRequestLocation.report(loc);
CHKBlock block = chkDatastore.fetch(key, dontPromote);
- if(block != null) return block;
- return chkDatacache.fetch(key, dontPromote);
+ if (block != null) {
+ nodeStats.avgStoreSuccess.report(loc);
+ if (dist > nodeStats.furthestStoreSuccess)
+ nodeStats.furthestStoreSuccess=dist;
+ return block;
+ }
+ block=chkDatacache.fetch(key, dontPromote);
+ if (block != null) {
+ nodeStats.avgCacheSuccess.report(loc);
+ if (dist > nodeStats.furthestCacheSuccess)
+ nodeStats.furthestCacheSuccess=dist;
+ }
+ return block;
} catch (IOException e) {
Logger.error(this, "Cannot fetch data: "+e, e);
return null;
@@ -1835,10 +1861,13 @@
private void store(CHKBlock block, boolean deep) {
try {
+ double loc=block.getKey().toNormalizedDouble();
if(deep) {
chkDatastore.put(block);
+ nodeStats.avgStoreLocation.report(loc);
}
chkDatacache.put(block);
+ nodeStats.avgCacheLocation.report(loc);
if(clientCore != null && clientCore.requestStarters !=
null)
clientCore.requestStarters.chkFetchScheduler.tripPendingKey(block);
} catch (IOException e) {
@@ -2498,6 +2527,10 @@
return usm;
}
+ public double getNodeLocation() {
+ return lm.getLocation();
+ }
+
public int getSwaps() {
return LocationManager.swaps;
}
Index: src/freenet/clients/http/StatisticsToadlet.java
===================================================================
--- src/freenet/clients/http/StatisticsToadlet.java (revision 16462)
+++ src/freenet/clients/http/StatisticsToadlet.java (working copy)
@@ -29,6 +29,7 @@
import freenet.support.SizeUtil;
import freenet.support.TimeUtil;
import freenet.support.api.HTTPRequest;
+import freenet.support.math.SimpleRunningAverage;
public class StatisticsToadlet extends Toadlet {
@@ -422,8 +423,47 @@
"\u00a0(" + ((storeHits*100) /
(storeAccesses)) + "%)");
storeSizeList.addChild("li",
- "Avg. access rate:\u00a0" +
thousendPoint.format(overallAccesses/
nodeUptimeSeconds) + "/sec");
+ "Avg. access rate:\u00a0" +
thousendPoint.format(cacheAccesses/
nodeUptimeSeconds) + "/sec, "+thousendPoint.format(storeAccesses/
nodeUptimeSeconds)+"/sec");
+ // location-based stats
+ boolean hasLoc=true;
+ double nodeLoc=0.0;
+ try {
+ nodeLoc=node.getNodeLocation();
+ } catch (Error e) {
+ //FIXME: PLEASE, how do we get the node location on the
stats page?
+ //Logger.error(this, "why?", e);
+ e.printStackTrace();
+ hasLoc=false;
+ }
+ double
avgCacheLocation=node.nodeStats.avgCacheLocation.currentValue();
+ double
avgStoreLocation=node.nodeStats.avgStoreLocation.currentValue();
+ double
avgCacheSuccess=node.nodeStats.avgCacheSuccess.currentValue();
+ double
avgStoreSuccess=node.nodeStats.avgStoreSuccess.currentValue();
+ double furthestCacheSuccess=node.nodeStats.furthestCacheSuccess;
+ double furthestStoreSuccess=node.nodeStats.furthestStoreSuccess;
+ double storeDist=Location.distance(nodeLoc, avgStoreLocation);
+ double cacheDist=Location.distance(nodeLoc, avgCacheLocation);
+
+ storeSizeList.addChild("li", "avgCacheLocation:\u00a0" +
thousendPoint.format(avgCacheLocation));
+ storeSizeList.addChild("li", "avgStoreLocation:\u00a0" +
thousendPoint.format(avgStoreLocation));
+
+ storeSizeList.addChild("li", "avgCacheSuccess:\u00a0" +
thousendPoint.format(avgCacheSuccess));
+ storeSizeList.addChild("li", "avgStoreSuccess:\u00a0" +
thousendPoint.format(avgStoreSuccess));
+
+ storeSizeList.addChild("li", "furthestCacheSuccess:\u00a0" +
thousendPoint.format(furthestCacheSuccess));
+ storeSizeList.addChild("li", "furthestStoreSuccess:\u00a0" +
thousendPoint.format(furthestStoreSuccess));
+
+ if (hasLoc) {
+ storeSizeList.addChild("li", "cacheDist:\u00a0" +
thousendPoint.format(cacheDist));
+ storeSizeList.addChild("li", "storeDist:\u00a0" +
thousendPoint.format(storeDist));
+ long
cacheLocationReports
=((SimpleRunningAverage)node.nodeStats.avgCacheLocation).countReports();
+ long
storeLocationReports
=((SimpleRunningAverage)node.nodeStats.avgStoreLocation).countReports();
+ double
cachePrimePercent=((1.0*cacheLocationReports)/cachedKeys);
+ double
storePrimePercent=((1.0*storeLocationReports)/storeKeys);
+ storeSizeList.addChild("li", "locStatsReliability:
\u00a0"+fix3p1pct.format(cachePrimePercent)+" /
"+fix3p1pct.format(storePrimePercent));
+ }
+
}
private void drawUnclaimedFIFOMessageCountsBox(HTMLNode
unclaimedFIFOMessageCountsInfobox) {
-------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20071211/ce2e454a/attachment.html>
