[freenet-dev] r19683 - trunk/freenet/src/freenet/store

[Matthew Toseland]

On Friday 02 May 2008 16:24, Robert Hailey wrote:
> 
> On May 2, 2008, at 9:59 AM, j16sdiz at freenetproject.org wrote:
> 
> > Author: j16sdiz
> > Date: 2008-05-02 14:59:50 + (Fri, 02 May 2008)
> > New Revision: 19683
> >
> > Modified:
> >   trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > Log:
> > oomhook: lower database cache size
> >
> >
> > Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > ===
> > --- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java  
> > 2008-05-02 14:59:29 UTC (rev 19682)
> > +++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java  
> > 2008-05-02 14:59:50 UTC (rev 19683)
> > @@ -25,6 +25,7 @@
> > import com.sleepycat.je.DatabaseNotFoundException;
> > import com.sleepycat.je.Environment;
> > import com.sleepycat.je.EnvironmentConfig;
> > +import com.sleepycat.je.EnvironmentMutableConfig;
> > import com.sleepycat.je.LockMode;
> > import com.sleepycat.je.OperationStatus;
> > import com.sleepycat.je.RunRecoveryException;
> > @@ -2247,6 +2248,11 @@
> > keysFC.force(true);
> > if (lruFC != null)
> > lruFC.force(true);
> > +   
> > +   EnvironmentMutableConfig dbmc = environment.getMutableConfig();
> > +   long cacheSize = (long) (dbmc.getCacheSize() * .9); // we have 
> > 6  
> > databases, 0.9^6 = 0.53
> > +   dbmc.setCacheSize(cacheSize);
> > +   Logger.normal(this, "low memory, set db cache = " + cacheSize);
> > }
> >
> 
> What is the significance of 0.9^6=0.53? It seems to me if you are  
> trying to half the memory usage, all the caches would have to be  
> reduced by 0.5; otherwise the comment could be made a bit more clear.

Be careful not to adjust it down to zero. Setting the max database memory to 0 
currently sets it unlimited, resulting in full GCs every <10 seconds which 
last for ~ 3 seconds...
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

[freenet-dev] r19683 - trunk/freenet/src/freenet/store

[Matthew Toseland]

On Friday 02 May 2008 16:28, Daniel Cheng wrote:
> On Fri, May 2, 2008 at 11:24 PM, Robert Hailey
>  wrote:
> > On May 2, 2008, at 9:59 AM, j16sdiz at freenetproject.org wrote:
> >
> > Author: j16sdiz
> > Date: 2008-05-02 14:59:50 + (Fri, 02 May 2008)
> > New Revision: 19683
> >
> > Modified:
> >trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > Log:
> > oomhook: lower database cache size
> >
> >
> > Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > ===
> > --- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java 2008-05-02
> > 14:59:29 UTC (rev 19682)
> > +++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java 2008-05-02
> > 14:59:50 UTC (rev 19683)
> > @@ -25,6 +25,7 @@
> >  import com.sleepycat.je.DatabaseNotFoundException;
> >  import com.sleepycat.je.Environment;
> >  import com.sleepycat.je.EnvironmentConfig;
> > +import com.sleepycat.je.EnvironmentMutableConfig;
> >  import com.sleepycat.je.LockMode;
> >  import com.sleepycat.je.OperationStatus;
> >  import com.sleepycat.je.RunRecoveryException;
> > @@ -2247,6 +2248,11 @@
> >  keysFC.force(true);
> >  if (lruFC != null)
> >  lruFC.force(true);
> > +
> > + EnvironmentMutableConfig dbmc = environment.getMutableConfig();
> > + long cacheSize = (long) (dbmc.getCacheSize() * .9); // we have 6
> > databases, 0.9^6 = 0.53
> > + dbmc.setCacheSize(cacheSize);
> > + Logger.normal(this, "low memory, set db cache = " + cacheSize);
> >  }
> >
> >
> > What is the significance of 0.9^6=0.53? It seems to me if you are trying 
to
> > half the memory usage, all the caches would have to be reduced by 0.5;
> > otherwise the comment could be made a bit more clear.
> >
> 
> Yes, I am trying to half the memory cache.
> This cache is global, setting to 90% 6 times give 53% of original size.

I'm not sure that makes sense. Don't you just want to set it to 90% overall?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

Re: [freenet-dev] r19683 - trunk/freenet/src/freenet/store

[Matthew Toseland]

On Friday 02 May 2008 16:24, Robert Hailey wrote:
> 
> On May 2, 2008, at 9:59 AM, [EMAIL PROTECTED] wrote:
> 
> > Author: j16sdiz
> > Date: 2008-05-02 14:59:50 + (Fri, 02 May 2008)
> > New Revision: 19683
> >
> > Modified:
> >   trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > Log:
> > oomhook: lower database cache size
> >
> >
> > Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > ===
> > --- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java  
> > 2008-05-02 14:59:29 UTC (rev 19682)
> > +++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java  
> > 2008-05-02 14:59:50 UTC (rev 19683)
> > @@ -25,6 +25,7 @@
> > import com.sleepycat.je.DatabaseNotFoundException;
> > import com.sleepycat.je.Environment;
> > import com.sleepycat.je.EnvironmentConfig;
> > +import com.sleepycat.je.EnvironmentMutableConfig;
> > import com.sleepycat.je.LockMode;
> > import com.sleepycat.je.OperationStatus;
> > import com.sleepycat.je.RunRecoveryException;
> > @@ -2247,6 +2248,11 @@
> > keysFC.force(true);
> > if (lruFC != null)
> > lruFC.force(true);
> > +   
> > +   EnvironmentMutableConfig dbmc = environment.getMutableConfig();
> > +   long cacheSize = (long) (dbmc.getCacheSize() * .9); // we have 
> > 6  
> > databases, 0.9^6 = 0.53
> > +   dbmc.setCacheSize(cacheSize);
> > +   Logger.normal(this, "low memory, set db cache = " + cacheSize);
> > }
> >
> 
> What is the significance of 0.9^6=0.53? It seems to me if you are  
> trying to half the memory usage, all the caches would have to be  
> reduced by 0.5; otherwise the comment could be made a bit more clear.

Be careful not to adjust it down to zero. Setting the max database memory to 0 
currently sets it unlimited, resulting in full GCs every <10 seconds which 
last for ~ 3 seconds...


pgpziARJN9LaJ.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] r19683 - trunk/freenet/src/freenet/store

[Matthew Toseland]

On Friday 02 May 2008 16:28, Daniel Cheng wrote:
> On Fri, May 2, 2008 at 11:24 PM, Robert Hailey
> <[EMAIL PROTECTED]> wrote:
> > On May 2, 2008, at 9:59 AM, [EMAIL PROTECTED] wrote:
> >
> > Author: j16sdiz
> > Date: 2008-05-02 14:59:50 + (Fri, 02 May 2008)
> > New Revision: 19683
> >
> > Modified:
> >trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > Log:
> > oomhook: lower database cache size
> >
> >
> > Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > ===
> > --- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java 2008-05-02
> > 14:59:29 UTC (rev 19682)
> > +++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java 2008-05-02
> > 14:59:50 UTC (rev 19683)
> > @@ -25,6 +25,7 @@
> >  import com.sleepycat.je.DatabaseNotFoundException;
> >  import com.sleepycat.je.Environment;
> >  import com.sleepycat.je.EnvironmentConfig;
> > +import com.sleepycat.je.EnvironmentMutableConfig;
> >  import com.sleepycat.je.LockMode;
> >  import com.sleepycat.je.OperationStatus;
> >  import com.sleepycat.je.RunRecoveryException;
> > @@ -2247,6 +2248,11 @@
> >  keysFC.force(true);
> >  if (lruFC != null)
> >  lruFC.force(true);
> > +
> > + EnvironmentMutableConfig dbmc = environment.getMutableConfig();
> > + long cacheSize = (long) (dbmc.getCacheSize() * .9); // we have 6
> > databases, 0.9^6 = 0.53
> > + dbmc.setCacheSize(cacheSize);
> > + Logger.normal(this, "low memory, set db cache = " + cacheSize);
> >  }
> >
> >
> > What is the significance of 0.9^6=0.53? It seems to me if you are trying 
to
> > half the memory usage, all the caches would have to be reduced by 0.5;
> > otherwise the comment could be made a bit more clear.
> >
> 
> Yes, I am trying to half the memory cache.
> This cache is global, setting to 90% 6 times give 53% of original size.

I'm not sure that makes sense. Don't you just want to set it to 90% overall?


pgpUbmGzonjcY.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] r19683 - trunk/freenet/src/freenet/store

[Daniel Cheng]

On Fri, May 2, 2008 at 11:24 PM, Robert Hailey
 wrote:
> On May 2, 2008, at 9:59 AM, j16sdiz at freenetproject.org wrote:
>
> Author: j16sdiz
> Date: 2008-05-02 14:59:50 + (Fri, 02 May 2008)
> New Revision: 19683
>
> Modified:
>trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> Log:
> oomhook: lower database cache size
>
>
> Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> ===
> --- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java 2008-05-02
> 14:59:29 UTC (rev 19682)
> +++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java 2008-05-02
> 14:59:50 UTC (rev 19683)
> @@ -25,6 +25,7 @@
>  import com.sleepycat.je.DatabaseNotFoundException;
>  import com.sleepycat.je.Environment;
>  import com.sleepycat.je.EnvironmentConfig;
> +import com.sleepycat.je.EnvironmentMutableConfig;
>  import com.sleepycat.je.LockMode;
>  import com.sleepycat.je.OperationStatus;
>  import com.sleepycat.je.RunRecoveryException;
> @@ -2247,6 +2248,11 @@
>  keysFC.force(true);
>  if (lruFC != null)
>  lruFC.force(true);
> +
> + EnvironmentMutableConfig dbmc = environment.getMutableConfig();
> + long cacheSize = (long) (dbmc.getCacheSize() * .9); // we have 6
> databases, 0.9^6 = 0.53
> + dbmc.setCacheSize(cacheSize);
> + Logger.normal(this, "low memory, set db cache = " + cacheSize);
>  }
>
>
> What is the significance of 0.9^6=0.53? It seems to me if you are trying to
> half the memory usage, all the caches would have to be reduced by 0.5;
> otherwise the comment could be made a bit more clear.
>

Yes, I am trying to half the memory cache.
This cache is global, setting to 90% 6 times give 53% of original size.

> --
> Robert Hailey
>

[freenet-dev] r19683 - trunk/freenet/src/freenet/store

[Robert Hailey]

On May 2, 2008, at 9:59 AM, j16sdiz at freenetproject.org wrote:

> Author: j16sdiz
> Date: 2008-05-02 14:59:50 + (Fri, 02 May 2008)
> New Revision: 19683
>
> Modified:
>   trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> Log:
> oomhook: lower database cache size
>
>
> Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> ===
> --- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> 2008-05-02 14:59:29 UTC (rev 19682)
> +++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> 2008-05-02 14:59:50 UTC (rev 19683)
> @@ -25,6 +25,7 @@
> import com.sleepycat.je.DatabaseNotFoundException;
> import com.sleepycat.je.Environment;
> import com.sleepycat.je.EnvironmentConfig;
> +import com.sleepycat.je.EnvironmentMutableConfig;
> import com.sleepycat.je.LockMode;
> import com.sleepycat.je.OperationStatus;
> import com.sleepycat.je.RunRecoveryException;
> @@ -2247,6 +2248,11 @@
>   keysFC.force(true);
>   if (lruFC != null)
>   lruFC.force(true);
> + 
> + EnvironmentMutableConfig dbmc = environment.getMutableConfig();
> + long cacheSize = (long) (dbmc.getCacheSize() * .9); // we have 
> 6  
> databases, 0.9^6 = 0.53
> + dbmc.setCacheSize(cacheSize);
> + Logger.normal(this, "low memory, set db cache = " + cacheSize);
>   }
>

What is the significance of 0.9^6=0.53? It seems to me if you are  
trying to half the memory usage, all the caches would have to be  
reduced by 0.5; otherwise the comment could be made a bit more clear.

--
Robert Hailey

-- next part --
An HTML attachment was scrubbed...
URL: 

Re: [freenet-dev] r19683 - trunk/freenet/src/freenet/store

[Daniel Cheng]

On Fri, May 2, 2008 at 11:24 PM, Robert Hailey
<[EMAIL PROTECTED]> wrote:
> On May 2, 2008, at 9:59 AM, [EMAIL PROTECTED] wrote:
>
> Author: j16sdiz
> Date: 2008-05-02 14:59:50 + (Fri, 02 May 2008)
> New Revision: 19683
>
> Modified:
>trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> Log:
> oomhook: lower database cache size
>
>
> Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> ===
> --- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java 2008-05-02
> 14:59:29 UTC (rev 19682)
> +++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java 2008-05-02
> 14:59:50 UTC (rev 19683)
> @@ -25,6 +25,7 @@
>  import com.sleepycat.je.DatabaseNotFoundException;
>  import com.sleepycat.je.Environment;
>  import com.sleepycat.je.EnvironmentConfig;
> +import com.sleepycat.je.EnvironmentMutableConfig;
>  import com.sleepycat.je.LockMode;
>  import com.sleepycat.je.OperationStatus;
>  import com.sleepycat.je.RunRecoveryException;
> @@ -2247,6 +2248,11 @@
>  keysFC.force(true);
>  if (lruFC != null)
>  lruFC.force(true);
> +
> + EnvironmentMutableConfig dbmc = environment.getMutableConfig();
> + long cacheSize = (long) (dbmc.getCacheSize() * .9); // we have 6
> databases, 0.9^6 = 0.53
> + dbmc.setCacheSize(cacheSize);
> + Logger.normal(this, "low memory, set db cache = " + cacheSize);
>  }
>
>
> What is the significance of 0.9^6=0.53? It seems to me if you are trying to
> half the memory usage, all the caches would have to be reduced by 0.5;
> otherwise the comment could be made a bit more clear.
>

Yes, I am trying to half the memory cache.
This cache is global, setting to 90% 6 times give 53% of original size.

> --
> Robert Hailey
>
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] r19683 - trunk/freenet/src/freenet/store

[Robert Hailey]

On May 2, 2008, at 9:59 AM, [EMAIL PROTECTED] wrote:


Author: j16sdiz
Date: 2008-05-02 14:59:50 + (Fri, 02 May 2008)
New Revision: 19683

Modified:
  trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
Log:
oomhook: lower database cache size


Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
===
--- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java	 
2008-05-02 14:59:29 UTC (rev 19682)
+++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java	 
2008-05-02 14:59:50 UTC (rev 19683)

@@ -25,6 +25,7 @@
import com.sleepycat.je.DatabaseNotFoundException;
import com.sleepycat.je.Environment;
import com.sleepycat.je.EnvironmentConfig;
+import com.sleepycat.je.EnvironmentMutableConfig;
import com.sleepycat.je.LockMode;
import com.sleepycat.je.OperationStatus;
import com.sleepycat.je.RunRecoveryException;
@@ -2247,6 +2248,11 @@
keysFC.force(true);
if (lruFC != null)
lruFC.force(true);
+   
+   EnvironmentMutableConfig dbmc = environment.getMutableConfig();
+		long cacheSize = (long) (dbmc.getCacheSize() * .9); // we have 6  
databases, 0.9^6 = 0.53

+   dbmc.setCacheSize(cacheSize);
+   Logger.normal(this, "low memory, set db cache = " + cacheSize);
}



What is the significance of 0.9^6=0.53? It seems to me if you are  
trying to half the memory usage, all the caches would have to be  
reduced by 0.5; otherwise the comment could be made a bit more clear.


--
Robert Hailey

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl