date:20080517

GCC 4.3 shipped in March, including the new ECJ frontend. It has full support 
for all the new 1.5 language features. IMHO this means that there is no 
longer any reason to stick to java 1.4.

Comments?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/a7b25690/attachment.pgp>

Re: [freenet-dev] New database for Freenet: db4o

* Matthew Toseland <[EMAIL PROTECTED]> [2008-05-17 19:00:13]:

> On Saturday 17 May 2008 00:29, Matthew Toseland wrote:
> > Ian and I have eventually come to the conclusion that we should include 
> db4o, 
> > and use it for our various persistence needs. I eventually reached the 
> > conclusion that while we can do most of what we need to do with simple 
> > flatfile databases, there are big chunks that will require a real database 
> of 
> > some kind (even if it's only a persistent hash table). db4o has various 
> > advantages:
> > - Robust in real-world use. See for example this testimonial from a company 
> > who used it on cell phones:
> > http://www.db4o.com/about/customers/success/mandalait.aspx
> > BDBJE has not met our expectations in this regard. It seems very sensitive 
> to 
> > unusual situations - in particular, it will spontaneously corrupt and lose 
> > all data on running out of disk space.
> > - True object database: no SQL, simple and powerful queries, etc.
> > - Transparent or manual activation of objects from storage.
> > - 800K jar, so not big enough to be a problem.
> > - Mature and actively maintained.
> > - Allows for future expansion (e.g. passive requests will need to store a 
> fair 
> > amount of persistent data).
> > - Much more flexible than the hand-coded solution I was thinking of. We can 
> > persistent the entire queue (not just the splitfiles), if it's useful to do 
> > that.
> > - Transactions (although this requires some juggling of in-memory objects 
> > on 
> > rollback).
> > 
> > Tasks:
> > - Add db4o to freenet-ext.jar.
> > - Think about using it for the datastore. We don't want to have two 
> databases! 
> > Sdiz's new datastore may be the One True Store, or it may not be. If it's 
> > not, we don't want to keep BDBJE: we could build a db4o-based store, with 
> > or 
> > without LRU replacement. It would have the advantage of filling up more 
> > quickly than sdiz's store. It should require reconstructing less frequently 
> > than BDBJE!
> > - Migrate the client layer, including splitfiles, pendingKeys, and so on, 
> > to 
> > be persisted via db4o. Of course there will be latency here when objects 
> > are 
> > not cached, so we will need to cache a few request choices in advance for 
> > each RequestStarter. And we will need to devise some way to deal with 
> > requests that don't want to be persisted - presumably we'd keep them in RAM.
> > 
> It turns out that db4o does indeed unrecoverably self-corrupt when it runs 
> out 
> of disk space. (Thanks nextgens for getting me to test this!)
> 
> http://amphibian.dyndns.org/bdb4o-test.log
> 

muhahahaha.

Last time I checked the bdb database was recoverable... Okay
it lost some^wmost of the data in the process but at least it did
attempt to recover!

> We will therefore have to keep a fallback. IMHO for the client layer the 
> fallback should be downloads.dat.gz. We are careful not to lose that when we 
> run out of disk space, and it should only contain what is needed to restart 
> requests from the beginning (in practice a lot will come from the store).
> 

...

While we are at it, what's wrong with bdb-je's persistence framework
again ?
http://www.oracle.com/database/berkeley-db/je/index.html

> I apologise if the above was presented as a fait accompli, any input on 
> databases would be appreciated. On Friday, me and Ian spent a long time 
> debating the issue, first and foremost of whether we should even have a 
> database; I was initially in favour of not having one at all, or using jdbm's 
> persistent hashtable class (HTree).
> 
> Personally I think if we have a database it should be a native object 
> database 
> i.e. either Perst or db4o. It also should be robust, low overhead, mature, 
> open source etc. I will start implementing the new client layer with db4o 
> soon, unless convinced to use something else in the meantime. But it seems 
> that with BDBJE (which isn't a native object database), you can lose the 
> database even by an unclean shutdown... can anyone confirm this from 
> experience? Or is it only out of disk space and memory corruption that causes 
> this?

I'm still not convinced that we need a database... as our requirements
are completely different from their typical use-cases... but well, your
immediate concern is to store persistent requests to disk, right? What
about using Hibernate or javax.persistence (from EE) to do that ?


signature.asc
Description: Digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Combating bloat (was: Re: Post 0.7 idea: off-grid darknet!

* Ian Clarke <[EMAIL PROTECTED]> [2008-05-17 13:35:40]:

> On Sat, May 17, 2008 at 6:10 AM, Matthew Toseland
> <[EMAIL PROTECTED]> wrote:
> >> Exactly, which is why Thaw, Freemail, etc are the apps that will
> >> motivate users to use Freenet.  Only developers download the JRE, most
> >> users get it bundled with Java apps.  The same will be true of
> >> Freenet, its a platform, most end-users don't want platforms on their
> >> own.  The solution is *not* to bundle, that is just pretending that
> >> Freenet is more than it is.
> >
> > We have a lot of traffic from wikipedia. We have a lot of traffic from
> > slashdot. For a user to even understand what Thaw is he must first 
> > understand
> > what Freenet is. Thaw, Freemail, FMS and jSite, don't have any sort of web
> > presence right now.
> 
> So they should get a web presence, we can't reinvent sourceforge, and
> we can't reinvent apt-get, we don't have the resources.
> 

Agreed

> > Freenet is not the same as Java. It's a bad metaphor. Maybe it would be a
> > better metaphor if any major freenet client had a web presence and
> > significant hits of its own, but none of them do. AND WE CAN'T WAIT FOR THEM
> > TO GET ONE
> 
> Why not?  It would be a 30 minute job for those apps to set up with Google 
> Code.
> 
> >, for much the same reason that we couldn't wait for FMS to release
> > 0.7.0. That means we have to do what we can for *our users*, which means
> > making it as easy as possible to get these client applications.
> 
> You must think our users are morons if the only way they can use an
> app is if we bundle it.  FMS isn't bundled, and it seems to have no
> shortage of users.
> 
> This "we've got to bundle everything" is a classic feature creep
> attitude.  If you think being user friendly means installing a bunch
> of software on someone's computer without them asking for it then you
> have a bizarre notion of user friendliness.
> 
> We aren't Google Code, we aren't apt-get, and we aren't Sourceforge.
> Trying to be those things will be a massive waste of resources.
> 

On the other hand, hosting freenet-related projects doesn't involve too
much overhead as far as emu's administration is concerned... And it
allows us to cross-reference bugs in between applications and the node,
which is very handy.

> And of course there is also the issue that we would be installing
> software on people's machines which we didn't write, and which for all
> we know could contain well hidden code to delete their hard disks on
> July 4th just for a laugh.   If we install this software, WE ARE
> RESPONSIBLE FOR WHAT IS DOES.  We don't have the resources to audit
> this code, and we can't install anonymously written code on people's
> computers without an audit.
> 

Agreed, that's a big concern... and reviewing all the 3rd party code we
bundle is unrealistic.


signature.asc
Description: Digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] [freenet-cvs] r19930 - trunk/freenet/test/freenet/crypt/ciphers

On Thu, May 15, 2008 at 10:39 PM, Matthew Toseland
 wrote:
> What's the problem with the 192-bit test?

see the discussion on r19912,
our Rijndael implementation does not follow FIPS standard. That test
would just fail.

>
> On Wednesday 14 May 2008 16:35, j16sdiz at freenetproject.org wrote:
>> Author: j16sdiz
>> Date: 2008-05-14 15:35:44 + (Wed, 14 May 2008)
>> New Revision: 19930
>>
>> Modified:
>>trunk/freenet/test/freenet/crypt/ciphers/RijndaelTest.java
>> Log:
>> Include (part of) FIPS standard test

Re: [freenet-dev] CPUID patch from FMS

* Matthew Toseland <[EMAIL PROTECTED]> [2008-05-17 14:44:27]:

> On Saturday 17 May 2008 13:58, Florent Daignière wrote:
> > * Florent Daignière <[EMAIL PROTECTED]> [2008-05-17 14:03:10]:
> > 
> > > * Matthew Toseland <[EMAIL PROTECTED]> [2008-05-17 12:29:09]:
> > > 
> > > > As nextgens is worried about anonymous contributions, I thought he 
> > > > might 
> want 
> > > > to review this before I apply it.
> > > > 
> > > 
> > > There is a related bug in the bug tracker... It's not the only bits that
> > > need updating :/
> > > 
> > > See #74
> > 
> > http://download.intel.com/design/processor/applnots/24161832.pdf
> > 
> http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/25481.pdf
> > 
> > Here are the CPUID codes if you want to update them.
> > 
> So should we or shouldn't we apply yongjhen's patch?

I think that applying that patch instead of updating the whole set of codes
is not the way to go...

If we were really after performances we would recompile the native
libraries with the new libgmp... which has performances boost on that
kind of (recent) processors.


signature.asc
Description: Digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] New database for Freenet: db4o

On Sat, May 17, 2008 at 7:06 PM, Matthew Toseland
 wrote:
> On Saturday 17 May 2008 06:24, Ian Clarke wrote:
>> On Fri, May 16, 2008 at 11:40 PM, Daniel Cheng
>>  wrote:
>> > On Sat, May 17, 2008 at 7:29 AM, Matthew Toseland
>> >  wrote:
>> >> Ian and I have eventually come to the conclusion that we should include
> db4o,
>>
>> Yay.
>>
>> >> Of course there will be latency here when objects are
>> >> not cached, so we will need to cache a few request choices in advance for
>> >> each RequestStarter. And we will need to devise some way to deal with
>> >> requests that don't want to be persisted - presumably we'd keep them in
> RAM.
>> >
>> > Please don't.
>> > According to what I have read, db4o should be good enough to use directly:
>>
>> I agree with Daniel, DIY may be an admirable quality when it comes to
>> home and car repair, but not with software.
>>
>> One of the benefits of using third-party stuff like db4o is that we
>> can outsource problems to people far more focussed on the solutions to
>> those problems than we can afford to be.
>>
>> If we spot a problem, we should fix it, but let's not fall into the
>> "premature optomization" trap.
>
> Which part of my proposal are you both criticising?
>
> If it's the "we should cache a few request choices in advance" bit then I
> stand by that. Any database query (assuming the working set is large) will
> involve many dependant disk accesses.

Did you read my original post? There were three features in db4o to
offload that a bit... CachedIoAdapter, prefetching, weakreferences ..
see my previous post for details on that.

> We have to send a request roughly every
> 800ms on my node (for SSKs). With slow commodity drives, often with other
> disk load going on, and with fairly complex database accesses involving
> multiple tables and therefore *many* mostly dependant seeks, we are not going
> to reliably meet that deadline no matter how good the database is. I will add
> a statistic for this, but my system is massively overpowered for this sort of
> effect, so we will need to test it on volunteers' slow systems.
>
> If it's the "requests that don't want to be persisted", what would you do with
> fproxy requests and other non-persistent requests? Store them to disk anyway?
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>

Re: [freenet-dev] CPUID patch from FMS

* Ian Clarke <[EMAIL PROTECTED]> [2008-05-17 20:43:17]:

> On Sat, May 17, 2008 at 12:53 PM, Matthew Toseland
> <[EMAIL PROTECTED]> wrote:
> > On Saturday 17 May 2008 16:35, Ian Clarke wrote:
> >> On Sat, May 17, 2008 at 6:29 AM, Matthew Toseland
> >> <[EMAIL PROTECTED]> wrote:
> >> > As nextgens is worried about anonymous contributions, I thought he might
> > want
> >> > to review this before I apply it.
> >>
> >> You know, I wish you guys would at least try using that Smartbear
> >> software, this is *exactly* what it is designed for and makes
> >> code-review much easier.
> >
> > I personally review all commits.
> 
> Who reviews your commits?  Perhaps if it were easier, someone would.
> 

Its a matter of time and will more than a tool-related problem afaic :/


signature.asc
Description: Digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] CPUID patch from FMS

On Sat, May 17, 2008 at 12:53 PM, Matthew Toseland
 wrote:
> On Saturday 17 May 2008 16:35, Ian Clarke wrote:
>> On Sat, May 17, 2008 at 6:29 AM, Matthew Toseland
>>  wrote:
>> > As nextgens is worried about anonymous contributions, I thought he might
> want
>> > to review this before I apply it.
>>
>> You know, I wish you guys would at least try using that Smartbear
>> software, this is *exactly* what it is designed for and makes
>> code-review much easier.
>
> I personally review all commits.

Who reviews your commits?  Perhaps if it were easier, someone would.

Ian.

-- 
Email: ian at uprizer.com
Cell: +1 512 422 3588
Skype: sanity

Re: [freenet-dev] Moving to java 1.5

2008-05-17 Thread Evan Daniel

On Sat, May 17, 2008 at 10:08 PM, Juiceman <[EMAIL PROTECTED]> wrote:
> On Sat, May 17, 2008 at 4:58 PM, Matthew Toseland
> <[EMAIL PROTECTED]> wrote:
>> GCC 4.3 shipped in March, including the new ECJ frontend. It has full support
>> for all the new 1.5 language features. IMHO this means that there is no
>> longer any reason to stick to java 1.4.
>>
>
> Are the opensource jvm's up to 1.5?  If so, I say go for it.  :)

IIRC, there are no JVM changes, only compiler changes.

Evan Daniel
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Moving to java 1.5

2008-05-17 Thread Juiceman

On Sat, May 17, 2008 at 4:58 PM, Matthew Toseland
<[EMAIL PROTECTED]> wrote:
> GCC 4.3 shipped in March, including the new ECJ frontend. It has full support
> for all the new 1.5 language features. IMHO this means that there is no
> longer any reason to stick to java 1.4.
>

Are the opensource jvm's up to 1.5?  If so, I say go for it.  :)


-- 
I may disagree with what you have to say, but I shall defend, to the
death, your right to say it. - Voltaire
Those who would give up Liberty, to purchase temporary Safety, deserve
neither Liberty nor Safety. - Ben Franklin
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] New database for Freenet: db4o

On Saturday 17 May 2008 00:29, Matthew Toseland wrote:
> Ian and I have eventually come to the conclusion that we should include 
db4o, 
> and use it for our various persistence needs. I eventually reached the 
> conclusion that while we can do most of what we need to do with simple 
> flatfile databases, there are big chunks that will require a real database 
of 
> some kind (even if it's only a persistent hash table). db4o has various 
> advantages:
> - Robust in real-world use. See for example this testimonial from a company 
> who used it on cell phones:
> http://www.db4o.com/about/customers/success/mandalait.aspx
> BDBJE has not met our expectations in this regard. It seems very sensitive 
to 
> unusual situations - in particular, it will spontaneously corrupt and lose 
> all data on running out of disk space.
> - True object database: no SQL, simple and powerful queries, etc.
> - Transparent or manual activation of objects from storage.
> - 800K jar, so not big enough to be a problem.
> - Mature and actively maintained.
> - Allows for future expansion (e.g. passive requests will need to store a 
fair 
> amount of persistent data).
> - Much more flexible than the hand-coded solution I was thinking of. We can 
> persistent the entire queue (not just the splitfiles), if it's useful to do 
> that.
> - Transactions (although this requires some juggling of in-memory objects on 
> rollback).
> 
> Tasks:
> - Add db4o to freenet-ext.jar.
> - Think about using it for the datastore. We don't want to have two 
databases! 
> Sdiz's new datastore may be the One True Store, or it may not be. If it's 
> not, we don't want to keep BDBJE: we could build a db4o-based store, with or 
> without LRU replacement. It would have the advantage of filling up more 
> quickly than sdiz's store. It should require reconstructing less frequently 
> than BDBJE!
> - Migrate the client layer, including splitfiles, pendingKeys, and so on, to 
> be persisted via db4o. Of course there will be latency here when objects are 
> not cached, so we will need to cache a few request choices in advance for 
> each RequestStarter. And we will need to devise some way to deal with 
> requests that don't want to be persisted - presumably we'd keep them in RAM.
> 
It turns out that db4o does indeed unrecoverably self-corrupt when it runs out 
of disk space. (Thanks nextgens for getting me to test this!)

http://amphibian.dyndns.org/bdb4o-test.log

We will therefore have to keep a fallback. IMHO for the client layer the 
fallback should be downloads.dat.gz. We are careful not to lose that when we 
run out of disk space, and it should only contain what is needed to restart 
requests from the beginning (in practice a lot will come from the store).

I apologise if the above was presented as a fait accompli, any input on 
databases would be appreciated. On Friday, me and Ian spent a long time 
debating the issue, first and foremost of whether we should even have a 
database; I was initially in favour of not having one at all, or using jdbm's 
persistent hashtable class (HTree).

Personally I think if we have a database it should be a native object database 
i.e. either Perst or db4o. It also should be robust, low overhead, mature, 
open source etc. I will start implementing the new client layer with db4o 
soon, unless convinced to use something else in the meantime. But it seems 
that with BDBJE (which isn't a native object database), you can lose the 
database even by an unclean shutdown... can anyone confirm this from 
experience? Or is it only out of disk space and memory corruption that causes 
this?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/4ebfc71d/attachment.pgp>

[freenet-dev] CPUID patch from FMS

On Saturday 17 May 2008 16:35, Ian Clarke wrote:
> On Sat, May 17, 2008 at 6:29 AM, Matthew Toseland
>  wrote:
> > As nextgens is worried about anonymous contributions, I thought he might 
want
> > to review this before I apply it.
> 
> You know, I wish you guys would at least try using that Smartbear
> software, this is *exactly* what it is designed for and makes
> code-review much easier.

I personally review all commits. Nextgens used to but doesn't have time any 
more... if a specific commit worries me I will forward it to him.
> 
> Ian.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/32e4addb/attachment.pgp>

Re: [freenet-dev] CPUID patch from FMS

On Sat, May 17, 2008 at 12:53 PM, Matthew Toseland
<[EMAIL PROTECTED]> wrote:
> On Saturday 17 May 2008 16:35, Ian Clarke wrote:
>> On Sat, May 17, 2008 at 6:29 AM, Matthew Toseland
>> <[EMAIL PROTECTED]> wrote:
>> > As nextgens is worried about anonymous contributions, I thought he might
> want
>> > to review this before I apply it.
>>
>> You know, I wish you guys would at least try using that Smartbear
>> software, this is *exactly* what it is designed for and makes
>> code-review much easier.
>
> I personally review all commits.

Who reviews your commits?  Perhaps if it were easier, someone would.

Ian.

-- 
Email: [EMAIL PROTECTED]
Cell: +1 512 422 3588
Skype: sanity
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] WoT plugin (db4o ?)

Matthew Toseland a ?crit :

> If you actually have code, you should commit it somewhere, whether it's 
> finished and working or not. This makes life easier for others who might want 
> to review it.

Well, I have some. But it is far too ugly to publish it ATM ;)
I'll work on it this week-end and, maybe, publish it tomorrow evening.

-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/2aa4f08d/attachment.pgp>

[freenet-dev] WoT plugin (db4o ?)

Julien Cornuwel a ?crit :

> The other problem is how to store the trust values. With derby, I simply
> have 2 tables, one with identites, the other with the trust values and
> ids if the truster and the trustee. How can I do that with an OO
> database ? The only idea I have is :
> Create a class Trust containing reference to the trusted identity and
> trust value, and store a list of Trust objects in every identites. I
> can't figure how bad it would be for the memory consumption.

Hey ! I just thought about an ugly-but-working solution : to recreate
the relation with an object...

If the Trust class contains trust value and ids of truster and trustee,
I'm able to load only what I want. And I can easily find out what
identites are trusting one particular identity, what's not possible with
a trustlist contained in the identity.

-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/3fddfa5c/attachment.pgp>

[freenet-dev] WoT plugin (db4o ?)

2008-05-17 Thread bbac...@googlemail.com

On Sat, May 17, 2008 at 2:25 PM, Julien Cornuwel  wrote:
> I've been reading DB4O's whitepaper (mentionned in Toad's post) and I'm
> wondering if I could use it for the WoT plugin. I could get rid of Derby
> and it would make the deployment simpler if db4o is shipped with freenet.
>
> Query possibilities are sufficient to my needs but I've got a few problems :
>
>
> With a RDBM, I can simply do the calculation of scores in the database
> (with a trigger and stored procedures).
> With db4o, persistence would be easier but I worry about memory
> consumption if I have to load every identities in memory each time a
> trust value is updated. An identity is about 200-250B. You see the
> problem if we have 1.000.000 identites !

I know perst, and I think db4o works similar. When you load persistent objects,
the engine takes care what objects fit in the memory cache. So you can hold
millions of objects but not all of them are always in memory. This can lead to
longer access times in reality.

>
> The other problem is how to store the trust values. With derby, I simply
> have 2 tables, one with identites, the other with the trust values and
> ids if the truster and the trustee. How can I do that with an OO
> database ? The only idea I have is :
> Create a class Trust containing reference to the trusted identity and
> trust value, and store a list of Trust objects in every identites. I
> can't figure how bad it would be for the memory consumption.
>

In perst I use different storages. Each one can have a different cache
size. This
way I can spread the load. But it is also possible to hold all objects
in one storage.

>
> Do you think it is feasible and worth the work ?
>
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>



-- 
__
GnuPG key: (0x48DBFA8A)
Keyserver: pgpkeys.pca.dfn.de
Fingerprint:
477D F057 1BD4 1AE7 8A54 8679 6690 E2EC 48DB FA8A
__

[freenet-dev] CPUID patch from FMS

* Florent Daigni?re  [2008-05-17 14:03:10]:

> * Matthew Toseland  [2008-05-17 12:29:09]:
> 
> > As nextgens is worried about anonymous contributions, I thought he might 
> > want 
> > to review this before I apply it.
> > 
> 
> There is a related bug in the bug tracker... It's not the only bits that
> need updating :/
> 
> See #74

http://download.intel.com/design/processor/applnots/24161832.pdf
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/25481.pdf

Here are the CPUID codes if you want to update them.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/f16912bc/attachment.pgp>

[freenet-dev] CPUID patch from FMS

On Saturday 17 May 2008 13:58, Florent Daigni?re wrote:
> * Florent Daigni?re  [2008-05-17 14:03:10]:
> 
> > * Matthew Toseland  [2008-05-17 12:29:09]:
> > 
> > > As nextgens is worried about anonymous contributions, I thought he might 
want 
> > > to review this before I apply it.
> > > 
> > 
> > There is a related bug in the bug tracker... It's not the only bits that
> > need updating :/
> > 
> > See #74
> 
> http://download.intel.com/design/processor/applnots/24161832.pdf
> 
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/25481.pdf
> 
> Here are the CPUID codes if you want to update them.
> 
So should we or shouldn't we apply yongjhen's patch?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/b121ce78/attachment.pgp>

[freenet-dev] WoT plugin (db4o ?)

On Saturday 17 May 2008 14:21, Julien Cornuwel wrote:
> Julien Cornuwel a ?crit :
> 
> > The other problem is how to store the trust values. With derby, I simply
> > have 2 tables, one with identites, the other with the trust values and
> > ids if the truster and the trustee. How can I do that with an OO
> > database ? The only idea I have is :
> > Create a class Trust containing reference to the trusted identity and
> > trust value, and store a list of Trust objects in every identites. I
> > can't figure how bad it would be for the memory consumption.
> 
> Hey ! I just thought about an ugly-but-working solution : to recreate
> the relation with an object...
> 
> If the Trust class contains trust value and ids of truster and trustee,
> I'm able to load only what I want. And I can easily find out what
> identites are trusting one particular identity, what's not possible with
> a trustlist contained in the identity.
> 
If you actually have code, you should commit it somewhere, whether it's 
finished and working or not. This makes life easier for others who might want 
to review it.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/ccb65b99/attachment.pgp>

[freenet-dev] New database for Freenet: db4o

On Saturday 17 May 2008 14:07, Daniel Cheng wrote:
> On Sat, May 17, 2008 at 7:06 PM, Matthew Toseland
>  wrote:
> > On Saturday 17 May 2008 06:24, Ian Clarke wrote:
> >> On Fri, May 16, 2008 at 11:40 PM, Daniel Cheng
> >>  wrote:
> >> > On Sat, May 17, 2008 at 7:29 AM, Matthew Toseland
> >> >  wrote:
> >> >> Ian and I have eventually come to the conclusion that we should 
include
> > db4o,
> >>
> >> Yay.
> >>
> >> >> Of course there will be latency here when objects are
> >> >> not cached, so we will need to cache a few request choices in advance 
for
> >> >> each RequestStarter. And we will need to devise some way to deal with
> >> >> requests that don't want to be persisted - presumably we'd keep them 
in
> > RAM.
> >> >
> >> > Please don't.
> >> > According to what I have read, db4o should be good enough to use 
directly:
> >>
> >> I agree with Daniel, DIY may be an admirable quality when it comes to
> >> home and car repair, but not with software.
> >>
> >> One of the benefits of using third-party stuff like db4o is that we
> >> can outsource problems to people far more focussed on the solutions to
> >> those problems than we can afford to be.
> >>
> >> If we spot a problem, we should fix it, but let's not fall into the
> >> "premature optomization" trap.
> >
> > Which part of my proposal are you both criticising?
> >
> > If it's the "we should cache a few request choices in advance" bit then I
> > stand by that. Any database query (assuming the working set is large) will
> > involve many dependant disk accesses.
> 
> Did you read my original post? There were three features in db4o to
> offload that a bit... CachedIoAdapter, prefetching, weakreferences ..
> see my previous post for details on that.

Strictly speaking this is prefetch we're talking about, not caching.
> 
> > We have to send a request roughly every
> > 800ms on my node (for SSKs). With slow commodity drives, often with other
> > disk load going on, and with fairly complex database accesses involving
> > multiple tables and therefore *many* mostly dependant seeks, we are not 
going
> > to reliably meet that deadline no matter how good the database is. I will 
add
> > a statistic for this, but my system is massively overpowered for this sort 
of
> > effect, so we will need to test it on volunteers' slow systems.
> >
> > If it's the "requests that don't want to be persisted", what would you do 
with
> > fproxy requests and other non-persistent requests? Store them to disk 
anyway?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/9a41bebe/attachment.pgp>

[freenet-dev] WoT plugin (db4o ?)

I've been reading DB4O's whitepaper (mentionned in Toad's post) and I'm
wondering if I could use it for the WoT plugin. I could get rid of Derby
and it would make the deployment simpler if db4o is shipped with freenet.

Query possibilities are sufficient to my needs but I've got a few problems :


With a RDBM, I can simply do the calculation of scores in the database
(with a trigger and stored procedures).
With db4o, persistence would be easier but I worry about memory
consumption if I have to load every identities in memory each time a
trust value is updated. An identity is about 200-250B. You see the
problem if we have 1.000.000 identites !

The other problem is how to store the trust values. With derby, I simply
have 2 tables, one with identites, the other with the trust values and
ids if the truster and the trustee. How can I do that with an OO
database ? The only idea I have is :
Create a class Trust containing reference to the trusted identity and
trust value, and store a list of Trust objects in every identites. I
can't figure how bad it would be for the memory consumption.


Do you think it is feasible and worth the work ?

-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/648269f7/attachment.pgp>

[freenet-dev] CPUID patch from FMS

* Matthew Toseland  [2008-05-17 12:29:09]:

> As nextgens is worried about anonymous contributions, I thought he might want 
> to review this before I apply it.
> 

There is a related bug in the bug tracker... It's not the only bits that
need updating :/

See #74

> CPUID patch
> From:
> yongjhen at okYcFnYIaKTYWj1K7EfA7LONeLu7Bwazdd4djTIe4eU
>   Date:
> Friday 16 May 2008 03:43:30
>   Groups:
> freenet
>   Followup-To:
> freenet
> Message was signed with unknown key 0xB3FFA5B9.
> The validity of the signature cannot be verified.
>   Hi all,
> 
> This patch makes Freenet recognize Intel Pentium M, Core, and Core 2 as
> Pentium 4 compatible, and load corresponding native BigInteger library.
> Tested on my laptop with Core CPU.
> 
> 
> --- freenet/src/freenet/support/CPUInformation/CPUID.java.orig
> 2008-04-23 15:24:19.0 +0800
> +++ freenet/src/freenet/support/CPUInformation/CPUID.java
> 2008-05-16 10:35:55.0 +0800
> @@ -280,7 +280,10 @@
> }
> public boolean IsPentium4Compatible()
> {
> -   return getCPUFamily() >= 15;
> +   int family = getCPUFamily();
> +   int model = getCPUModel();
> +   return (family >= 15 ||
> +   (family == 6 && (model == 9 || model
> >=13)));
> }
> public String getCPUModelString() throws
> UnknownCPUException {
> if(getCPUFamily() == 4){
> 
> 
> Cheers!
> 
> --
> FMS:  yongjhen at okYcFnYIaKTYWj1K7EfA7LONeLu7Bwazdd4djTIe4eU
> Freemail: yongjhen at alqualonde.freemail
> PGP fingerprint: 88B6 7F21 38E1 01E6 CF3E B1AD 31CC C1F2 B3FF A5B9
> 
> 
>   End of signed message



> ___
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/3e31b4b9/attachment.pgp>

[freenet-dev] Moving to java 1.5

GCC 4.3 shipped in March, including the new ECJ frontend. It has full support 
for all the new 1.5 language features. IMHO this means that there is no 
longer any reason to stick to java 1.4.

Comments?


pgpE1QU7ZKe6r.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Combating bloat (was: Re: Post 0.7 idea: off-grid darknet!

On Sat, May 17, 2008 at 6:10 AM, Matthew Toseland
 wrote:
>> Exactly, which is why Thaw, Freemail, etc are the apps that will
>> motivate users to use Freenet.  Only developers download the JRE, most
>> users get it bundled with Java apps.  The same will be true of
>> Freenet, its a platform, most end-users don't want platforms on their
>> own.  The solution is *not* to bundle, that is just pretending that
>> Freenet is more than it is.
>
> We have a lot of traffic from wikipedia. We have a lot of traffic from
> slashdot. For a user to even understand what Thaw is he must first understand
> what Freenet is. Thaw, Freemail, FMS and jSite, don't have any sort of web
> presence right now.

So they should get a web presence, we can't reinvent sourceforge, and
we can't reinvent apt-get, we don't have the resources.

> Freenet is not the same as Java. It's a bad metaphor. Maybe it would be a
> better metaphor if any major freenet client had a web presence and
> significant hits of its own, but none of them do. AND WE CAN'T WAIT FOR THEM
> TO GET ONE

Why not?  It would be a 30 minute job for those apps to set up with Google Code.

>, for much the same reason that we couldn't wait for FMS to release
> 0.7.0. That means we have to do what we can for *our users*, which means
> making it as easy as possible to get these client applications.

You must think our users are morons if the only way they can use an
app is if we bundle it.  FMS isn't bundled, and it seems to have no
shortage of users.

This "we've got to bundle everything" is a classic feature creep
attitude.  If you think being user friendly means installing a bunch
of software on someone's computer without them asking for it then you
have a bizarre notion of user friendliness.

We aren't Google Code, we aren't apt-get, and we aren't Sourceforge.
Trying to be those things will be a massive waste of resources.

And of course there is also the issue that we would be installing
software on people's machines which we didn't write, and which for all
we know could contain well hidden code to delete their hard disks on
July 4th just for a laugh.   If we install this software, WE ARE
RESPONSIBLE FOR WHAT IS DOES.  We don't have the resources to audit
this code, and we can't install anonymously written code on people's
computers without an audit.

Ian.

-- 
Email: ian at uprizer.com
Cell: +1 512 422 3588
Skype: sanity

[freenet-dev] New database for Freenet: db4o

On Sat, May 17, 2008 at 7:29 AM, Matthew Toseland
 wrote:
> Ian and I have eventually come to the conclusion that we should include db4o,
> and use it for our various persistence needs. I eventually reached the
> conclusion that while we can do most of what we need to do with simple
> flatfile databases, there are big chunks that will require a real database of
> some kind (even if it's only a persistent hash table). db4o has various
> advantages:
> - Robust in real-world use. See for example this testimonial from a company
> who used it on cell phones:
> http://www.db4o.com/about/customers/success/mandalait.aspx
> BDBJE has not met our expectations in this regard. It seems very sensitive to
> unusual situations - in particular, it will spontaneously corrupt and lose
> all data on running out of disk space.
> - True object database: no SQL, simple and powerful queries, etc.
> - Transparent or manual activation of objects from storage.
> - 800K jar, so not big enough to be a problem.
> - Mature and actively maintained.
> - Allows for future expansion (e.g. passive requests will need to store a fair
> amount of persistent data).
> - Much more flexible than the hand-coded solution I was thinking of. We can
> persistent the entire queue (not just the splitfiles), if it's useful to do
> that.
> - Transactions (although this requires some juggling of in-memory objects on
> rollback).

Look good to me (except it's website need registration)

> Tasks:
> - Add db4o to freenet-ext.jar.
> - Think about using it for the datastore. We don't want to have two databases!
> Sdiz's new datastore may be the One True Store, or it may not be. If it's
> not, we don't want to keep BDBJE: we could build a db4o-based store, with or
> without LRU replacement. It would have the advantage of filling up more
> quickly than sdiz's store. It should require reconstructing less frequently
> than BDBJE!


> - Migrate the client layer, including splitfiles, pendingKeys, and so on, to
> be persisted via db4o.

> Of course there will be latency here when objects are
> not cached, so we will need to cache a few request choices in advance for
> each RequestStarter. And we will need to devise some way to deal with
> requests that don't want to be persisted - presumably we'd keep them in RAM.

Please don't.

According to what I have read, db4o should be good enough to use directly:
  - Active objects are kept with a WeakReference, so as long as it is not GC'ed,
you don't have to read the disk  (there is an option to use
hard reference too)
  - Data are prefetched and activated in batch.. if the query is
well-written, we have
the items in memory already.
  - CachedIoAdapter provide a low level disk cache.

This maybe not as good as a custom cache scheme.. But I really hate
the idea of having yet-another-caching scheme for some marginal (or
even hypo-theoretical) performance benefit -- we have had too much of
them already.

Please, do that ONLY if it's supported by some benchmarks / cpu profiles.

--

[freenet-dev] CPUID patch from FMS

As nextgens is worried about anonymous contributions, I thought he might want 
to review this before I apply it.

CPUID patch
From:
yongjhen at okYcFnYIaKTYWj1K7EfA7LONeLu7Bwazdd4djTIe4eU
  Date:
Friday 16 May 2008 03:43:30
  Groups:
freenet
  Followup-To:
freenet
Message was signed with unknown key 0xB3FFA5B9.
The validity of the signature cannot be verified.
  Hi all,

This patch makes Freenet recognize Intel Pentium M, Core, and Core 2 as
Pentium 4 compatible, and load corresponding native BigInteger library.
Tested on my laptop with Core CPU.

--- freenet/src/freenet/support/CPUInformation/CPUID.java.orig
2008-04-23 15:24:19.0 +0800
+++ freenet/src/freenet/support/CPUInformation/CPUID.java
2008-05-16 10:35:55.0 +0800
@@ -280,7 +280,10 @@
}
public boolean IsPentium4Compatible()
{
-   return getCPUFamily() >= 15;
+   int family = getCPUFamily();
+   int model = getCPUModel();
+   return (family >= 15 ||
+   (family == 6 && (model == 9 || model
>=13)));
}
public String getCPUModelString() throws
UnknownCPUException {
if(getCPUFamily() == 4){

Cheers!

--
FMS:  yongjhen at okYcFnYIaKTYWj1K7EfA7LONeLu7Bwazdd4djTIe4eU
Freemail: yongjhen at alqualonde.freemail
PGP fingerprint: 88B6 7F21 38E1 01E6 CF3E B1AD 31CC C1F2 B3FF A5B9

  End of signed message
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/c95a9f6b/attachment.pgp>

[freenet-dev] Combating bloat (was: Re: Post 0.7 idea: off-grid darknet!

On Friday 16 May 2008 22:09, Ian Clarke wrote:
> On Fri, May 16, 2008 at 11:12 AM, Colin Davis  wrote:
> >> Why are you so obsessed with turning us into Sourceforge for Freenet
> >> apps?  If we are successful there could be hundreds of apps, there is
> >> no reason for us to host all of them - that is rediculous.  Let them
> >> use sourceforge, or google code, or set up their own website.
> 
> > For the same reason, I suspect, that Apple is hosting a page of Mac
> > Applications, and heavily-pushes third-party applications in it's retail
> > stores, and the same reason that developers around the world are excited
> > that the iPhone will have a built-in appstore.
> 
> You may not have noticed, but we have slightly fewer resources than Apple.
> 
> > Distribution matters. Users don't buy/install a product because of it's
> > inherent properties, they install it because it solves a need.
> > Freenet/Fproxy alone don't solve very many people's need, but Thaw,
> > Freemail, etc are services that people find useful.
> 
> Exactly, which is why Thaw, Freemail, etc are the apps that will
> motivate users to use Freenet.  Only developers download the JRE, most
> users get it bundled with Java apps.  The same will be true of
> Freenet, its a platform, most end-users don't want platforms on their
> own.  The solution is *not* to bundle, that is just pretending that
> Freenet is more than it is.

We have a lot of traffic from wikipedia. We have a lot of traffic from 
slashdot. For a user to even understand what Thaw is he must first understand 
what Freenet is. Thaw, Freemail, FMS and jSite, don't have any sort of web 
presence right now.

Freenet is not the same as Java. It's a bad metaphor. Maybe it would be a 
better metaphor if any major freenet client had a web presence and 
significant hits of its own, but none of them do. AND WE CAN'T WAIT FOR THEM 
TO GET ONE, for much the same reason that we couldn't wait for FMS to release 
0.7.0. That means we have to do what we can for *our users*, which means 
making it as easy as possible to get these client applications.
> 
> Ian.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/e13f48f1/attachment.pgp>

[freenet-dev] New database for Freenet: db4o

On Saturday 17 May 2008 06:24, Ian Clarke wrote:
> On Fri, May 16, 2008 at 11:40 PM, Daniel Cheng
>  wrote:
> > On Sat, May 17, 2008 at 7:29 AM, Matthew Toseland
> >  wrote:
> >> Ian and I have eventually come to the conclusion that we should include 
db4o,
> 
> Yay.
> 
> >> Of course there will be latency here when objects are
> >> not cached, so we will need to cache a few request choices in advance for
> >> each RequestStarter. And we will need to devise some way to deal with
> >> requests that don't want to be persisted - presumably we'd keep them in 
RAM.
> >
> > Please don't.
> > According to what I have read, db4o should be good enough to use directly:
> 
> I agree with Daniel, DIY may be an admirable quality when it comes to
> home and car repair, but not with software.
> 
> One of the benefits of using third-party stuff like db4o is that we
> can outsource problems to people far more focussed on the solutions to
> those problems than we can afford to be.
> 
> If we spot a problem, we should fix it, but let's not fall into the
> "premature optomization" trap.

Which part of my proposal are you both criticising?

If it's the "we should cache a few request choices in advance" bit then I 
stand by that. Any database query (assuming the working set is large) will 
involve many dependant disk accesses. We have to send a request roughly every 
800ms on my node (for SSKs). With slow commodity drives, often with other 
disk load going on, and with fairly complex database accesses involving 
multiple tables and therefore *many* mostly dependant seeks, we are not going 
to reliably meet that deadline no matter how good the database is. I will add 
a statistic for this, but my system is massively overpowered for this sort of 
effect, so we will need to test it on volunteers' slow systems.

If it's the "requests that don't want to be persisted", what would you do with 
fproxy requests and other non-persistent requests? Store them to disk anyway?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/1f709c54/attachment.pgp>

Re: [freenet-dev] Combating bloat (was: Re: Post 0.7 idea: off-grid darknet!

On Sat, May 17, 2008 at 6:10 AM, Matthew Toseland
<[EMAIL PROTECTED]> wrote:
>> Exactly, which is why Thaw, Freemail, etc are the apps that will
>> motivate users to use Freenet.  Only developers download the JRE, most
>> users get it bundled with Java apps.  The same will be true of
>> Freenet, its a platform, most end-users don't want platforms on their
>> own.  The solution is *not* to bundle, that is just pretending that
>> Freenet is more than it is.
>
> We have a lot of traffic from wikipedia. We have a lot of traffic from
> slashdot. For a user to even understand what Thaw is he must first understand
> what Freenet is. Thaw, Freemail, FMS and jSite, don't have any sort of web
> presence right now.

So they should get a web presence, we can't reinvent sourceforge, and
we can't reinvent apt-get, we don't have the resources.

> Freenet is not the same as Java. It's a bad metaphor. Maybe it would be a
> better metaphor if any major freenet client had a web presence and
> significant hits of its own, but none of them do. AND WE CAN'T WAIT FOR THEM
> TO GET ONE

Why not?  It would be a 30 minute job for those apps to set up with Google Code.

>, for much the same reason that we couldn't wait for FMS to release
> 0.7.0. That means we have to do what we can for *our users*, which means
> making it as easy as possible to get these client applications.

You must think our users are morons if the only way they can use an
app is if we bundle it.  FMS isn't bundled, and it seems to have no
shortage of users.

This "we've got to bundle everything" is a classic feature creep
attitude.  If you think being user friendly means installing a bunch
of software on someone's computer without them asking for it then you
have a bizarre notion of user friendliness.

We aren't Google Code, we aren't apt-get, and we aren't Sourceforge.
Trying to be those things will be a massive waste of resources.

And of course there is also the issue that we would be installing
software on people's machines which we didn't write, and which for all
we know could contain well hidden code to delete their hard disks on
July 4th just for a laugh.   If we install this software, WE ARE
RESPONSIBLE FOR WHAT IS DOES.  We don't have the resources to audit
this code, and we can't install anonymously written code on people's
computers without an audit.

Ian.

-- 
Email: [EMAIL PROTECTED]
Cell: +1 512 422 3588
Skype: sanity
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] New database for Freenet: db4o

On Saturday 17 May 2008 00:29, Matthew Toseland wrote:
> Ian and I have eventually come to the conclusion that we should include 
db4o, 
> and use it for our various persistence needs. I eventually reached the 
> conclusion that while we can do most of what we need to do with simple 
> flatfile databases, there are big chunks that will require a real database 
of 
> some kind (even if it's only a persistent hash table). db4o has various 
> advantages:
> - Robust in real-world use. See for example this testimonial from a company 
> who used it on cell phones:
> http://www.db4o.com/about/customers/success/mandalait.aspx
> BDBJE has not met our expectations in this regard. It seems very sensitive 
to 
> unusual situations - in particular, it will spontaneously corrupt and lose 
> all data on running out of disk space.
> - True object database: no SQL, simple and powerful queries, etc.
> - Transparent or manual activation of objects from storage.
> - 800K jar, so not big enough to be a problem.
> - Mature and actively maintained.
> - Allows for future expansion (e.g. passive requests will need to store a 
fair 
> amount of persistent data).
> - Much more flexible than the hand-coded solution I was thinking of. We can 
> persistent the entire queue (not just the splitfiles), if it's useful to do 
> that.
> - Transactions (although this requires some juggling of in-memory objects on 
> rollback).
> 
> Tasks:
> - Add db4o to freenet-ext.jar.
> - Think about using it for the datastore. We don't want to have two 
databases! 
> Sdiz's new datastore may be the One True Store, or it may not be. If it's 
> not, we don't want to keep BDBJE: we could build a db4o-based store, with or 
> without LRU replacement. It would have the advantage of filling up more 
> quickly than sdiz's store. It should require reconstructing less frequently 
> than BDBJE!
> - Migrate the client layer, including splitfiles, pendingKeys, and so on, to 
> be persisted via db4o. Of course there will be latency here when objects are 
> not cached, so we will need to cache a few request choices in advance for 
> each RequestStarter. And we will need to devise some way to deal with 
> requests that don't want to be persisted - presumably we'd keep them in RAM.
> 
It turns out that db4o does indeed unrecoverably self-corrupt when it runs out 
of disk space. (Thanks nextgens for getting me to test this!)

http://amphibian.dyndns.org/bdb4o-test.log

We will therefore have to keep a fallback. IMHO for the client layer the 
fallback should be downloads.dat.gz. We are careful not to lose that when we 
run out of disk space, and it should only contain what is needed to restart 
requests from the beginning (in practice a lot will come from the store).

I apologise if the above was presented as a fait accompli, any input on 
databases would be appreciated. On Friday, me and Ian spent a long time 
debating the issue, first and foremost of whether we should even have a 
database; I was initially in favour of not having one at all, or using jdbm's 
persistent hashtable class (HTree).

Personally I think if we have a database it should be a native object database 
i.e. either Perst or db4o. It also should be robust, low overhead, mature, 
open source etc. I will start implementing the new client layer with db4o 
soon, unless convinced to use something else in the meantime. But it seems 
that with BDBJE (which isn't a native object database), you can lose the 
database even by an unclean shutdown... can anyone confirm this from 
experience? Or is it only out of disk space and memory corruption that causes 
this?

pgpGocJq9CYWc.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CPUID patch from FMS

On Saturday 17 May 2008 16:35, Ian Clarke wrote:
> On Sat, May 17, 2008 at 6:29 AM, Matthew Toseland
> <[EMAIL PROTECTED]> wrote:
> > As nextgens is worried about anonymous contributions, I thought he might 
want
> > to review this before I apply it.
> 
> You know, I wish you guys would at least try using that Smartbear
> software, this is *exactly* what it is designed for and makes
> code-review much easier.

I personally review all commits. Nextgens used to but doesn't have time any 
more... if a specific commit worries me I will forward it to him.
> 
> Ian.


pgpBrXprC7cqe.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] CPUID patch from FMS

On Sat, May 17, 2008 at 6:29 AM, Matthew Toseland
 wrote:
> As nextgens is worried about anonymous contributions, I thought he might want
> to review this before I apply it.

You know, I wish you guys would at least try using that Smartbear
software, this is *exactly* what it is designed for and makes
code-review much easier.

Ian.

-- 
Email: ian at uprizer.com
Cell: +1 512 422 3588
Skype: sanity

[freenet-dev] GPG auto-signing key on emu

* Florent Daigni?re  [2008-05-15 14:56:56]:

> (2) emu's gpg autosigning key has expired and will be renewed when I get
> around to do it

The new key has been generated, you can find details about the key on
http://downloads.freenetproject.org/alpha/README

Ian, toad and me can revoke it if needed.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/b4f85f12/attachment.pgp>

Re: [freenet-dev] CPUID patch from FMS

On Sat, May 17, 2008 at 6:29 AM, Matthew Toseland
<[EMAIL PROTECTED]> wrote:
> As nextgens is worried about anonymous contributions, I thought he might want
> to review this before I apply it.

You know, I wish you guys would at least try using that Smartbear
software, this is *exactly* what it is designed for and makes
code-review much easier.

Ian.

-- 
Email: [EMAIL PROTECTED]
Cell: +1 512 422 3588
Skype: sanity
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] WoT plugin (db4o ?)

Matthew Toseland a écrit :

> If you actually have code, you should commit it somewhere, whether it's 
> finished and working or not. This makes life easier for others who might want 
> to review it.

Well, I have some. But it is far too ugly to publish it ATM ;)
I'll work on it this week-end and, maybe, publish it tomorrow evening.



signature.asc
Description: OpenPGP digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CPUID patch from FMS

On Saturday 17 May 2008 13:58, Florent Daignière wrote:
> * Florent Daignière <[EMAIL PROTECTED]> [2008-05-17 14:03:10]:
> 
> > * Matthew Toseland <[EMAIL PROTECTED]> [2008-05-17 12:29:09]:
> > 
> > > As nextgens is worried about anonymous contributions, I thought he might 
want 
> > > to review this before I apply it.
> > > 
> > 
> > There is a related bug in the bug tracker... It's not the only bits that
> > need updating :/
> > 
> > See #74
> 
> http://download.intel.com/design/processor/applnots/24161832.pdf
> 
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/25481.pdf
> 
> Here are the CPUID codes if you want to update them.
> 
So should we or shouldn't we apply yongjhen's patch?


pgpyCVWloqQTs.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] WoT plugin (db4o ?)

On Saturday 17 May 2008 14:21, Julien Cornuwel wrote:
> Julien Cornuwel a écrit :
> 
> > The other problem is how to store the trust values. With derby, I simply
> > have 2 tables, one with identites, the other with the trust values and
> > ids if the truster and the trustee. How can I do that with an OO
> > database ? The only idea I have is :
> > Create a class Trust containing reference to the trusted identity and
> > trust value, and store a list of Trust objects in every identites. I
> > can't figure how bad it would be for the memory consumption.
> 
> Hey ! I just thought about an ugly-but-working solution : to recreate
> the relation with an object...
> 
> If the Trust class contains trust value and ids of truster and trustee,
> I'm able to load only what I want. And I can easily find out what
> identites are trusting one particular identity, what's not possible with
> a trustlist contained in the identity.
> 
If you actually have code, you should commit it somewhere, whether it's 
finished and working or not. This makes life easier for others who might want 
to review it.


pgpRtD8ZA2lPi.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] New database for Freenet: db4o

On Saturday 17 May 2008 14:07, Daniel Cheng wrote:
> On Sat, May 17, 2008 at 7:06 PM, Matthew Toseland
> <[EMAIL PROTECTED]> wrote:
> > On Saturday 17 May 2008 06:24, Ian Clarke wrote:
> >> On Fri, May 16, 2008 at 11:40 PM, Daniel Cheng
> >> <[EMAIL PROTECTED]> wrote:
> >> > On Sat, May 17, 2008 at 7:29 AM, Matthew Toseland
> >> > <[EMAIL PROTECTED]> wrote:
> >> >> Ian and I have eventually come to the conclusion that we should 
include
> > db4o,
> >>
> >> Yay.
> >>
> >> >> Of course there will be latency here when objects are
> >> >> not cached, so we will need to cache a few request choices in advance 
for
> >> >> each RequestStarter. And we will need to devise some way to deal with
> >> >> requests that don't want to be persisted - presumably we'd keep them 
in
> > RAM.
> >> >
> >> > Please don't.
> >> > According to what I have read, db4o should be good enough to use 
directly:
> >>
> >> I agree with Daniel, DIY may be an admirable quality when it comes to
> >> home and car repair, but not with software.
> >>
> >> One of the benefits of using third-party stuff like db4o is that we
> >> can outsource problems to people far more focussed on the solutions to
> >> those problems than we can afford to be.
> >>
> >> If we spot a problem, we should fix it, but let's not fall into the
> >> "premature optomization" trap.
> >
> > Which part of my proposal are you both criticising?
> >
> > If it's the "we should cache a few request choices in advance" bit then I
> > stand by that. Any database query (assuming the working set is large) will
> > involve many dependant disk accesses.
> 
> Did you read my original post? There were three features in db4o to
> offload that a bit... CachedIoAdapter, prefetching, weakreferences ..
> see my previous post for details on that.

Strictly speaking this is prefetch we're talking about, not caching.
> 
> > We have to send a request roughly every
> > 800ms on my node (for SSKs). With slow commodity drives, often with other
> > disk load going on, and with fairly complex database accesses involving
> > multiple tables and therefore *many* mostly dependant seeks, we are not 
going
> > to reliably meet that deadline no matter how good the database is. I will 
add
> > a statistic for this, but my system is massively overpowered for this sort 
of
> > effect, so we will need to test it on volunteers' slow systems.
> >
> > If it's the "requests that don't want to be persisted", what would you do 
with
> > fproxy requests and other non-persistent requests? Store them to disk 
anyway?


pgpcuqy4VAHBq.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] WoT plugin (db4o ?)

Julien Cornuwel a écrit :

> The other problem is how to store the trust values. With derby, I simply
> have 2 tables, one with identites, the other with the trust values and
> ids if the truster and the trustee. How can I do that with an OO
> database ? The only idea I have is :
> Create a class Trust containing reference to the trusted identity and
> trust value, and store a list of Trust objects in every identites. I
> can't figure how bad it would be for the memory consumption.

Hey ! I just thought about an ugly-but-working solution : to recreate
the relation with an object...

If the Trust class contains trust value and ids of truster and trustee,
I'm able to load only what I want. And I can easily find out what
identites are trusting one particular identity, what's not possible with
a trustlist contained in the identity.



signature.asc
Description: OpenPGP digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] [freenet-cvs] r19930 - trunk/freenet/test/freenet/crypt/ciphers

On Thu, May 15, 2008 at 10:39 PM, Matthew Toseland
<[EMAIL PROTECTED]> wrote:
> What's the problem with the 192-bit test?

see the discussion on r19912,
our Rijndael implementation does not follow FIPS standard. That test
would just fail.

>
> On Wednesday 14 May 2008 16:35, [EMAIL PROTECTED] wrote:
>> Author: j16sdiz
>> Date: 2008-05-14 15:35:44 + (Wed, 14 May 2008)
>> New Revision: 19930
>>
>> Modified:
>>trunk/freenet/test/freenet/crypt/ciphers/RijndaelTest.java
>> Log:
>> Include (part of) FIPS standard test
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] New database for Freenet: db4o

On Sat, May 17, 2008 at 7:06 PM, Matthew Toseland
<[EMAIL PROTECTED]> wrote:
> On Saturday 17 May 2008 06:24, Ian Clarke wrote:
>> On Fri, May 16, 2008 at 11:40 PM, Daniel Cheng
>> <[EMAIL PROTECTED]> wrote:
>> > On Sat, May 17, 2008 at 7:29 AM, Matthew Toseland
>> > <[EMAIL PROTECTED]> wrote:
>> >> Ian and I have eventually come to the conclusion that we should include
> db4o,
>>
>> Yay.
>>
>> >> Of course there will be latency here when objects are
>> >> not cached, so we will need to cache a few request choices in advance for
>> >> each RequestStarter. And we will need to devise some way to deal with
>> >> requests that don't want to be persisted - presumably we'd keep them in
> RAM.
>> >
>> > Please don't.
>> > According to what I have read, db4o should be good enough to use directly:
>>
>> I agree with Daniel, DIY may be an admirable quality when it comes to
>> home and car repair, but not with software.
>>
>> One of the benefits of using third-party stuff like db4o is that we
>> can outsource problems to people far more focussed on the solutions to
>> those problems than we can afford to be.
>>
>> If we spot a problem, we should fix it, but let's not fall into the
>> "premature optomization" trap.
>
> Which part of my proposal are you both criticising?
>
> If it's the "we should cache a few request choices in advance" bit then I
> stand by that. Any database query (assuming the working set is large) will
> involve many dependant disk accesses.

Did you read my original post? There were three features in db4o to
offload that a bit... CachedIoAdapter, prefetching, weakreferences ..
see my previous post for details on that.

> We have to send a request roughly every
> 800ms on my node (for SSKs). With slow commodity drives, often with other
> disk load going on, and with fairly complex database accesses involving
> multiple tables and therefore *many* mostly dependant seeks, we are not going
> to reliably meet that deadline no matter how good the database is. I will add
> a statistic for this, but my system is massively overpowered for this sort of
> effect, so we will need to test it on volunteers' slow systems.
>
> If it's the "requests that don't want to be persisted", what would you do with
> fproxy requests and other non-persistent requests? Store them to disk anyway?
>
> ___
> Devl mailing list
> Devl@freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] WoT plugin (db4o ?)

2008-05-17 Thread bbackde

On Sat, May 17, 2008 at 2:25 PM, Julien Cornuwel <[EMAIL PROTECTED]> wrote:
> I've been reading DB4O's whitepaper (mentionned in Toad's post) and I'm
> wondering if I could use it for the WoT plugin. I could get rid of Derby
> and it would make the deployment simpler if db4o is shipped with freenet.
>
> Query possibilities are sufficient to my needs but I've got a few problems :
>
>
> With a RDBM, I can simply do the calculation of scores in the database
> (with a trigger and stored procedures).
> With db4o, persistence would be easier but I worry about memory
> consumption if I have to load every identities in memory each time a
> trust value is updated. An identity is about 200-250B. You see the
> problem if we have 1.000.000 identites !

I know perst, and I think db4o works similar. When you load persistent objects,
the engine takes care what objects fit in the memory cache. So you can hold
millions of objects but not all of them are always in memory. This can lead to
longer access times in reality.

>
> The other problem is how to store the trust values. With derby, I simply
> have 2 tables, one with identites, the other with the trust values and
> ids if the truster and the trustee. How can I do that with an OO
> database ? The only idea I have is :
> Create a class Trust containing reference to the trusted identity and
> trust value, and store a list of Trust objects in every identites. I
> can't figure how bad it would be for the memory consumption.
>

In perst I use different storages. Each one can have a different cache
size. This
way I can spread the load. But it is also possible to hold all objects
in one storage.

>
> Do you think it is feasible and worth the work ?
>
>
> ___
> Devl mailing list
> Devl@freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>



-- 
__
GnuPG key: (0x48DBFA8A)
Keyserver: pgpkeys.pca.dfn.de
Fingerprint:
477D F057 1BD4 1AE7 8A54 8679 6690 E2EC 48DB FA8A
__
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CPUID patch from FMS

* Florent Daignière <[EMAIL PROTECTED]> [2008-05-17 14:03:10]:

> * Matthew Toseland <[EMAIL PROTECTED]> [2008-05-17 12:29:09]:
> 
> > As nextgens is worried about anonymous contributions, I thought he might 
> > want 
> > to review this before I apply it.
> > 
> 
> There is a related bug in the bug tracker... It's not the only bits that
> need updating :/
> 
> See #74

http://download.intel.com/design/processor/applnots/24161832.pdf
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/25481.pdf

Here are the CPUID codes if you want to update them.


signature.asc
Description: Digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] indent verification broken (was: Verification of r19955 on emu)

* Daniel Cheng  [2008-05-16 23:22:15]:

> an empty indent verification message. =(
> 

That's because of two things:
1) we used a 1.4 jvm with assertion disabled to compile and it
didn't like your "assert"
2) the gpg key we use to sign messages has expired and is
pending renewal (same key that the one we use for mirrors)

I'll make emu regenerate the indent-verification message when those two
issues have been addressed.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/0b880d11/attachment.pgp>

Re: [freenet-dev] WoT plugin (db4o ?)

I've been reading DB4O's whitepaper (mentionned in Toad's post) and I'm
wondering if I could use it for the WoT plugin. I could get rid of Derby
and it would make the deployment simpler if db4o is shipped with freenet.

Query possibilities are sufficient to my needs but I've got a few problems :


With a RDBM, I can simply do the calculation of scores in the database
(with a trigger and stored procedures).
With db4o, persistence would be easier but I worry about memory
consumption if I have to load every identities in memory each time a
trust value is updated. An identity is about 200-250B. You see the
problem if we have 1.000.000 identites !

The other problem is how to store the trust values. With derby, I simply
have 2 tables, one with identites, the other with the trust values and
ids if the truster and the trustee. How can I do that with an OO
database ? The only idea I have is :
Create a class Trust containing reference to the trusted identity and
trust value, and store a list of Trust objects in every identites. I
can't figure how bad it would be for the memory consumption.


Do you think it is feasible and worth the work ?



signature.asc
Description: OpenPGP digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CPUID patch from FMS

* Matthew Toseland <[EMAIL PROTECTED]> [2008-05-17 12:29:09]:

> As nextgens is worried about anonymous contributions, I thought he might want 
> to review this before I apply it.
> 

There is a related bug in the bug tracker... It's not the only bits that
need updating :/

See #74

> CPUID patch
> From:
> [EMAIL PROTECTED]
>   Date:
> Friday 16 May 2008 03:43:30
>   Groups:
> freenet
>   Followup-To:
> freenet
> Message was signed with unknown key 0xB3FFA5B9.
> The validity of the signature cannot be verified.
>   Hi all,
> 
> This patch makes Freenet recognize Intel Pentium M, Core, and Core 2 as
> Pentium 4 compatible, and load corresponding native BigInteger library.
> Tested on my laptop with Core CPU.
> 
> 
> --- freenet/src/freenet/support/CPUInformation/CPUID.java.orig
> 2008-04-23 15:24:19.0 +0800
> +++ freenet/src/freenet/support/CPUInformation/CPUID.java
> 2008-05-16 10:35:55.0 +0800
> @@ -280,7 +280,10 @@
> }
> public boolean IsPentium4Compatible()
> {
> -   return getCPUFamily() >= 15;
> +   int family = getCPUFamily();
> +   int model = getCPUModel();
> +   return (family >= 15 ||
> +   (family == 6 && (model == 9 || model
> >=13)));
> }
> public String getCPUModelString() throws
> UnknownCPUException {
> if(getCPUFamily() == 4){
> 
> 
> Cheers!
> 
> --
> FMS:  [EMAIL PROTECTED]
> Freemail: [EMAIL PROTECTED]
> PGP fingerprint: 88B6 7F21 38E1 01E6 CF3E B1AD 31CC C1F2 B3FF A5B9
> 
> 
>   End of signed message



> ___
> Devl mailing list
> Devl@freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl


signature.asc
Description: Digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] CPUID patch from FMS

As nextgens is worried about anonymous contributions, I thought he might want 
to review this before I apply it.

CPUID patch
From:
[EMAIL PROTECTED]
  Date:
Friday 16 May 2008 03:43:30
  Groups:
freenet
  Followup-To:
freenet
Message was signed with unknown key 0xB3FFA5B9.
The validity of the signature cannot be verified.
  Hi all,

This patch makes Freenet recognize Intel Pentium M, Core, and Core 2 as
Pentium 4 compatible, and load corresponding native BigInteger library.
Tested on my laptop with Core CPU.

--- freenet/src/freenet/support/CPUInformation/CPUID.java.orig
2008-04-23 15:24:19.0 +0800
+++ freenet/src/freenet/support/CPUInformation/CPUID.java
2008-05-16 10:35:55.0 +0800
@@ -280,7 +280,10 @@
}
public boolean IsPentium4Compatible()
{
-   return getCPUFamily() >= 15;
+   int family = getCPUFamily();
+   int model = getCPUModel();
+   return (family >= 15 ||
+   (family == 6 && (model == 9 || model
>=13)));
}
public String getCPUModelString() throws
UnknownCPUException {
if(getCPUFamily() == 4){

Cheers!

--
FMS:  [EMAIL PROTECTED]
Freemail: [EMAIL PROTECTED]
PGP fingerprint: 88B6 7F21 38E1 01E6 CF3E B1AD 31CC C1F2 B3FF A5B9

  End of signed message

pgpjmUNPjWZx1.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] [freenet-cvs] r19930 - trunk/freenet/test/freenet/crypt/ciphers

What's the problem with the 192-bit test?

On Wednesday 14 May 2008 16:35, [EMAIL PROTECTED] wrote:
> Author: j16sdiz
> Date: 2008-05-14 15:35:44 + (Wed, 14 May 2008)
> New Revision: 19930
> 
> Modified:
>trunk/freenet/test/freenet/crypt/ciphers/RijndaelTest.java
> Log:
> Include (part of) FIPS standard test
> 
> 
> Modified: trunk/freenet/test/freenet/crypt/ciphers/RijndaelTest.java
> ===
> --- trunk/freenet/test/freenet/crypt/ciphers/RijndaelTest.java
> 2008-05-14 
15:35:21 UTC (rev 19929)
> +++ trunk/freenet/test/freenet/crypt/ciphers/RijndaelTest.java
> 2008-05-14 
15:35:44 UTC (rev 19930)
> @@ -25,11 +25,11 @@
>   private final byte[] CIPHER192_1 = 
HexUtil.hexToBytes("7fae974786a9741d96693654bc7a8aff09b3f116840ffced");
>  
>   private final byte[] PLAINTXT256_1 = HexUtil
> - 
> .hexToBytes("0123456789abcdef1123456789abcdef2123456789abcdef3123456789abcdef");
> 
+   
.hexToBytes("0123456789abcdef1123456789abcdef2123456789abcdef3123456789abcdef");
>   private final byte[] KEY256_1 = HexUtil
> - 
> .hexToBytes("deadbeefcafebabe0123456789abcdefcafebabedeadbeefcafebabe01234567");
> 
+   
.hexToBytes("deadbeefcafebabe0123456789abcdefcafebabedeadbeefcafebabe01234567");
>   private final byte[] CIPHER256_1 = HexUtil
> - 
> .hexToBytes("6fcbc68fc938e5f5a7c24d7422f4b5f153257b6fb53e0bca26770497dd65078c");
> 
+   
.hexToBytes("6fcbc68fc938e5f5a7c24d7422f4b5f153257b6fb53e0bca26770497dd65078c");
>  
>   private static final Random rand = new Random();
>  
> @@ -62,6 +62,999 @@
>   assertTrue("(256,256) DECIPHER", Arrays.equals(des256, 
> PLAINTXT256_1));
>   }
>  
> + // Standard test vector, available at 
http://csrc.nist.gov/archive/aes/rijndael/rijndael-vals.zip
> +
> + // ecb_vk.txt
> + private byte[] TEST_VK_PT = 
HexUtil.hexToBytes("");
> + private final static byte[][][] TEST_VK128 = { //
> + /* I=1 */
> + { 
> HexUtil.hexToBytes("8000"),
> + 
> HexUtil.hexToBytes("0EDD33D3C621E546455BD8BA1418BEC8") }, //
> + /* I=2 */
> + { 
> HexUtil.hexToBytes("4000"),
> + 
> HexUtil.hexToBytes("C0CC0C5DA5BD63ACD44A80774FAD5222"), }, //
> + /* I=3 */
> + { 
> HexUtil.hexToBytes("2000"),
> + 
> HexUtil.hexToBytes("2F0B4B71BC77851B9CA56D42EB8FF080"), }, //
> + /* I=4 */
> + { 
> HexUtil.hexToBytes("1000"),
> + 
> HexUtil.hexToBytes("6B1E2FFFE8A114009D8FE22F6DB5F876"), }, //
> + /* I=5 */
> + { 
> HexUtil.hexToBytes("0800"),
> + 
> HexUtil.hexToBytes("9AA042C315F94CBB97B62202F83358F5"), }, //
> + /* I=6 */
> + { 
> HexUtil.hexToBytes("0400"),
> + 
> HexUtil.hexToBytes("DBE01DE67E346A800C4C4B4880311DE4"), }, //
> + /* I=7 */
> + { 
> HexUtil.hexToBytes("0200"),
> + 
> HexUtil.hexToBytes("C117D2238D53836ACD92DDCDB85D6A21"), }, //
> + /* I=8 */
> + { 
> HexUtil.hexToBytes("0100"),
> + 
> HexUtil.hexToBytes("DC0ED85DF9611ABB7249CDD168C5467E"), }, //
> + /* I=9 */
> + { 
> HexUtil.hexToBytes("0080"),
> + 
> HexUtil.hexToBytes("807D678FFF1F56FA92DE3381904842F2"), }, //
> + /* I=10 */
> + { 
> HexUtil.hexToBytes("0040"),
> + 
> HexUtil.hexToBytes("0E53B3FCAD8E4B130EF73AEB957FB402"), }, //
> + /* I=11 */
> + { 
> HexUtil.hexToBytes("0020"),
> + 
> HexUtil.hexToBytes("969FFD3B7C35439417E7BDE923035D65"), }, //
> + /* I=12 */
> + { 
> HexUtil.hexToBytes("0010"),
> + 
> HexUtil.hexToBytes("A99B512C19CA56070491166A1503BF15"), }, //
> + /* I=13 */
> + { 
> HexUtil.hexToBytes("0008"),
> + 
> HexUtil.hexToBytes("6E9985252126EE344D26AE369D2327E3"), }, /

Re: [freenet-dev] Combating bloat (was: Re: Post 0.7 idea: off-grid darknet!

On Friday 16 May 2008 22:09, Ian Clarke wrote:
> On Fri, May 16, 2008 at 11:12 AM, Colin Davis <[EMAIL PROTECTED]> wrote:
> >> Why are you so obsessed with turning us into Sourceforge for Freenet
> >> apps?  If we are successful there could be hundreds of apps, there is
> >> no reason for us to host all of them - that is rediculous.  Let them
> >> use sourceforge, or google code, or set up their own website.
> 
> > For the same reason, I suspect, that Apple is hosting a page of Mac
> > Applications, and heavily-pushes third-party applications in it's retail
> > stores, and the same reason that developers around the world are excited
> > that the iPhone will have a built-in appstore.
> 
> You may not have noticed, but we have slightly fewer resources than Apple.
> 
> > Distribution matters. Users don't buy/install a product because of it's
> > inherent properties, they install it because it solves a need.
> > Freenet/Fproxy alone don't solve very many people's need, but Thaw,
> > Freemail, etc are services that people find useful.
> 
> Exactly, which is why Thaw, Freemail, etc are the apps that will
> motivate users to use Freenet.  Only developers download the JRE, most
> users get it bundled with Java apps.  The same will be true of
> Freenet, its a platform, most end-users don't want platforms on their
> own.  The solution is *not* to bundle, that is just pretending that
> Freenet is more than it is.

We have a lot of traffic from wikipedia. We have a lot of traffic from 
slashdot. For a user to even understand what Thaw is he must first understand 
what Freenet is. Thaw, Freemail, FMS and jSite, don't have any sort of web 
presence right now.

Freenet is not the same as Java. It's a bad metaphor. Maybe it would be a 
better metaphor if any major freenet client had a web presence and 
significant hits of its own, but none of them do. AND WE CAN'T WAIT FOR THEM 
TO GET ONE, for much the same reason that we couldn't wait for FMS to release 
0.7.0. That means we have to do what we can for *our users*, which means 
making it as easy as possible to get these client applications.
> 
> Ian.


pgp8vXYYCTqU1.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] New database for Freenet: db4o

On Saturday 17 May 2008 06:24, Ian Clarke wrote:
> On Fri, May 16, 2008 at 11:40 PM, Daniel Cheng
> <[EMAIL PROTECTED]> wrote:
> > On Sat, May 17, 2008 at 7:29 AM, Matthew Toseland
> > <[EMAIL PROTECTED]> wrote:
> >> Ian and I have eventually come to the conclusion that we should include 
db4o,
> 
> Yay.
> 
> >> Of course there will be latency here when objects are
> >> not cached, so we will need to cache a few request choices in advance for
> >> each RequestStarter. And we will need to devise some way to deal with
> >> requests that don't want to be persisted - presumably we'd keep them in 
RAM.
> >
> > Please don't.
> > According to what I have read, db4o should be good enough to use directly:
> 
> I agree with Daniel, DIY may be an admirable quality when it comes to
> home and car repair, but not with software.
> 
> One of the benefits of using third-party stuff like db4o is that we
> can outsource problems to people far more focussed on the solutions to
> those problems than we can afford to be.
> 
> If we spot a problem, we should fix it, but let's not fall into the
> "premature optomization" trap.

Which part of my proposal are you both criticising?

If it's the "we should cache a few request choices in advance" bit then I 
stand by that. Any database query (assuming the working set is large) will 
involve many dependant disk accesses. We have to send a request roughly every 
800ms on my node (for SSKs). With slow commodity drives, often with other 
disk load going on, and with fairly complex database accesses involving 
multiple tables and therefore *many* mostly dependant seeks, we are not going 
to reliably meet that deadline no matter how good the database is. I will add 
a statistic for this, but my system is massively overpowered for this sort of 
effect, so we will need to test it on volunteers' slow systems.

If it's the "requests that don't want to be persisted", what would you do with 
fproxy requests and other non-persistent requests? Store them to disk anyway?

pgp5OIGFuOWd4.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] GPG auto-signing key on emu

* Florent Daignière <[EMAIL PROTECTED]> [2008-05-15 14:56:56]:

> (2) emu's gpg autosigning key has expired and will be renewed when I get
> around to do it

The new key has been generated, you can find details about the key on
http://downloads.freenetproject.org/alpha/README

Ian, toad and me can revoke it if needed.


signature.asc
Description: Digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] New database for Freenet: db4o

Ian and I have eventually come to the conclusion that we should include db4o,
and use it for our various persistence needs. I eventually reached the
conclusion that while we can do most of what we need to do with simple
flatfile databases, there are big chunks that will require a real database of
some kind (even if it's only a persistent hash table). db4o has various
advantages:
- Robust in real-world use. See for example this testimonial from a company
who used it on cell phones:
http://www.db4o.com/about/customers/success/mandalait.aspx
BDBJE has not met our expectations in this regard. It seems very sensitive to
unusual situations - in particular, it will spontaneously corrupt and lose
all data on running out of disk space.
- True object database: no SQL, simple and powerful queries, etc.
- Transparent or manual activation of objects from storage.
- 800K jar, so not big enough to be a problem.
- Mature and actively maintained.
- Allows for future expansion (e.g. passive requests will need to store a fair
amount of persistent data).
- Much more flexible than the hand-coded solution I was thinking of. We can
persistent the entire queue (not just the splitfiles), if it's useful to do
that.
- Transactions (although this requires some juggling of in-memory objects on
rollback).

Tasks:
- Add db4o to freenet-ext.jar.
- Think about using it for the datastore. We don't want to have two databases!
Sdiz's new datastore may be the One True Store, or it may not be. If it's
not, we don't want to keep BDBJE: we could build a db4o-based store, with or
without LRU replacement. It would have the advantage of filling up more
quickly than sdiz's store. It should require reconstructing less frequently
than BDBJE!
- Migrate the client layer, including splitfiles, pendingKeys, and so on, to
be persisted via db4o. Of course there will be latency here when objects are
not cached, so we will need to cache a few request choices in advance for
each RequestStarter. And we will need to devise some way to deal with
requests that don't want to be persisted - presumably we'd keep them in RAM.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20080517/c78314d6/attachment.pgp>

[freenet-dev] New database for Freenet: db4o