[freenet-dev] What is Freenet? still too unclear for newbies? was Fwd: [freenet-support] I Like the Idea but....
This was some time back, so it might not be a comment on the current website. However, he may have a point? One of the responses, from Dennis Nezic, was basically that Freenet is complex, deal with it. However we could make it a bit clearer that Freenet is a metanetwork, that it is NOT a proxy network, but that it's still useful because XYZ? Or is that too negative a point of view - I guess that depends on how good our XYZ are? -- next part -- An embedded message was scrubbed... From: Tim Jones Subject: [freenet-support] I Like the Idea but Date: Mon, 30 Nov 2009 19:58:19 -0500 Size: 5212 URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/af86ada6/attachment.mht> -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/af86ada6/attachment.pgp>
[freenet-dev] Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive
On Friday 02 April 2010 17:31:13 Matthew Toseland wrote: > On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote: > > You should really send these to the support list; that's what it's for. > > > > You can change the physical security level setting independently of > > the network seclevels -- see configuration -> security levels. > > > > I'm not sure what else to suggest at this point. You could try > > increasing the amount of ram for temp buckets (configuration -> core > > settings), but that's mostly a stab in the dark. > > > > I suspect you need to reduce the amount of stuff in your queue. > > Thanks Evan for helping Daniel. In theory it ought to be possible to have a > nearly unlimited number of downloads in the queue: That is precisely why we > decided to use a database to store the progress of downloads. Unfortunately, > in practice, disks are slow, and the more stuff is queued, the less of it > will be cached in RAM i.e. the more reliant we are on slow disks. > > There are many options for optimising the code so that it uses the disk less. > But unfortunately they are all a significant amount of work. > > See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is > marked as related to. So I guess the real question here is, how important is it that we be able to queue 60 downloads and still have acceptable performance? How many users use Freenet filesharing in that sort of way? -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/c55a4113/attachment.pgp>
[freenet-dev] Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive
> > >> > >> > >> From: Evan Daniel > >> To: support at freenetproject.org; Daniel Stork > >> Sent: Mon, January 25, 2010 7:13:43 PM > >> Subject: Re: [freenet-support] major problems - stuck at 100%, > >> nonresponsive > >> > >> On Wed, Jan 20, 2010 at 6:26 PM, Daniel Stork > >> wrote: > >>> Hi, > >>> > >>> I'm having major problems with freenet on Windows, I have 60 downloads of > >>> which 60 have been stuck at 100% for days. > >>> Running freenet makes Windows completely unresponsive. > >>> It takes literally 10 minutes for frost to start up. > >>> This happened in the past. I deleted node.db4o and the permanent > >>> downloads > >>> folder and this fixed it for a while, > >>> But it goes back to the same state in a few days. > >>> Right now my node.db4o is 230 Mb and I don't want to lost the 60 > >>> downloads > >>> (almost 10 gigs total) which are complete. > >>> The CPU usage is 50-80% on a very strong pc. > >>> > >>> My questions are the following: > >>> > >>> 1. Could the unresponsiveness be a memory issue with Java ? I have 4 gigs > >>> but freenet and frost use only 160 and 210 megabytes. Is java putting a > >>> limit on these somehow? > >>> What's the proper way to allocate memory to freenet and frost ? > >> > >> Freenet has a configuration option. You can set it from configuration > >> -> core settings -> max memory. > >> > >> For frost, run it with "java -Xmx256M -jar frost.jar" (or whatever > >> setting you prefer) instead of the normal "java -jar frost.jar". > >> > >> It's possible your issue is Freenet memory; I'm not certain. Please > >> let me know if increasing memory available helps. > >> > >>> > >>> 2. Does setting priority in task manager have any effect ? I noticed they > >>> are on "below normal" and cannot be changed. > >> > >> I'm not certain. Freenet normally runs most of its threads at very > >> low priority, and a couple at higher priority. Reducing the priority > >> too far on some OSes can mean the high priority threads get starved > >> for CPU, causing timeouts and restarts and such. I'm not sure if this > >> happens on windows. > >> > >>> > >>> 3. Is there a way to save these completed downloads that freenet is not > >>> finishing (i.e. command line utility)? > >> > >> Just the normal download process. Reducing the size of your queue > >> will fix the problem, and increasing the memory available may help. > >> > >>> > >>> Also another issue I noticed: > >>> > >>> - When I select "Download the file in the background and store in > >>> R:\Freenet\downloads" or "Fetch the file in the background" from the > >>> freenet > >>> UI, > >>> it doesn't do anything. Are these supposed to work? > >> > >> They should add the file to your download queue; they work fine here. > >> What does happen? What error message are you getting? > >> > >> Evan Daniel -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/0806df23/attachment.pgp>
[freenet-dev] [Tech] GSoC 2010 --- developing a firefox plugin
On Friday 02 April 2010 00:37:13 Ian Clarke wrote: > Hi Kyle, > > Very glad that you are enthusiastic about Freenet. > > Can a browser plugin install and run a piece of Java software with full > access to a user's computer? This needs to be discussed on devl, and has been a bit so far. Basically it comes down to uptime and storage (IF the Java problem is surmountable; I don't think plugins are sandboxed much on firefox, though Google does?): If it only runs when you're actually using Freenet, it will have very bad uptime and this may have bad effects on the network, although maybe we don't want to completely rule it out on that basis. If it runs whenever your browser runs, it will slow down your browser (but maybe this isn't an issue any more than when installing it). And in any case Freenet needs substantial long-term storage. > > Ian. > > On Thu, Mar 25, 2010 at 1:18 AM, Kyle Messner gmail.com>wrote: > > > Hi Freenet! > > > > Browsing through the Google Summer of Code 2010 list of approved > > organization one name in particular caught my attention, I'm sure you can > > guess what this is. "Freenet..." I thought, "Hmm, with a name like that must > > be sure to do something pretty neat!" I was not let down :) > > > > Censorship has been something that has always been one of the truest evils > > in my eyes. So when I learned of the Freenet objective, I was pretty > > excited. This is a project that can really bring the "wide" (as in open) > > back into the "world wide web." In short, I'd really like to help. > > > > People are usually a little apprehensive about installing new applications > > on their computer, I don't blame them. Who wants to bloat up their computer > > with useless software (not that freenet falls in this category, but > > generally many programs do). But a browser-plug in? Sure, why not, one > > click, I let my browser do it's thing, and poof, I have added functionality > > to my computer. So I thought, "Why not develop a firefox plug-in that allows > > users access to a lot of freenet's functionality?" > > > > So there it is, there's my idea. I'd like to implement as much of the > > freenet features as possible as a browser plug-in to firefox. The most > > important features I'd like to implement are viewing freenet sites, using > > the email client, and using the message service, although I'd like to get to > > as many as I can. I'd like to know what Freenet thinks of this idea, and if > > they think it would be a feasible GSoC project to propose. > > > > A little more about me. I'm currently finishing up my sophomore year in > > pursuit of a bachelor's degree in Computer Science (hopefully move on to my > > master's after). I'm very passionate about software design, especially in > > the world of open-source (although I believe proprietary software has its > > place, too). I have some experience with C++ and JavaScript, most of my > > experience lies with Java, Visual C#, and Visual Basic (I also had 5 > > semesters of Pascal in high school which helped teach me many fundamental > > concepts about programming). I'm a quick learner, and self-teacher. > > Developing software for an open-source project sounds like an ideal summer > > job to me :) > > > > Look forward to hearing back, > > Kyle > > > > ___ > > Tech mailing list > > Tech at freenetproject.org > > http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/tech > > > > > > -- > Ian Clarke > CEO, SenseArray > Email: ian at sensearray.com > Ph: +1 512 422 3588 > -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/8e4bc729/attachment.pgp>
[freenet-dev] Please use an interface
cedfab1d3f866c26930575d008174236868a0a62: Please use an interface for reporting the stats, and make NodeStats implement it. We ideally want classes in support/ to be reusable. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/70733316/attachment.pgp>
[freenet-dev] Development of Free Net Project
is at > http://github.com/infinity0/plugin-Library-staging > > there are some notes in ./doc/ and ./TODO - you can pick things out of that. > if > anything is unclear, ask me. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/c664f99f/attachment.pgp>
[freenet-dev] Development of Free Net Project
> On Saturday 06 February 2010 16:39:53 Karol Pysniak wrote: >> Hi, I would like to take part in the development of Free Net Project as my >> contribution to Google Summer of Code. I am especially interested in Data >> Bases, Algorithms and AI. I think that it would be great to try to create >> new, 'intelligent' search engine. what do you mean by "intelligent" search engine? there is another student, lusha on IRC, lushawang at gmail.com, interested in doing a project on search. you two should talk to each other. if we take you both, then you will need to be doing two distinct things - google doesn't allow students to collaborate on a single proposal, but two related (but distinct) proposals are fine. On 04/02/2010 01:11 PM, Matthew Toseland wrote: > - Unfortunately there are severe scaling problems with this format: The > sub-indexes can get huge, and inserting them all as a single freesite also > causes problems. - Plus, the spider's architecture, involving a database of > terms and URIs, also doesn't scale. It takes a week or more to write the > index from the database. It would be better to maintain the database on the > fly, rewriting affected parts every few hours as new keys are spidered. the work to do in this area is described in some detail at http://new-wiki.freenetproject.org/Talk:Library you'll need to understand how SkeletonBTreeMap.update() works first; if the source code is too unintuitive (async so components are scattered), then ask me for an explanation. > - We have a new format, created by infinity0, a Summer of Code student last > year, which should scale much better. This is based on b-trees, and > therefore data can be loaded into it progressively and just the changed > nodes are re-uploaded. However, currently the spider uses the old format. So > the first task would be to make the spider use the new format and load data > into it progressively (on the fly). Also, when the new format is on Freenet, > it is forkable - meaning that not only can the original author of the index > add data and only insert those nodes affected (including their parents), but > anyone else can also add their own changes - which do not affect the > original btree - and reuse the existing ones. In other words, it is a "copy > on write btree", although I don't think it has all the tweaks that the COW > btrees paper talks about. This may have many wider applications, e.g. > merging others' indexes, and maybe eventually distributing the spidering > process. there is a COW btree paper? do you have a link? > - The other half of infinity0's Summer of Code project was to be > distributed search. This would allow each user to publish indexes and to > link to other users' indexes; it is described on the wiki. i'm working on this atm as part of my uni course; i've coded a prototype and atm i'm collecting data to test it with (hoop-jumping for dissertation :/). my deadline is mid-may so if you/lusha want to pick it up afterwards and work on it for GSoC then i'm happy to explain how it works. however there is plenty of stuff to work on Library already, IMO it would be better to get the basics working before trying to graft a more complex system onto it. > - Most likely infinity0 would be your mentor. that's me, btw :) i'm on irc under that nick. > You should read: http://new-wiki.freenetproject.org/Library > http://new-wiki.freenetproject.org/B-tree_indexes > http://new-wiki.freenetproject.org/Web_of_Trust And of course some of the > papers/videos/introductory stuff on our web page. source code is at http://github.com/infinity0/plugin-Library-staging there are some notes in ./doc/ and ./TODO - you can pick things out of that. if anything is unclear, ask me. X
[freenet-dev] Development of Free Net Project
On Saturday 06 February 2010 16:39:53 Karol Pysniak wrote: > Hi, > I would like to take part in the development of Free Net Project as my > contribution to Google Summer of Code. I am especially interested in > Data Bases, Algorithms and AI. I think that it would be great to try to > create new, 'intelligent' search engine. > > I can program in Java/C++/C/Haskell/Assembler(IA-32). Great. You will need to apply via the Summer of Code web interface. If you want to send us your proposal so we can look at it and suggest improvements, please do so here. Before we accept you we will need you to demonstrate some basic coding ability by making some small change (bugfix or feature) to Freenet. See the bug tracker at https://bugs.freenetproject.org/ for ideas. You should apply for at least two tasks within Freenet, so that we are able to choose both students and projects, and don't have to drop a good student because we already have somebody else for that project. Please read up on Freenet first or this may not make much sense: The most fundamental thing is that Freenet only provides "insert" (publish data to a key) and "request" (fetch a key) operations and everything else is built on top of it, including searching. There have been proposals for distributed searching but doing so in a secure and spam-proof way is remarkably difficult so for now we will probably continue to build search on top of inserts and requests... As regards searching, here is the current situation. - A spider (XMLSpider plugin) spiders Freenet freesites and generates an index, which is inserted into Freenet in the same way as a freesite would be. - Another plugin downloads the relevant parts of the index when you do a search, combines them and displays the matching URIs. - The old XMLLibrarian plugin implemented a simple XML-based search index format. One file contains the list of sub-indexes (split by md5 of the word being looked up), and then there is one file for each sub-index. Within each sub-index, we have a list of URIs, words, and which URIs each word is contained in. For an example of this format, please have a look here (install Freenet first): USK at 5hH~39FtjA7A9~VXWtBKI~prUDTuJZURudDG0xFn3KA,GDgRGt5f6xqbmo-WraQtU54x4H~871Sho9Hz6hC-0RA,AQACAAE/Search/24/index.xml USK at 5hH~39FtjA7A9~VXWtBKI~prUDTuJZURudDG0xFn3KA,GDgRGt5f6xqbmo-WraQtU54x4H~871Sho9Hz6hC-0RA,AQACAAE/Search/24/index_00.xml - This format is still supported by the new Library plugin, which replaced XMLLibrarian for the frontend some time ago. - Unfortunately there are severe scaling problems with this format: The sub-indexes can get huge, and inserting them all as a single freesite also causes problems. - Plus, the spider's architecture, involving a database of terms and URIs, also doesn't scale. It takes a week or more to write the index from the database. It would be better to maintain the database on the fly, rewriting affected parts every few hours as new keys are spidered. - We have a new format, created by infinity0, a Summer of Code student last year, which should scale much better. This is based on b-trees, and therefore data can be loaded into it progressively and just the changed nodes are re-uploaded. However, currently the spider uses the old format. So the first task would be to make the spider use the new format and load data into it progressively (on the fly). Also, when the new format is on Freenet, it is forkable - meaning that not only can the original author of the index add data and only insert those nodes affected (including their parents), but anyone else can also add their own changes - which do not affect the original btree - and reuse the existing ones. In other words, it is a "copy on write btree", although I don't think it has all the tweaks that the COW btrees paper talks about. This may have many wider applications, e.g. merging others' indexes, and maybe eventually distributing the spidering process. - The other half of infinity0's Summer of Code project was to be distributed search. This would allow each user to publish indexes and to link to other users' indexes; it is described on the wiki. - Most likely infinity0 would be your mentor. - There is limited support for basic page ranking based on word frequencies. I don't think the current search indexes support the metadata needed, but the spider should if I remember correctly. You should read: http://new-wiki.freenetproject.org/Library http://new-wiki.freenetproject.org/B-tree_indexes http://new-wiki.freenetproject.org/Web_of_Trust And of course some of the papers/videos/introductory stuff on our web page. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/49eb283c/attachment.pgp>
[freenet-dev] Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive
On Fri, Apr 2, 2010 at 12:39 PM, Matthew Toseland wrote: > On Friday 02 April 2010 17:31:13 Matthew Toseland wrote: >> On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote: >> > You should really send these to the support list; that's what it's for. >> > >> > You can change the physical security level setting independently of >> > the network seclevels -- see configuration -> security levels. >> > >> > I'm not sure what else to suggest at this point. ?You could try >> > increasing the amount of ram for temp buckets (configuration -> core >> > settings), but that's mostly a stab in the dark. >> > >> > I suspect you need to reduce the amount of stuff in your queue. >> >> Thanks Evan for helping Daniel. In theory it ought to be possible to have a >> nearly unlimited number of downloads in the queue: That is precisely why we >> decided to use a database to store the progress of downloads. Unfortunately, >> in practice, disks are slow, and the more stuff is queued, the less of it >> will be cached in RAM i.e. the more reliant we are on slow disks. >> >> There are many options for optimising the code so that it uses the disk >> less. But unfortunately they are all a significant amount of work. >> >> See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is >> marked as related to. > > So I guess the real question here is, how important is it that we be able to > queue 60 downloads and still have acceptable performance? How many users use > Freenet filesharing in that sort of way? All of them, I suspect. If a file is mostly downloaded, but not complete, the natural response seems to be to leave it there in hopes it will complete, and add other files in the mean time. Combined with unretrievable files due to missing blocks, this will produce very large download queues. Evan Daniel
[freenet-dev] Announcing the GSOC on the website
On Thursday 25 March 2010 15:30:02 cvollet at gmail.com wrote: > AFAIK, there is no news on the website to say that the freenet project has > been accepted to the GSOC. > > Imho, it would be a good idea to add one, given that it's not really easy to > find out where to find the infos (and that it doesn't take much time...) > Done. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100402/97872eb8/attachment.pgp>
[freenet-dev] What is Freenet? still too unclear for newbies? was Fwd: [freenet-support] I Like the Idea but....
This was some time back, so it might not be a comment on the current website. However, he may have a point? One of the responses, from Dennis Nezic, was basically that Freenet is complex, deal with it. However we could make it a bit clearer that Freenet is a metanetwork, that it is NOT a proxy network, but that it's still useful because XYZ? Or is that too negative a point of view - I guess that depends on how good our XYZ are? --- Begin Message --- I think you need to publish a more user friendly description of the product and how it works. I am a fairly experienced PC/Windows user for many years and I made a career of data administration using technology. However, reading your website just makes my head spin much as trying to decipher Linux and trying to understand what advantages I gain from deploying freenet that is more and better than something like PGP or using proxies. Regardless of one's views of the relative merits of the software platforms (Windows, MacOS, Linux, etc) the fact remains that we do live in a MS Windows world and I think that describing your product relative to the Windows experience would go a long way towards enticing users to try your product. Personally, I think this approach of keeping the description and explanation cryptic and mystical has been the single block that Linux has embedded in the public perception and has drastically limited its adoption on a much larger scale. I see your product headed down that same road, a good if not outstanding product that is hamstrung by its own creators for lack of a focus on explaining the product to the prospective end-user base in a way that base can understand. I think developers lose sight of their audience when promoting their work and forget that the target audience (end-user) is largely ignorant of what goes on "under the hood". Hopefully, you will take this message as constructive and not a nag. In either event it's my opinion and I hope you give it more than a little consideration. Best Regards, Tim Jones Sebring, Florida USA ___ Support mailing list supp...@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe --- End Message --- signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive
On Fri, Apr 2, 2010 at 12:39 PM, Matthew Toseland wrote: > On Friday 02 April 2010 17:31:13 Matthew Toseland wrote: >> On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote: >> > You should really send these to the support list; that's what it's for. >> > >> > You can change the physical security level setting independently of >> > the network seclevels -- see configuration -> security levels. >> > >> > I'm not sure what else to suggest at this point. You could try >> > increasing the amount of ram for temp buckets (configuration -> core >> > settings), but that's mostly a stab in the dark. >> > >> > I suspect you need to reduce the amount of stuff in your queue. >> >> Thanks Evan for helping Daniel. In theory it ought to be possible to have a >> nearly unlimited number of downloads in the queue: That is precisely why we >> decided to use a database to store the progress of downloads. Unfortunately, >> in practice, disks are slow, and the more stuff is queued, the less of it >> will be cached in RAM i.e. the more reliant we are on slow disks. >> >> There are many options for optimising the code so that it uses the disk >> less. But unfortunately they are all a significant amount of work. >> >> See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is >> marked as related to. > > So I guess the real question here is, how important is it that we be able to > queue 60 downloads and still have acceptable performance? How many users use > Freenet filesharing in that sort of way? All of them, I suspect. If a file is mostly downloaded, but not complete, the natural response seems to be to leave it there in hopes it will complete, and add other files in the mean time. Combined with unretrievable files due to missing blocks, this will produce very large download queues. Evan Daniel ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive
On Friday 02 April 2010 17:31:13 Matthew Toseland wrote: > On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote: > > You should really send these to the support list; that's what it's for. > > > > You can change the physical security level setting independently of > > the network seclevels -- see configuration -> security levels. > > > > I'm not sure what else to suggest at this point. You could try > > increasing the amount of ram for temp buckets (configuration -> core > > settings), but that's mostly a stab in the dark. > > > > I suspect you need to reduce the amount of stuff in your queue. > > Thanks Evan for helping Daniel. In theory it ought to be possible to have a > nearly unlimited number of downloads in the queue: That is precisely why we > decided to use a database to store the progress of downloads. Unfortunately, > in practice, disks are slow, and the more stuff is queued, the less of it > will be cached in RAM i.e. the more reliant we are on slow disks. > > There are many options for optimising the code so that it uses the disk less. > But unfortunately they are all a significant amount of work. > > See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is > marked as related to. So I guess the real question here is, how important is it that we be able to queue 60 downloads and still have acceptable performance? How many users use Freenet filesharing in that sort of way? signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
[freenet-dev] Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive
On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote: > You should really send these to the support list; that's what it's for. > > You can change the physical security level setting independently of > the network seclevels -- see configuration -> security levels. > > I'm not sure what else to suggest at this point. You could try > increasing the amount of ram for temp buckets (configuration -> core > settings), but that's mostly a stab in the dark. > > I suspect you need to reduce the amount of stuff in your queue. Thanks Evan for helping Daniel. In theory it ought to be possible to have a nearly unlimited number of downloads in the queue: That is precisely why we decided to use a database to store the progress of downloads. Unfortunately, in practice, disks are slow, and the more stuff is queued, the less of it will be cached in RAM i.e. the more reliant we are on slow disks. There are many options for optimising the code so that it uses the disk less. But unfortunately they are all a significant amount of work. See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is marked as related to. > On Mon, Mar 8, 2010 at 8:05 AM, Daniel Stork wrote: > > Defragmenting the database did help. It went from 520 Mb to 160 Mb, > > This made it a bit more responsive and the smaller files now finished in > > about an hour, > > but the larger ones are still stuck at 100%. > > > > Could you tell me how to change the location of the persistent temp folder? > > I didn't see this in freenet.ini > > > > I'd like to put to node.db4o.crypt file on the ramdisk, but the persistent > > temp is way too big for that. > > Does the node.db4o have to be in the same folder as the persistent temp ? > > > > Also, if I'm running freenet from either a ramdisk or a truecrypt volume, > > than does it make sense to have the persistent temp and the datastore and > > db4o encrypted ? > > Is it possible to just have these unencrypted without affecting my online > > security settings ? > > > > > > > > Thanks a lot, > > > > Dan > > > > > > > > > > From: Evan Daniel > > To: Daniel Stork ; supp...@freenetproject.org > > Sent: Sat, March 6, 2010 4:38:23 PM > > Subject: Re: [freenet-support] major problems - stuck at 100%, nonresponsive > > > > I suspect the stalled downloads are the same problem as the heavy IO, > > and that both come from the downloads database. I would expect > > increasing the memory available to help; I'm somewhat surprised it > > doesn't. I doubt there's much io to the datastore in comparison. If > > you want to play with ram disks, putting the data store on a normal > > hard disk and the node.db4o (or node.db4o.crypt) file on a ram disk is > > more likely to help. However, first I would try defragmenting your > > node.db4o file (configuration -> core settings -> Defragment the > > downloads database during the next startup? -> true). Does setting > > that and then restarting the node help? How big was your node.db4o > > file before / after defragmenting? > > > > If none of this helps, then I suspect you simply have more downloads > > queued than Freenet can handle. I recommend removing some or all of > > the files, and then re-adding them when others finish, keeping the > > total size queued at any one time limited. > > > > Evan Daniel > > > > On Sat, Mar 6, 2010 at 10:03 AM, Daniel Stork wrote: > >> Evan, thanks for the response, I tried playing around with the memory, and > >> giving freenet 2 gb makes it crash, > >> but it works with 1,5gb (I have a total of 4gb installed). > >> > >> The memory did not change anything. The disk was churning a lot so I > >> transferred the datastore to a 2gb ramdisk, which reduced some of it. > >> But still the system becomes really unresponsive, when using freenet. > >> > >> Any ideas what this could be? All my hardware is really more than enough, > >> I > >> have one of the best Core 2 Duos and all resources are underutilized. > >> > >> Also, - I know others have asked already, but am not sure if this issue > >> was > >> ever resolved - I have numerous downloads at 100% that do not complete. > >> I have been waiting for hours and days. > >> > >> Any idea why this happens? > >> > >> I usually have about 80-100 simultaneous downloads, is this too much for > >> freenet to handle? > >> > >> Thanks a lot, > >> > >> > >> > >> From: Evan Daniel > >> To: supp...@freenetproject.org; Daniel Stork > >> Sent: Mon, January 25, 2010 7:13:43 PM > >> Subject: Re: [freenet-support] major problems - stuck at 100%, > >> nonresponsive > >> > >> On Wed, Jan 20, 2010 at 6:26 PM, Daniel Stork wrote: > >>> Hi, > >>> > >>> I'm having major problems with freenet on Windows, I have 60 downloads of > >>> which 60 have been stuck at 100% for days. > >>> Running freenet makes Windows completely unresponsive. > >>> It takes literally 10 minutes for frost to start up. > >>> This happened in the past. I deleted node.db
Re: [freenet-dev] [Tech] GSoC 2010 --- developing a firefox plugin
On Friday 02 April 2010 00:37:13 Ian Clarke wrote: > Hi Kyle, > > Very glad that you are enthusiastic about Freenet. > > Can a browser plugin install and run a piece of Java software with full > access to a user's computer? This needs to be discussed on devl, and has been a bit so far. Basically it comes down to uptime and storage (IF the Java problem is surmountable; I don't think plugins are sandboxed much on firefox, though Google does?): If it only runs when you're actually using Freenet, it will have very bad uptime and this may have bad effects on the network, although maybe we don't want to completely rule it out on that basis. If it runs whenever your browser runs, it will slow down your browser (but maybe this isn't an issue any more than when installing it). And in any case Freenet needs substantial long-term storage. > > Ian. > > On Thu, Mar 25, 2010 at 1:18 AM, Kyle Messner wrote: > > > Hi Freenet! > > > > Browsing through the Google Summer of Code 2010 list of approved > > organization one name in particular caught my attention, I'm sure you can > > guess what this is. "Freenet..." I thought, "Hmm, with a name like that must > > be sure to do something pretty neat!" I was not let down :) > > > > Censorship has been something that has always been one of the truest evils > > in my eyes. So when I learned of the Freenet objective, I was pretty > > excited. This is a project that can really bring the "wide" (as in open) > > back into the "world wide web." In short, I'd really like to help. > > > > People are usually a little apprehensive about installing new applications > > on their computer, I don't blame them. Who wants to bloat up their computer > > with useless software (not that freenet falls in this category, but > > generally many programs do). But a browser-plug in? Sure, why not, one > > click, I let my browser do it's thing, and poof, I have added functionality > > to my computer. So I thought, "Why not develop a firefox plug-in that allows > > users access to a lot of freenet's functionality?" > > > > So there it is, there's my idea. I'd like to implement as much of the > > freenet features as possible as a browser plug-in to firefox. The most > > important features I'd like to implement are viewing freenet sites, using > > the email client, and using the message service, although I'd like to get to > > as many as I can. I'd like to know what Freenet thinks of this idea, and if > > they think it would be a feasible GSoC project to propose. > > > > A little more about me. I'm currently finishing up my sophomore year in > > pursuit of a bachelor's degree in Computer Science (hopefully move on to my > > master's after). I'm very passionate about software design, especially in > > the world of open-source (although I believe proprietary software has its > > place, too). I have some experience with C++ and JavaScript, most of my > > experience lies with Java, Visual C#, and Visual Basic (I also had 5 > > semesters of Pascal in high school which helped teach me many fundamental > > concepts about programming). I'm a quick learner, and self-teacher. > > Developing software for an open-source project sounds like an ideal summer > > job to me :) > > > > Look forward to hearing back, > > Kyle > > > > ___ > > Tech mailing list > > t...@freenetproject.org > > http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/tech > > > > > > -- > Ian Clarke > CEO, SenseArray > Email: i...@sensearray.com > Ph: +1 512 422 3588 > signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
[freenet-dev] Please use an interface
cedfab1d3f866c26930575d008174236868a0a62: Please use an interface for reporting the stats, and make NodeStats implement it. We ideally want classes in support/ to be reusable. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Development of Free Net Project
On Friday 02 April 2010 13:41:12 Ximin Luo wrote: > > On Saturday 06 February 2010 16:39:53 Karol Pysniak wrote: > >> Hi, I would like to take part in the development of Free Net Project as my > >> contribution to Google Summer of Code. I am especially interested in Data > >> Bases, Algorithms and AI. I think that it would be great to try to create > >> new, 'intelligent' search engine. > > what do you mean by "intelligent" search engine? Google is now getting into social search. The WoT stuff might deliver something similar... > > there is another student, lusha on IRC, lushaw...@gmail.com, interested in > doing a project on search. you two should talk to each other. if we take you > both, then you will need to be doing two distinct things - google doesn't > allow > students to collaborate on a single proposal, but two related (but distinct) > proposals are fine. We did this last year, it worked fine. > > On 04/02/2010 01:11 PM, Matthew Toseland wrote: > > - Unfortunately there are severe scaling problems with this format: The > > sub-indexes can get huge, and inserting them all as a single freesite also > > causes problems. - Plus, the spider's architecture, involving a database of > > terms and URIs, also doesn't scale. It takes a week or more to write the > > index from the database. It would be better to maintain the database on the > > fly, rewriting affected parts every few hours as new keys are spidered. s/database/index. Maintain the index on the fly. The database then is just a list of URLs that we have and have not spidered yet. > > the work to do in this area is described in some detail at > http://new-wiki.freenetproject.org/Talk:Library > > you'll need to understand how SkeletonBTreeMap.update() works first; if the > source code is too unintuitive (async so components are scattered), then ask > me > for an explanation. > > > - We have a new format, created by infinity0, a Summer of Code student last > > year, which should scale much better. This is based on b-trees, and > > therefore data can be loaded into it progressively and just the changed > > nodes are re-uploaded. However, currently the spider uses the old format. So > > the first task would be to make the spider use the new format and load data > > into it progressively (on the fly). Also, when the new format is on Freenet, > > it is forkable - meaning that not only can the original author of the index > > add data and only insert those nodes affected (including their parents), but > > anyone else can also add their own changes - which do not affect the > > original btree - and reuse the existing ones. In other words, it is a "copy > > on write btree", although I don't think it has all the tweaks that the COW > > btrees paper talks about. This may have many wider applications, e.g. > > merging others' indexes, and maybe eventually distributing the spidering > > process. > > there is a COW btree paper? do you have a link? http://en.wikipedia.org/wiki/BTRFS#History The core data structure of Btrfs—the copy-on-write B-tree—was originally proposed by IBM researcher Ohad Rodeh at a presentation at USENIX 2007. Rodeh suggested adding reference counts and certain relaxations to the balancing algorithms of standard B-trees that would make them suitable for a high-performance object store with copy-on-write snapshots, yet maintain good concurrency.[19] https://www.usenix.org/events/lsf07/tech/rodeh.pdf (Did I mention that COW btrees are incredibly awesome? ;) ) > > > - The other half of infinity0's Summer of Code project was to be > > distributed search. This would allow each user to publish indexes and to > > link to other users' indexes; it is described on the wiki. > > i'm working on this atm as part of my uni course; i've coded a prototype and > atm i'm collecting data to test it with (hoop-jumping for dissertation :/). my > deadline is mid-may so if you/lusha want to pick it up afterwards and work on > it for GSoC then i'm happy to explain how it works. > > however there is plenty of stuff to work on Library already, IMO it would be > better to get the basics working before trying to graft a more complex system > onto it. > > > - Most likely infinity0 would be your mentor. > > that's me, btw :) i'm on irc under that nick. > > > You should read: http://new-wiki.freenetproject.org/Library > > http://new-wiki.freenetproject.org/B-tree_indexes > > http://new-wiki.freenetproject.org/Web_of_Trust And of course some of the > > papers/videos/introductory stuff on our web page. > > source code is at > http://github.com/infinity0/plugin-Library-staging > > there are some notes in ./doc/ and ./TODO - you can pick things out of that. > if > anything is unclear, ask me. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Development of Free Net Project
> On Saturday 06 February 2010 16:39:53 Karol Pysniak wrote: >> Hi, I would like to take part in the development of Free Net Project as my >> contribution to Google Summer of Code. I am especially interested in Data >> Bases, Algorithms and AI. I think that it would be great to try to create >> new, 'intelligent' search engine. what do you mean by "intelligent" search engine? there is another student, lusha on IRC, lushaw...@gmail.com, interested in doing a project on search. you two should talk to each other. if we take you both, then you will need to be doing two distinct things - google doesn't allow students to collaborate on a single proposal, but two related (but distinct) proposals are fine. On 04/02/2010 01:11 PM, Matthew Toseland wrote: > - Unfortunately there are severe scaling problems with this format: The > sub-indexes can get huge, and inserting them all as a single freesite also > causes problems. - Plus, the spider's architecture, involving a database of > terms and URIs, also doesn't scale. It takes a week or more to write the > index from the database. It would be better to maintain the database on the > fly, rewriting affected parts every few hours as new keys are spidered. the work to do in this area is described in some detail at http://new-wiki.freenetproject.org/Talk:Library you'll need to understand how SkeletonBTreeMap.update() works first; if the source code is too unintuitive (async so components are scattered), then ask me for an explanation. > - We have a new format, created by infinity0, a Summer of Code student last > year, which should scale much better. This is based on b-trees, and > therefore data can be loaded into it progressively and just the changed > nodes are re-uploaded. However, currently the spider uses the old format. So > the first task would be to make the spider use the new format and load data > into it progressively (on the fly). Also, when the new format is on Freenet, > it is forkable - meaning that not only can the original author of the index > add data and only insert those nodes affected (including their parents), but > anyone else can also add their own changes - which do not affect the > original btree - and reuse the existing ones. In other words, it is a "copy > on write btree", although I don't think it has all the tweaks that the COW > btrees paper talks about. This may have many wider applications, e.g. > merging others' indexes, and maybe eventually distributing the spidering > process. there is a COW btree paper? do you have a link? > - The other half of infinity0's Summer of Code project was to be > distributed search. This would allow each user to publish indexes and to > link to other users' indexes; it is described on the wiki. i'm working on this atm as part of my uni course; i've coded a prototype and atm i'm collecting data to test it with (hoop-jumping for dissertation :/). my deadline is mid-may so if you/lusha want to pick it up afterwards and work on it for GSoC then i'm happy to explain how it works. however there is plenty of stuff to work on Library already, IMO it would be better to get the basics working before trying to graft a more complex system onto it. > - Most likely infinity0 would be your mentor. that's me, btw :) i'm on irc under that nick. > You should read: http://new-wiki.freenetproject.org/Library > http://new-wiki.freenetproject.org/B-tree_indexes > http://new-wiki.freenetproject.org/Web_of_Trust And of course some of the > papers/videos/introductory stuff on our web page. source code is at http://github.com/infinity0/plugin-Library-staging there are some notes in ./doc/ and ./TODO - you can pick things out of that. if anything is unclear, ask me. X ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Development of Free Net Project
On Saturday 06 February 2010 16:39:53 Karol Pysniak wrote: > Hi, > I would like to take part in the development of Free Net Project as my > contribution to Google Summer of Code. I am especially interested in > Data Bases, Algorithms and AI. I think that it would be great to try to > create new, 'intelligent' search engine. > > I can program in Java/C++/C/Haskell/Assembler(IA-32). Great. You will need to apply via the Summer of Code web interface. If you want to send us your proposal so we can look at it and suggest improvements, please do so here. Before we accept you we will need you to demonstrate some basic coding ability by making some small change (bugfix or feature) to Freenet. See the bug tracker at https://bugs.freenetproject.org/ for ideas. You should apply for at least two tasks within Freenet, so that we are able to choose both students and projects, and don't have to drop a good student because we already have somebody else for that project. Please read up on Freenet first or this may not make much sense: The most fundamental thing is that Freenet only provides "insert" (publish data to a key) and "request" (fetch a key) operations and everything else is built on top of it, including searching. There have been proposals for distributed searching but doing so in a secure and spam-proof way is remarkably difficult so for now we will probably continue to build search on top of inserts and requests... As regards searching, here is the current situation. - A spider (XMLSpider plugin) spiders Freenet freesites and generates an index, which is inserted into Freenet in the same way as a freesite would be. - Another plugin downloads the relevant parts of the index when you do a search, combines them and displays the matching URIs. - The old XMLLibrarian plugin implemented a simple XML-based search index format. One file contains the list of sub-indexes (split by md5 of the word being looked up), and then there is one file for each sub-index. Within each sub-index, we have a list of URIs, words, and which URIs each word is contained in. For an example of this format, please have a look here (install Freenet first): u...@5hh~39ftja7a9~vxwtbki~prudtujzuruddg0xfn3ka,GDgRGt5f6xqbmo-WraQtU54x4H~871Sho9Hz6hC-0RA,AQACAAE/Search/24/index.xml u...@5hh~39ftja7a9~vxwtbki~prudtujzuruddg0xfn3ka,GDgRGt5f6xqbmo-WraQtU54x4H~871Sho9Hz6hC-0RA,AQACAAE/Search/24/index_00.xml - This format is still supported by the new Library plugin, which replaced XMLLibrarian for the frontend some time ago. - Unfortunately there are severe scaling problems with this format: The sub-indexes can get huge, and inserting them all as a single freesite also causes problems. - Plus, the spider's architecture, involving a database of terms and URIs, also doesn't scale. It takes a week or more to write the index from the database. It would be better to maintain the database on the fly, rewriting affected parts every few hours as new keys are spidered. - We have a new format, created by infinity0, a Summer of Code student last year, which should scale much better. This is based on b-trees, and therefore data can be loaded into it progressively and just the changed nodes are re-uploaded. However, currently the spider uses the old format. So the first task would be to make the spider use the new format and load data into it progressively (on the fly). Also, when the new format is on Freenet, it is forkable - meaning that not only can the original author of the index add data and only insert those nodes affected (including their parents), but anyone else can also add their own changes - which do not affect the original btree - and reuse the existing ones. In other words, it is a "copy on write btree", although I don't think it has all the tweaks that the COW btrees paper talks about. This may have many wider applications, e.g. merging others' indexes, and maybe eventually distributing the spidering process. - The other half of infinity0's Summer of Code project was to be distributed search. This would allow each user to publish indexes and to link to other users' indexes; it is described on the wiki. - Most likely infinity0 would be your mentor. - There is limited support for basic page ranking based on word frequencies. I don't think the current search indexes support the metadata needed, but the spider should if I remember correctly. You should read: http://new-wiki.freenetproject.org/Library http://new-wiki.freenetproject.org/B-tree_indexes http://new-wiki.freenetproject.org/Web_of_Trust And of course some of the papers/videos/introductory stuff on our web page. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Announcing the GSOC on the website
On Thursday 25 March 2010 15:30:02 cvol...@gmail.com wrote: > AFAIK, there is no news on the website to say that the freenet project has > been accepted to the GSOC. > > Imho, it would be a good idea to add one, given that it's not really easy to > find out where to find the infos (and that it doesn't take much time...) > Done. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl
[freenet-dev] French wiki rss feed down
Le jeudi 01 avril 2010 20:05:11, Ximin Luo a ?crit : > should be fixed; let me know if it isn't. > > X > It works. Thanks :)
