[freenet-dev] git/hg hosting
On Thursday 02 April 2009 19:09:14 Ian Clarke wrote: > Whichever source control system we switch to, I propose that we outsource > the repository hosting. For example, if its git, we should use github. > > Reasons: > > - At least in the case of github, it will be free > - We don't have to worry about setting up and administering the repository > - Services like github have a lot of other useful add-ons like a nice web > interface to the repository > - Because of how git works, we can trivially migrate the repository > elsewhere if there is a problem > > Anyone disagree? If so, why? IMHO hosting git (or hg) externally makes sense, for the reasons given and for other reasons. Caveats relate to binaries/workflow: Stable builds: A trusted developer should review changes (from the repository and not from the cvs list), create a tag and sign it, and then, on his own computer, build the binaries and source tarballs for that build, and upload them to our web hosting and to Freenet. Testing builds: We may or may not continue to provide testing builds. If we do they should be an unofficial service, auto-built but should automatically shut down if critical files (e.g. build.xml) are modified. The above is close to what we do now, but Done Right. In the long term we may end up with a more distributed workflow, but it depends on what people want, how it works out in practice; if I get a pseudonymous user asking me to pull his in-Freenet repo (branch), I would certainly have a look! > > Ian. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090404/6965cde4/attachment.pgp>
[freenet-dev] git/hg hosting
On Friday 03 April 2009 15:50:28 Ian Clarke wrote: > On Thu, Apr 2, 2009 at 5:49 PM, Florent Daigni?re < > nextgens at freenetproject.org> wrote: > > > Okay, so it's technically possible (anyway, pulling on a regular basis > > was also an option)... but do we want to fetch code from a remote host > > we don't control and auto-run it on emu? The building process involves > > running the build-scripts. > > Well, we don't control emu either, its sitting in Bytemark's datacenter. > I'd say that github are at least as trustworthy as Bytemark. You miss the point. The auto-build runs on emu, so having compromised emu you can compromise emu! No net gain for any attacker. > I assume > scripts will be run in a walled-off user account, and we can take measures > to sandbox it - but it isn't like we are running the scripts after > downloading them from wikipedia. It is a legitimate concern, if it is misconfigured by either us or them. The right solution is for a trusted dev to code review, create a tag and sign it, and then release binaries from that tag. They can be built on his local machine, eliminating another reason for a central server hosted by us. > > Ian. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090404/ee7c8dce/attachment.pgp>
[freenet-dev] git/hg hosting
On Friday 03 April 2009 09:31:06 Arne Babenhauserheide wrote: > Am Freitag 03 April 2009 02:22:05 schrieb Daniel Cheng: > > DVCS does _NOT_ means accepting anonymous contribution. > > > > However, if we want to, there is nothing stopping us. > > Personally I think it important for freenet to slowly establish a workflow > where people contribute pseudonymously, because that will build a group of > trusted committers which will be essential should freenet ever be outlawed. > > Also that will open the way to a seamless transition to completely freenet- > based development. > > For that workflow freenet needs to use a DVCS. Agreed, however we need to be careful as we can be sued for any code which is copyrighted by somebody else; if we can provide the would-be litigant with the identity of the committer, we don't have this problem. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090404/d12adbfa/attachment.pgp>
[freenet-dev] DB4O branch merged!
p space leaks, as we record the list of buckets to delete before committing a transaction. At some point, not necessarily before releasing a new build, it is planned to implement an automatic garbage collector for temp files (sadly doing this for structures would be a rather larger project, but for temp buckets it's not difficult). -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090404/257ff8b7/attachment.pgp>
[freenet-dev] Invalid number format
I configured port 9000 as FProxy-Port. In the config page the value for FProxy-Port is now shown as "9k". Seems as if a wrong number format is used. -- __ GnuPG key: (0x48DBFA8A) Keyserver: pgpkeys.pca.dfn.de Fingerprint: 477D F057 1BD4 1AE7 8A54 8679 6690 E2EC 48DB FA8A __
Re: [freenet-dev] git/hg hosting
On Thursday 02 April 2009 19:09:14 Ian Clarke wrote: > Whichever source control system we switch to, I propose that we outsource > the repository hosting. For example, if its git, we should use github. > > Reasons: > > - At least in the case of github, it will be free > - We don't have to worry about setting up and administering the repository > - Services like github have a lot of other useful add-ons like a nice web > interface to the repository > - Because of how git works, we can trivially migrate the repository > elsewhere if there is a problem > > Anyone disagree? If so, why? IMHO hosting git (or hg) externally makes sense, for the reasons given and for other reasons. Caveats relate to binaries/workflow: Stable builds: A trusted developer should review changes (from the repository and not from the cvs list), create a tag and sign it, and then, on his own computer, build the binaries and source tarballs for that build, and upload them to our web hosting and to Freenet. Testing builds: We may or may not continue to provide testing builds. If we do they should be an unofficial service, auto-built but should automatically shut down if critical files (e.g. build.xml) are modified. The above is close to what we do now, but Done Right. In the long term we may end up with a more distributed workflow, but it depends on what people want, how it works out in practice; if I get a pseudonymous user asking me to pull his in-Freenet repo (branch), I would certainly have a look! > > Ian. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] git/hg hosting
On Friday 03 April 2009 15:50:28 Ian Clarke wrote: > On Thu, Apr 2, 2009 at 5:49 PM, Florent Daignière < > nextg...@freenetproject.org> wrote: > > > Okay, so it's technically possible (anyway, pulling on a regular basis > > was also an option)... but do we want to fetch code from a remote host > > we don't control and auto-run it on emu? The building process involves > > running the build-scripts. > > Well, we don't control emu either, its sitting in Bytemark's datacenter. > I'd say that github are at least as trustworthy as Bytemark. You miss the point. The auto-build runs on emu, so having compromised emu you can compromise emu! No net gain for any attacker. > I assume > scripts will be run in a walled-off user account, and we can take measures > to sandbox it - but it isn't like we are running the scripts after > downloading them from wikipedia. It is a legitimate concern, if it is misconfigured by either us or them. The right solution is for a trusted dev to code review, create a tag and sign it, and then release binaries from that tag. They can be built on his local machine, eliminating another reason for a central server hosted by us. > > Ian. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] git/hg hosting
On Friday 03 April 2009 09:31:06 Arne Babenhauserheide wrote: > Am Freitag 03 April 2009 02:22:05 schrieb Daniel Cheng: > > DVCS does _NOT_ means accepting anonymous contribution. > > > > However, if we want to, there is nothing stopping us. > > Personally I think it important for freenet to slowly establish a workflow > where people contribute pseudonymously, because that will build a group of > trusted committers which will be essential should freenet ever be outlawed. > > Also that will open the way to a seamless transition to completely freenet- > based development. > > For that workflow freenet needs to use a DVCS. Agreed, however we need to be careful as we can be sued for any code which is copyrighted by somebody else; if we can provide the would-be litigant with the identity of the committer, we don't have this problem. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] DB4O branch merged!
The DB4O branch has finally been merged! Actually it was merged on Thursday, but I've been going through the diffs to make sure I haven't done anything catastrophic. The db4o branch was a project to make Freenet use a database (db4o) for tracking the status of persistent requests and inserts, thus making requests restart instantly on restarting the node, and radically reducing memory usage for large queues. Both of these aims have been achieved; the main catch is slightly greater disk access. Downloads take around 400K static memory usage per queued gigabyte, uploads don't have any static memory usage but in any case you need 80-100M or so of "scratch space". Uploads do however require significant memory when they are being started, so in the default 192MB memory limit, you should be able to start a 4GB insert (and then another one, and then another one), but not anything much larger than that, and/or queue a huge quantity of downloads. More memory does help performance, of course, but in 192M you should be able to do almost anything. The diff merging db4o was over 35,000 lines of code (not all of this additions), bringing the total (not including contrib, but including plugins) to over 298,000. The project was started on May 20th last year. Other notable changes include persistent blob temp files: persistent 32K temporary buckets are allocated from a single large file, persistent-temp-/persistent-blob.tmp. This will automatically grow and shrink as needed, although it does not at present defragment itself. One advantage is that we use much fewer file descriptors, which was getting to be a problem on some nodes; another is that enumerating the files in the persistent temp directory was causing out of memory on startup on nodes with small amounts of memory and 4GB inserts queued. But we may make it optional if people don't like it / if it doesn't work well. HOW YOU CAN HELP The db4o branch was extensively tested before merging, but it is likely that I have missed critical bugs; if you can, please download the new build and test it out, report any bugs you find via https://bugs.freenetproject.org/ or IRC (#freenet or #freenet-testers on irc.freenode.net). Check the known bugs below first. There is considerable work to do before we can release a new stable build (1207) including db4o. Known issues MIGRATING OLD DOWNLOADS/UPLOADS Old downloads and uploads from downloads.dat.gz should generally be transparently imported to the new system, and inserts should keep their existing progress. However, non-global persistent requests may be broken. I think Frost uses, or at least can use, non-global persistent requests, so this will need to be fixed before releasing a new build. PERSISTENCE=REBOOT Persistence=reboot (FCP requests which persist only until the node restarts) may be broken. IMHO this isn't used much, but it will need to be fixed at some point. GARBAGE COLLECTION: STRUCTURES Because db4o does not provide any form of garbage collection, we have to remove structures manually when e.g. requests finish. I have spent considerable time debugging such leaks, however if there ARE any such leaks, it will be very difficult to either recover the space (which should be minimal in the vast majority of cases), or be sure that there isn't any incriminating data from past requests still in the database. Details for the paranoid or the curious: You can however examine the contents of your database to see whether there are any leaks by setting your log level details to freenet.node.node:DEBUG,freenet.client.async.ClientRequestScheduler:DEBUG. You will then get a dump of the count of each type of object in the wrapper.log on startup, and logs/ will have the details of each individual object (search for "DATABASE:"). If you can reproduce such a leak let me know. Note that a few objects are just overhead and are to be expected, even if nothing is queued, also a few objects (FCPClient, a few arrays/lists/maps iirc; if you investigate you'll see a pattern) are the result of FCP clients, and again are not of any great concern. However, if you run a request, and then delete it, and after restarting there are leaking FreenetURIs or ClientCHKs or any other kind of key, there is a serious problem, which I need to know about. (Note that this may be caused by an ERROR e.g. a NullPointerException, check the logs). GARBAGE COLLECTION: TEMP FILES Right now there is no automatic collection of old temp files. Temporary files are removed when a request is removed and/or when they are no longer needed, but if there is a bug (and there always are), it is possible we could have an on-disk space leak. I have tried to eliminate this, it doesn't happen in my testing, but it is possible especially if something breaks. An unclean shutdown at the wrong point *should* not cause such temp space leaks, as we record the list of buckets to delete before commi
[freenet-dev] Invalid number format
I configured port 9000 as FProxy-Port. In the config page the value for FProxy-Port is now shown as "9k". Seems as if a wrong number format is used. -- __ GnuPG key: (0x48DBFA8A) Keyserver: pgpkeys.pca.dfn.de Fingerprint: 477D F057 1BD4 1AE7 8A54 8679 6690 E2EC 48DB FA8A __ ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] git/hg hosting
On Friday 03 April 2009 18:29:04 Arne Babenhauserheide wrote: > > > $ hg sign [REVISION] > > git tag -s -m > Is that a GnuPG signed tag? Yes. Check [1] for an example. David [1]: http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.29.y.git;a=tag;h=5dfd736f95b3d84a18b5bb8e50ac71f245438acf -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090404/13dd9987/attachment.pgp>
[freenet-dev] git/hg hosting
2009/4/4 Arne Babenhauserheide : > Am Freitag 03 April 2009 17:19:13 schrieb David ?Bombe? Roden: >> On Friday 03 April 2009 14:14:41 Arne Babenhauserheide wrote: >> > $ hg sign [REVISION] >> >> git tag -s -m > > Is that a GnuPG signed tag? "-s" is "sign" > > Best wishes, > Arne
