[freenet-dev] Refactoring Freenet and Library was Re: Gun.IO and Freenet
We already have a layout that adheres to the FHS, see debian-staging for
details :)
But this is only for run-time data, NOT the binaries themselves. That part is
rigid and would require much more work, because (to implement it properly)
would need to integrate well with all the various existing installers, as well
as the built-in updater.
I'll respond to the other points some other time, need to be off somewhere now.
(Theoretically it would be possible for fproxy to expose an APT repo under e.g.
localhost:/debian/ that actually gets its data from freenet, but this is
again extra work.)
X
On 16/03/12 20:38, Marco Schulze wrote:
> On 16-03-2012 15:13, Matthew Toseland wrote:
>> Updating its own binaries is incompatible with the standard unix way of doing
>> things, isn't it? Even if it's not technically a violation of FHS?
>
> I'd just like to point out that this is not the case at all, specially because
> flexibility is a major characteristic in this Unix Way of Doing Stuff. Where
> it
> might be problematic, if at all, is on the package management level:
>
> - The ugly custom installer would have to be replaced by a
> distribution-specific package;
> - Some distributions have special rules regarding Java packages. You'd have to
> check those;
>
> You _can_ conform to the FHS without any change by being installed under /opt.
> This will make fred accessible system-wide, so you might want to check if it's
> ok to let multiple users delve inside the Freenet directory tree. However,
> AFAIR, install scripts can do almost anything, including creating a
> fred-specific user and group, and allowing freenet{,-ext}.jar to be updated by
> that user without root privileges.
>
> The new directory layout might look like this:
>
> /etc: freenet.ini.
> /usr/bin: shell scripts to launch and update freenet.
> /usr/lib: native libraries.
> /usr/lib/fred: jars. This might be dependent on the distribution.
> /srv/fred: default download location (if system-wide daemon, ~/Downloads
> otherwise).
> /var/cache/fred: datastore and other miscellaneous persistent files.
> /var/cache/fred/plugins: plugins directory trees.
> /var/log: logs.
> /var/log/old/fred: compressed old logs (do you really need those?).
>
> Plus, distribution specific scripts to control the daemon (run.sh-ish).
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://freenetproject.org/cgi-bin/mailman/listinfo/devl
--
GPG: 4096R/5FBBDBCE
https://github.com/infinity0
https://bitbucket.org/infinity0
https://launchpad.net/~infinity0
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 900 bytes
Desc: OpenPGP digital signature
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20120316/43af71d9/attachment.pgp>
[freenet-dev] Refactoring Freenet and Library was Re: Gun.IO and Freenet
On 16-03-2012 15:13, Matthew Toseland wrote:
> Updating its own binaries is incompatible with the standard unix way
> of doing things, isn't it? Even if it's not technically a violation of
> FHS?
I'd just like to point out that this is not the case at all, specially
because flexibility is a major characteristic in this Unix Way of Doing
Stuff. Where it might be problematic, if at all, is on the package
management level:
- The ugly custom installer would have to be replaced by a
distribution-specific package;
- Some distributions have special rules regarding Java packages. You'd
have to check those;
You _can_ conform to the FHS without any change by being installed under
/opt. This will make fred accessible system-wide, so you might want to
check if it's ok to let multiple users delve inside the Freenet
directory tree. However, AFAIR, install scripts can do almost anything,
including creating a fred-specific user and group, and allowing
freenet{,-ext}.jar to be updated by that user without root privileges.
The new directory layout might look like this:
/etc: freenet.ini.
/usr/bin: shell scripts to launch and update freenet.
/usr/lib: native libraries.
/usr/lib/fred: jars. This might be dependent on the distribution.
/srv/fred: default download location (if system-wide daemon, ~/Downloads
otherwise).
/var/cache/fred: datastore and other miscellaneous persistent files.
/var/cache/fred/plugins: plugins directory trees.
/var/log: logs.
/var/log/old/fred: compressed old logs (do you really need those?).
Plus, distribution specific scripts to control the daemon (run.sh-ish).
[freenet-dev] Refactoring Freenet and Library was Re: Gun.IO and Freenet
Well, the obvious question is 'why?'. Using /opt + /usr/bin scripts +
service scripts seems to be good enough. Either way, fred .jar paths are
configurable, the jars themselves should have 6** permissions and be
owned by the fred user. Why shouldn't autoupdate work?
An unrelated question: it seems debian-staging not only builds slightly
different jars, but are locked on specific build numbers. Why?
On 16-03-2012 17:51, Ximin Luo wrote:
> We already have a layout that adheres to the FHS, see debian-staging for
> details :)
>
> But this is only for run-time data, NOT the binaries themselves. That part is
> rigid and would require much more work, because (to implement it properly)
> would need to integrate well with all the various existing installers, as well
> as the built-in updater.
>
> I'll respond to the other points some other time, need to be off somewhere
> now.
>
> (Theoretically it would be possible for fproxy to expose an APT repo under
> e.g.
> localhost:/debian/ that actually gets its data from freenet, but this is
> again extra work.)
>
> X
>
> On 16/03/12 20:38, Marco Schulze wrote:
>> On 16-03-2012 15:13, Matthew Toseland wrote:
>>> Updating its own binaries is incompatible with the standard unix way of
>>> doing
>>> things, isn't it? Even if it's not technically a violation of FHS?
>> I'd just like to point out that this is not the case at all, specially
>> because
>> flexibility is a major characteristic in this Unix Way of Doing Stuff. Where
>> it
>> might be problematic, if at all, is on the package management level:
>>
>> - The ugly custom installer would have to be replaced by a
>> distribution-specific package;
>> - Some distributions have special rules regarding Java packages. You'd have
>> to
>> check those;
>>
>> You _can_ conform to the FHS without any change by being installed under
>> /opt.
>> This will make fred accessible system-wide, so you might want to check if
>> it's
>> ok to let multiple users delve inside the Freenet directory tree. However,
>> AFAIR, install scripts can do almost anything, including creating a
>> fred-specific user and group, and allowing freenet{,-ext}.jar to be updated
>> by
>> that user without root privileges.
>>
>> The new directory layout might look like this:
>>
>> /etc: freenet.ini.
>> /usr/bin: shell scripts to launch and update freenet.
>> /usr/lib: native libraries.
>> /usr/lib/fred: jars. This might be dependent on the distribution.
>> /srv/fred: default download location (if system-wide daemon, ~/Downloads
>> otherwise).
>> /var/cache/fred: datastore and other miscellaneous persistent files.
>> /var/cache/fred/plugins: plugins directory trees.
>> /var/log: logs.
>> /var/log/old/fred: compressed old logs (do you really need those?).
>>
>> Plus, distribution specific scripts to control the daemon (run.sh-ish).
>> ___
>> Devl mailing list
>> Devl at freenetproject.org
>> http://freenetproject.org/cgi-bin/mailman/listinfo/devl
>
>
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://freenetproject.org/cgi-bin/mailman/listinfo/devl
-- next part --
An HTML attachment was scrubbed...
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20120316/c56000b6/attachment.html>
[freenet-dev] Refactoring Freenet and Library was Re: Gun.IO and Freenet
formance) helps greatly to make the algorithm understandable to other > people, and yourself if you don't work on it for ages. > > If, instead of saying "only fetch a single block" you say "the on-freenet > structure format has persistence issues, and here's a potential solution", > it's > easier for other people to understand what you mean. "only fetching a single block in the common case" is a specific mechanism (relying on significant format changes) for achieving the goal "radical performance optimisation for the average case for library searches". And I have explained it on the bug I quoted. However I agree documentation is a good idea especially in Library. I agree that layering is good. The disagreement comes when the layering is impeding major performance optimisations - then it needs to be altered. > > ## config management > > Code for read/writing the config in embedded inside the classes they control, > which clutters up the code and makes it confusing. Config code should be > separated from the actual application. It would also be nice if this was > exposed to plugins. Separated how exactly? You want to call getters via reflection, like java beans? It is always going to be the case that changing some settings on the fly will be somewhat involved and require methods for it; the classes are only one way to write such an interface, I'm open to other mechanisms given that config happens very infrequently. > > Being a daemon is why the current config system is NOT SUFFICIENT. I need to > be > able to lock certain config settings, such as where to keep the datastore / > run-time files themselves, to conform to the FHS. It's best that such settings > are kept in a separate read-only file. The current system only reads one file, > freenet.ini. Sounds to me like that could be achieved with some fairly small changes; I still don't see what the problem is. > > Updating its own binaries is not a problem if freenet knows where the binaries > are. The current installer puts them in a rigid place, however this is > incompatible with FHS. A rigid place? I don't follow. If you are doing FHS you are using a package; the installer is never going to comply with FHS as, apart from anything else, it shouldn't be run as root! Updating its own binaries is incompatible with the standard unix way of doing things, isn't it? Even if it's not technically a violation of FHS? -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120316/ea6c2b19/attachment.pgp>
[freenet-dev] Refactoring Freenet and Library was Re: Gun.IO and Freenet
On 16-03-2012 15:13, Matthew Toseland wrote:
> Updating its own binaries is incompatible with the standard unix way
> of doing things, isn't it? Even if it's not technically a violation of
> FHS?
I'd just like to point out that this is not the case at all, specially
because flexibility is a major characteristic in this Unix Way of Doing
Stuff. Where it might be problematic, if at all, is on the package
management level:
- The ugly custom installer would have to be replaced by a
distribution-specific package;
- Some distributions have special rules regarding Java packages. You'd
have to check those;
You _can_ conform to the FHS without any change by being installed under
/opt. This will make fred accessible system-wide, so you might want to
check if it's ok to let multiple users delve inside the Freenet
directory tree. However, AFAIR, install scripts can do almost anything,
including creating a fred-specific user and group, and allowing
freenet{,-ext}.jar to be updated by that user without root privileges.
The new directory layout might look like this:
/etc: freenet.ini.
/usr/bin: shell scripts to launch and update freenet.
/usr/lib: native libraries.
/usr/lib/fred: jars. This might be dependent on the distribution.
/srv/fred: default download location (if system-wide daemon, ~/Downloads
otherwise).
/var/cache/fred: datastore and other miscellaneous persistent files.
/var/cache/fred/plugins: plugins directory trees.
/var/log: logs.
/var/log/old/fred: compressed old logs (do you really need those?).
Plus, distribution specific scripts to control the daemon (run.sh-ish).
Re: [freenet-dev] Refactoring Freenet and Library was Re: Gun.IO and Freenet
On Thursday 15 Mar 2012 21:02:26 Ximin Luo wrote: (Top-posting because previous post is too long to reply to in-line.) ## refactoring Refactoring the code helps to make it more understandable for other coders. One of the reasons why I don't work more on freenet myself is because it's extremely difficult to make any changes that are more than a simple bug fix. Hmmm, good point. But from FPI point of view, unless it's a very quick and high impact refactoring, it's probably too long term. When I was writing code for the config subsystem to separate out different files into different directories, this was very frustrating and took up much more time than I expected. I'm still curious as to why. NB: I'm not implying you should be doing all the work toad, please don't think this. And please don't think that my opinions are automatically invalid / less worthy just because I haven't committed code for ages. I just want to express my view of What I Would Do If I Were God. Okay. :) I don't mean to criticise un-constructively, just have a useful conversation. People can do what they want but it doesn't mean it's a good idea. Granted, I consider code quality independently of any issues such as funding, but focusing too much on the latter leads to bad software. If there's pressure like this, ignore it, find something else in the meantime until it goes away. I plan to continue working for FPI for the time being. Partly because it's part time and fits conveniently with studying. Partly because getting a programming job in the UK requires a degree even if you have some rather odd experience. ## freenet Events API freenet does not provide a great API for running callbacks and scheduled tasks. I'm not sure what you mean here. For example, the client layer has unique requirements. We simply can't just change it to use java.util.concurrent. Database jobs must run on the database thread, and no other thread may have access to the database, because the client layer is not designed to deal with simultaneous transactions. This avoids performance problems (separate caches for each client; simultaneous disk I/O), and complexity (refreshing everything constantly, which requires a more query-oriented architecture; we'd have to change pretty much everything, and it'd probably be slower). But it's messy, in particular it requires passing around ObjectContainer's. Converting everything to use java.util.concurrent and/or com.google.common.util.concurrent would help this a lot. Of course, Library can and currently does implement this itself, but it's fairly sloppy, and other plugins would benefit if such a framework were provided. If you simply mean replacing Ticker and Executor, that's fine by me. Some common tasks that plugins would like to do, which really should be provided by the underlying application: - run tasks in the background - run tasks according to a particular schedule - cancel running tasks How do you propose to cancel a task once it's started? I guess it depends what sort of task it is. If it has a boolean and periodically checks it then fine; this would require a subclass ... - handle dependencies between tasks, so that e.g. if A depends on B and I cancel B, A is automatically cancelled. That's a nice bit of functionality yeah. Nothing in fred needs it at present, although radical refactorings might make it more useful. - group tasks into related categories so one group doesn't affect the other. (e.g. tasks for the group Library/b-tree-write/index-URL won't starve the resources of group WoT/background-trust-calculation) If you are scheduling tasks in a limited pool this makes sense. Freenet generally doesn't do this because most tasks are either CPU intensive and complete quickly, or are blocking / I/O intensive and take ages but don't use much resources. Also many tasks are latency sensitive. And on modern JVMs, lots of threads are cheap, although memory is an issue, we do need to keep it down if possible ... ## Library algorithm You keep talking about specific things like only fetching a single block in the common case. Nobody else knows what this means, even I don't (off the top of my head) because I haven't looked at the code in several years. I quoted a bug explaining this. On 15/03/12 13:03, Matthew Toseland wrote: https://bugs.freenetproject.org/view.php?id=4066 And categories: library b-tree-index Back to your mail: Having a proper specification that separates different concerns out into layers (e.g. logical structure; on-freenet structure; data format; network performance) helps greatly to make the algorithm understandable to other people, and yourself if you don't work on it for ages. If, instead of saying only fetch a single block you say the on-freenet structure format has persistence issues, and here's a potential solution, it's easier for other people to understand what you
Re: [freenet-dev] Fewer pages was Re: webui and accessibility
Am Donnerstag, 15. März 2012, 13:15:26 schrieb Matthew Toseland: On Sunday 11 Mar 2012 12:53:11 Arne Babenhauserheide wrote: Am Freitag, 9. März 2012, 21:52:22 schrieb Steve Dougherty: Should we get rid of more of the steps in the first-run setup? I think yes - if people choose the default installation, they should not be forced to take additional decisions. The main difficulty is, one complex page is not better than four simple pages. That’s why it’s important to create that one page in a way that most people can just click Finish. It’s simple, because most people don’t actually need to take a decision here. And I think such an easy way is mostly needed when you use low security: The “just let me run this”-mode. We still need to ask for low security/high security as the first page, with custom = expert mode. Maybe we should rename Custom security to Expert mode ? That’s what I thought: Keep the first page (3 choices: Opennet/Darknet/Custom). In case of Opennet, just show a single page with the most important default settings, so people can change that, but do not have to. Browser warnings should go away most of the time. Please check the default configuration. Change values which don’t fit: * Encrypted Datastore Size: 2 GiB [change] This is sensible. We can produce a sensible default, we can combine it. * Assigned Bandwidth: 10 KiB/s (26 GiB per month) [change] Most connections are asymmetric but Freenet's usage is fairly asymmetric. Whether ISPs charge for / cap total monthly transfer or just download varies. Showing just the output limit probably makes sense here, there's no sense limiting download since it's almost entirely driven by usage... unless you have a monthly cap. Hmmm ... We are trying to avoid having to show cover-your-backside explanation messages here as they bloat everything out to multiple pages ... Maybe we can get away with 26GB per month plus your downloads or something though. Actually my freenet just uses the full assigned bandwidth all the time, and pretty symmetric. Using the lower of in and out as base should work. It would be 51 GiB, though (I think TheSeeker pointed out that I forgot to use in *and* out). [Finish and connect now] I2P works very nicely with something that would work without Javascript, but works much better with: it shows for a given speed setting how much it will likely use per month. That a way better way to deal with monthly bandwidth limits than asking directly. I would like that. I agree that is important. It doesn't need Javascript though. But only after setting it. Javascript shows it directly. And I think that would be nice: Javascript as some grease, but not necessary for features. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. - Arne (http://draketo.de) signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://freenetproject.org/cgi-bin/mailman/listinfo/devl
