[freenet-dev] Update on getting rid of emu

[Juiceman]

On Fri, Oct 30, 2009 at 3:16 PM, Matthew Toseland  wrote:

> On Friday 30 October 2009 19:13:18 Florent Daigni?re wrote:
> > * Ian Clarke  [2009-10-30 10:57:41]:
> >
> > > On Fri, Oct 30, 2009 at 10:29 AM, Matthew Toseland
> > >  wrote:
> > > > A disk seems to be failing. Fortunately we have two in RAID1. I have
> taken a recent
> > > > backup and should contact bytemark if we decide to keep it...
> > >
> > > We should tell them anyway.  Doesn't cost us anything if they fix it,
> > > and they'd probably want to know.
> > >
> >
> > It costs downtime and there is always a risk of loosing data/not being
> able to
> > reboot (raid1 is weak).
> >
> > > > The website is currently all static
> > >
> > > I still think this is ridiculous, and pointlessly limiting.
> > >
> > > > There are a few free options e.g. sourceforge, berlios, also Google
> if we don't mind
> > > > crappy I-am-not-an-Iranian click-throughs.
> > >
> > > Google Groups has spam issues :-/
> > >
> > > > BUG TRACKER:
> > >
> > > I'm fond of "Lighthouse", its got a simple clean flexible interface,
> and an API.
> >
> > What about switching everything back to SF? The have both mantis (where
> > importing the current DB might be possible) and phpBB (forum for user
> > support; imho one of the most deployed).
>
> Re data import on mantis, not likely afaics, they don't support data import
> now and when we asked they said ask for an enhancement.
>
> Apart from that, you may have a point.
>

Surely there are ways of importing a csv text file at least?  We could get
that out of Mantis, no?
-- next part --
An HTML attachment was scrubbed...
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20091030/5f5aafd3/attachment.html>

[freenet-dev] Update on getting rid of emu

[xor]

On Friday 30 October 2009 16:29:57 Matthew Toseland wrote:
>
> If we don't keep the bug tracker:

> - We will need to do a "spring clean": Keep the current bug tracker up for
> a while but read-only, *manually* migrate any important bugs and issues to
> the new tracker. - This will be significant work.
> - It will involve going over the bugs, dumping those which are out of date,
> abandoned etc, and rewriting those bugs and feature issues that are still
> valid. Trac's wiki functionality may be useful for this, although it loses
> the ability to link bugs formally. - It may be a useful exercise in terms
> of prioritising and de-junking.
>

Migrating it in read-only mode is insane because there would be no sane way 
for monitoring the migration progress: Which bugs have been reviewed & 
migrated?

Instead, we should clean up mantis itself until ALL bugs there are suitable 
for migration and then migrate. 

Besides, as I've said numerous times, we MUST migrate all bugs because from a 
software engineering point of view the content of a bugtracker is the most 
valuable data of a project besides the soure code and the documentation.
Deleting any of it without reviewing it is just unprofessional, so we must 
make very sure that no issues in mantis get lost.

BTW: When did someone do the last backup of it's database? Does emu have a 
backup system?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20091030/ee82bb45/attachment.pgp>

[freenet-dev] Update on getting rid of emu

[Florent Daignière]

* xor  [2009-10-30 20:21:51]:

> On Friday 30 October 2009 16:29:57 Matthew Toseland wrote:
> >
> > If we don't keep the bug tracker:
> 
> > - We will need to do a "spring clean": Keep the current bug tracker up for
> > a while but read-only, *manually* migrate any important bugs and issues to
> > the new tracker. - This will be significant work.
> > - It will involve going over the bugs, dumping those which are out of date,
> > abandoned etc, and rewriting those bugs and feature issues that are still
> > valid. Trac's wiki functionality may be useful for this, although it loses
> > the ability to link bugs formally. - It may be a useful exercise in terms
> > of prioritising and de-junking.
> >
> 
> Migrating it in read-only mode is insane because there would be no sane way 
> for monitoring the migration progress: Which bugs have been reviewed & 
> migrated?
> 
> Instead, we should clean up mantis itself until ALL bugs there are suitable 
> for migration and then migrate. 
> 
> Besides, as I've said numerous times, we MUST migrate all bugs because from a 
> software engineering point of view the content of a bugtracker is the most 
> valuable data of a project besides the soure code and the documentation.
> Deleting any of it without reviewing it is just unprofessional, so we must 
> make very sure that no issues in mantis get lost.
> 

HAHAHAHA. At least that made me laugh!

> BTW: When did someone do the last backup of it's database? Does emu have a 
> backup system?

There use to be a daily backup... on the local disk from another VM, but
as it's full I'm almost sure it's not working anymore... So the remote
backups are probably backing up over and over the same useless data.

NextGen$
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20091030/6fc2509f/attachment.pgp>

[freenet-dev] Update on getting rid of emu

[Matthew Toseland]

On Friday 30 October 2009 17:10:02 Zero3 wrote:
> Matthew Toseland wrote:
> > We need to migrate the Wikka wiki to MediaWiki, because the latter is 
> > standard and we will be able to host it externally. E.g. sourceforge Hosted 
> > Apps allows for data import for MediaWiki.
> 
> The wiki isn't really kept enough up-to-date, is it? I personally think 
> the idea of something more integrated (like Trac) would be awesome. 
> MediaWiki seems a bit over-do for Freenet?
> 
> > BUG TRACKER:
> > - Dev intelligence i.e. stuff people have said. If these are corroborated 
> > quickly they should be acted upon, else they should be closed.
> 
> Something integrated (same user account, for starters) might help 
> motivating people to put this stuff in the wiki as well.

I agree it would help, but maybe not by very much? It would however make it a 
lot more readable.
> 
> > If we keep the bug tracker:
> > - We need to find somewhere to host MANTIS. Probably we will have to pay 
> > for this.
> > - We need to keep it up to date ourselves, which is somewhat involved. It 
> > may not be as bad as nextgens implies though.
> > - Minimum immediate work.
> 
> I personally think Mantis is *very* bad usability-wise. Trac, Launchpad, 
> and many other bug trackers are much easier to use. If we even have to 
> pay just to keep that thing running, I'd say find something else.

Is Trac easy enough *for end-users*?
> 
> > If we don't keep the bug tracker:
> > - We can use any hosted bug tracker anywhere. E.g. Sourceforge Hosted Apps 
> > includes both Mantis and Trac. We will likely be able to avoid any fixed 
> > monthly payments.
> > - We can use any bug tracker: Mantis, Trac, etc. See below.
> > - We will need to do a "spring clean": Keep the current bug tracker up for 
> > a while but read-only, *manually* migrate any important bugs and issues to 
> > the new tracker.
> > - This will be significant work.
> > - It will involve going over the bugs, dumping those which are out of date, 
> > abandoned etc, and rewriting those bugs and feature issues that are still 
> > valid. Trac's wiki functionality may be useful for this, although it loses 
> > the ability to link bugs formally.
> > - It may be a useful exercise in terms of prioritising and de-junking.
> > 
> > However, we have 20 weeks left of funding, so we have to ask whether 
> > spending a week de-junking is worth it?
> 
> If it comes down to costing us ?40-?80 per month... It quickly runs up. 
> I'm up for giving a hand and doing 5 'a day of the de-junking and 
> moving. I'm sure there are more people around willing to give a hand.

That would be helpful yeah. I can then subscribe to some sort of feed and 
review rather than having to do everything myself. :)
> 
> > An important related point: Relatively few end-users use the current bug 
> > tracker. It is on the Contribute menu on the website, but the main reason 
> > IMHO is it is not very newbie friendly. Uservoice is a reasonable solution 
> > for end-user feature suggestions and gauging public opinion, but because it 
> > does not force users to register their email addresses, it is worthless for 
> > solving individual reproducible bugs. It might reasonably be argued that we 
> > should have a separate issue tracker or forums system for end-user bug 
> > reports. Also, it may make sense for the developer-oriented bug tracker to 
> > be open source, whereas it matters less for the end-user tracker, because 
> > 1) end-users care less, and 2) long term stuff with detailed implementation 
> > notes is likely to be on the developer-oriented bug tracker.
> 
> Again, let's find something more user-friendly than Mantis :/
> 
> > If this line of reasoning is correct, we need to choose an 
> > end-user-oriented issue tracker or forums system (either way ideally gratis 
> > and hosted) to complement Uservoice. Suggestions?
> 
> It would make sense to find a tracker that both users and devs can use. 
> Saves the overhead of moving things from e.g. a forums system to a bug 
> tracker.

Is it possible? Is Trac something that end users can use?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20091030/9d37dcd7/attachment.pgp>

[freenet-dev] Update on getting rid of emu

[Matthew Toseland]

On Friday 30 October 2009 19:13:18 Florent Daigni?re wrote:
> * Ian Clarke  [2009-10-30 10:57:41]:
> 
> > On Fri, Oct 30, 2009 at 10:29 AM, Matthew Toseland
> >  wrote:
> > > A disk seems to be failing. Fortunately we have two in RAID1. I have 
> > > taken a recent
> > > backup and should contact bytemark if we decide to keep it...
> > 
> > We should tell them anyway.  Doesn't cost us anything if they fix it,
> > and they'd probably want to know.
> > 
> 
> It costs downtime and there is always a risk of loosing data/not being able to
> reboot (raid1 is weak).
> 
> > > The website is currently all static
> > 
> > I still think this is ridiculous, and pointlessly limiting.
> > 
> > > There are a few free options e.g. sourceforge, berlios, also Google if we 
> > > don't mind
> > > crappy I-am-not-an-Iranian click-throughs.
> > 
> > Google Groups has spam issues :-/
> > 
> > > BUG TRACKER:
> > 
> > I'm fond of "Lighthouse", its got a simple clean flexible interface, and an 
> > API.
> 
> What about switching everything back to SF? The have both mantis (where
> importing the current DB might be possible) and phpBB (forum for user
> support; imho one of the most deployed).

Re data import on mantis, not likely afaics, they don't support data import now 
and when we asked they said ask for an enhancement.

Apart from that, you may have a point.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20091030/a7612fe6/attachment.pgp>

[freenet-dev] Update on getting rid of emu

[Florent Daignière]

* Ian Clarke  [2009-10-30 10:57:41]:

> On Fri, Oct 30, 2009 at 10:29 AM, Matthew Toseland
>  wrote:
> > A disk seems to be failing. Fortunately we have two in RAID1. I have taken 
> > a recent
> > backup and should contact bytemark if we decide to keep it...
> 
> We should tell them anyway.  Doesn't cost us anything if they fix it,
> and they'd probably want to know.
> 

It costs downtime and there is always a risk of loosing data/not being able to
reboot (raid1 is weak).

> > The website is currently all static
> 
> I still think this is ridiculous, and pointlessly limiting.
> 
> > There are a few free options e.g. sourceforge, berlios, also Google if we 
> > don't mind
> > crappy I-am-not-an-Iranian click-throughs.
> 
> Google Groups has spam issues :-/
> 
> > BUG TRACKER:
> 
> I'm fond of "Lighthouse", its got a simple clean flexible interface, and an 
> API.
> 

What about switching everything back to SF? The have both mantis (where
importing the current DB might be possible) and phpBB (forum for user
support; imho one of the most deployed).

NextGen$
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20091030/9ac20751/attachment.pgp>

[freenet-dev] Update on getting rid of emu

[Zero3]

Matthew Toseland wrote:
> We need to migrate the Wikka wiki to MediaWiki, because the latter is 
> standard and we will be able to host it externally. E.g. sourceforge Hosted 
> Apps allows for data import for MediaWiki.

The wiki isn't really kept enough up-to-date, is it? I personally think 
the idea of something more integrated (like Trac) would be awesome. 
MediaWiki seems a bit over-do for Freenet?

> BUG TRACKER:
> - Dev intelligence i.e. stuff people have said. If these are corroborated 
> quickly they should be acted upon, else they should be closed.

Something integrated (same user account, for starters) might help 
motivating people to put this stuff in the wiki as well.

> If we keep the bug tracker:
> - We need to find somewhere to host MANTIS. Probably we will have to pay for 
> this.
> - We need to keep it up to date ourselves, which is somewhat involved. It may 
> not be as bad as nextgens implies though.
> - Minimum immediate work.

I personally think Mantis is *very* bad usability-wise. Trac, Launchpad, 
and many other bug trackers are much easier to use. If we even have to 
pay just to keep that thing running, I'd say find something else.

> If we don't keep the bug tracker:
> - We can use any hosted bug tracker anywhere. E.g. Sourceforge Hosted Apps 
> includes both Mantis and Trac. We will likely be able to avoid any fixed 
> monthly payments.
> - We can use any bug tracker: Mantis, Trac, etc. See below.
> - We will need to do a "spring clean": Keep the current bug tracker up for a 
> while but read-only, *manually* migrate any important bugs and issues to the 
> new tracker.
> - This will be significant work.
> - It will involve going over the bugs, dumping those which are out of date, 
> abandoned etc, and rewriting those bugs and feature issues that are still 
> valid. Trac's wiki functionality may be useful for this, although it loses 
> the ability to link bugs formally.
> - It may be a useful exercise in terms of prioritising and de-junking.
> 
> However, we have 20 weeks left of funding, so we have to ask whether spending 
> a week de-junking is worth it?

If it comes down to costing us ?40-?80 per month... It quickly runs up. 
I'm up for giving a hand and doing 5 'a day of the de-junking and 
moving. I'm sure there are more people around willing to give a hand.

> An important related point: Relatively few end-users use the current bug 
> tracker. It is on the Contribute menu on the website, but the main reason 
> IMHO is it is not very newbie friendly. Uservoice is a reasonable solution 
> for end-user feature suggestions and gauging public opinion, but because it 
> does not force users to register their email addresses, it is worthless for 
> solving individual reproducible bugs. It might reasonably be argued that we 
> should have a separate issue tracker or forums system for end-user bug 
> reports. Also, it may make sense for the developer-oriented bug tracker to be 
> open source, whereas it matters less for the end-user tracker, because 1) 
> end-users care less, and 2) long term stuff with detailed implementation 
> notes is likely to be on the developer-oriented bug tracker.

Again, let's find something more user-friendly than Mantis :/

> 
> If this line of reasoning is correct, we need to choose an end-user-oriented 
> issue tracker or forums system (either way ideally gratis and hosted) to 
> complement Uservoice. Suggestions?

It would make sense to find a tracker that both users and devs can use. 
Saves the overhead of moving things from e.g. a forums system to a bug 
tracker.

- Zero3

[freenet-dev] Update on getting rid of emu

[Matthew Toseland]

Emu:

A disk seems to be failing. Fortunately we have two in RAID1. I have taken a 
recent backup and should contact bytemark if we decide to keep it...

Emu costs us approx ?80/month. Ian has been generously paying half of this, as 
it was used for his other projects (but isn't really atm), but I am not sure 
whether that is half of ?80/month or half of ?160/month.

Mirrors network:

The Java Web Start installer now uses Google Code (as do other downloads), so 
we now only use the mirror network for update.cmd/update.sh. That requires a 
fixed, secure URL to fetch the latest pointer and its sha1sum from, but could 
fetch the data from the mirrors network. That URL could be through Google App 
Engine or through some cheap hosting provider or possibly even sourceforge.

Website:

The website is currently all static. In future we will need language 
negotiation (which can be done statically), and possibly OS detection for the 
front page/download page (which can also be done statically although there may 
be reasons not to). Cheap/free hosting may not be fast enough when we get a 
slashdot, especially if we add some scripting but even if we don't (as they 
generally use apache so static content isn't necessarily dramatically faster). 
Google App Engine probably would be plenty fast, although we may need a paid 
account (which will probably only cost us when we are slashdotted).

Wiki:

We need to migrate the Wikka wiki to MediaWiki, because the latter is standard 
and we will be able to host it externally. E.g. sourceforge Hosted Apps allows 
for data import for MediaWiki.

Mailing lists:

There are a few free options e.g. sourceforge, berlios, also Google if we don't 
mind crappy I-am-not-an-Iranian click-throughs.

BUG TRACKER:

Basically it comes down to do we want to keep the existing bugs.

As I see it, bugs fall into 5 categories:
- Bugs abandoned by the end-user. These should be closed.
- Out of date bugs. These should *usually* be closed, but sometimes are in fact 
live bugs.
- Dev intelligence i.e. stuff people have said. If these are corroborated 
quickly they should be acted upon, else they should be closed.
- Live bugs.
- Feature requests, mostly by developers, often inter-linked with many other 
feature requests, linked to various end-user bug reports, and often with a lot 
of info on 1) consequences of implementation, and 2) how to implement.

If we keep the bug tracker:
- We need to find somewhere to host MANTIS. Probably we will have to pay for 
this.
- We need to keep it up to date ourselves, which is somewhat involved. It may 
not be as bad as nextgens implies though.
- Minimum immediate work.

If we don't keep the bug tracker:
- We can use any hosted bug tracker anywhere. E.g. Sourceforge Hosted Apps 
includes both Mantis and Trac. We will likely be able to avoid any fixed 
monthly payments.
- We can use any bug tracker: Mantis, Trac, etc. See below.
- We will need to do a "spring clean": Keep the current bug tracker up for a 
while but read-only, *manually* migrate any important bugs and issues to the 
new tracker.
- This will be significant work.
- It will involve going over the bugs, dumping those which are out of date, 
abandoned etc, and rewriting those bugs and feature issues that are still 
valid. Trac's wiki functionality may be useful for this, although it loses the 
ability to link bugs formally.
- It may be a useful exercise in terms of prioritising and de-junking.

However, we have 20 weeks left of funding, so we have to ask whether spending a 
week de-junking is worth it?

An important related point: Relatively few end-users use the current bug 
tracker. It is on the Contribute menu on the website, but the main reason IMHO 
is it is not very newbie friendly. Uservoice is a reasonable solution for 
end-user feature suggestions and gauging public opinion, but because it does 
not force users to register their email addresses, it is worthless for solving 
individual reproducible bugs. It might reasonably be argued that we should have 
a separate issue tracker or forums system for end-user bug reports. Also, it 
may make sense for the developer-oriented bug tracker to be open source, 
whereas it matters less for the end-user tracker, because 1) end-users care 
less, and 2) long term stuff with detailed implementation notes is likely to be 
on the developer-oriented bug tracker.

If this line of reasoning is correct, we need to choose an end-user-oriented 
issue tracker or forums system (either way ideally gratis and hosted) to 
complement Uservoice. Suggestions?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20091030/eecca6b3/attachment.pgp>

[freenet-dev] Depency on emu

[Zero3]

Speaking of servers, how is our depency on emu looking? There was some 
discussion a while ago to move things off emu to save the monthly expense?

- Zero3

Matthew Toseland wrote:
> On Tuesday 20 October 2009 04:46:10 Ian Clarke wrote:
>> I reported this bug a few days ago:
>>
>>   https://bugs.freenetproject.org/view.php?id=3604
>>
>> Read it for the details, but basically I'm seeing an intermittent
>> problem where Webstart can't download the installer.  Is anyone else
>> seeing it?
> 
> Reproduced the bug, fixed it by pointing the JWS installer to Google Code. 
> Which means that we now only use the mirror network for update.sh / 
> update.cmd. The freenet.jnlp itself is served from emu directly, so the JNLP 
> still says it's downloading from checksums.freenetproject.org.
> 
> 
> 
> 
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Update on getting rid of emu

[Evan Daniel]

On Fri, Oct 30, 2009 at 1:10 PM, Zero3  wrote:
> Matthew Toseland wrote:
>> We need to migrate the Wikka wiki to MediaWiki, because the latter is 
>> standard and we will be able to host it externally. E.g. sourceforge Hosted 
>> Apps allows for data import for MediaWiki.
>
> The wiki isn't really kept enough up-to-date, is it? I personally think
> the idea of something more integrated (like Trac) would be awesome.
> MediaWiki seems a bit over-do for Freenet?

I find MediaWiki far easier to use; it's overkill, but that doesn't
much matter.  If it's easier to use, it's easier to keep up to date.

Among other things, the current search sucks (making it hard to use
the wiki to answer a question), there aren't redirects (making it hard
to find the correct page), and we don't have categories and other
navigational aids.  Moving to MediaWiki would make it far easier for
users to get information from the wiki, and therefore a more
attractive place to keep documentation.

Evan Daniel

[freenet-dev] Update on getting rid of emu

[Ian Clarke]

On Fri, Oct 30, 2009 at 10:29 AM, Matthew Toseland
 wrote:
> A disk seems to be failing. Fortunately we have two in RAID1. I have taken a 
> recent
> backup and should contact bytemark if we decide to keep it...

We should tell them anyway.  Doesn't cost us anything if they fix it,
and they'd probably want to know.

> The website is currently all static

I still think this is ridiculous, and pointlessly limiting.

> There are a few free options e.g. sourceforge, berlios, also Google if we 
> don't mind
> crappy I-am-not-an-Iranian click-throughs.

Google Groups has spam issues :-/

> BUG TRACKER:

I'm fond of "Lighthouse", its got a simple clean flexible interface, and an API.

Ian.

-- 
Ian Clarke
CEO, Uprizer Labs
Email: ian at uprizer.com
Ph: +1 512 422 3588

[freenet-dev] is anyone else experiencing problems with the JNLP installer?

[Matthew Toseland]

On Tuesday 20 October 2009 04:46:10 Ian Clarke wrote:
> I reported this bug a few days ago:
> 
>   https://bugs.freenetproject.org/view.php?id=3604
> 
> Read it for the details, but basically I'm seeing an intermittent
> problem where Webstart can't download the installer.  Is anyone else
> seeing it?

Reproduced the bug, fixed it by pointing the JWS installer to Google Code. 
Which means that we now only use the mirror network for update.sh / update.cmd. 
The freenet.jnlp itself is served from emu directly, so the JNLP still says 
it's downloading from checksums.freenetproject.org.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-dev] Errors on first start of Freenet after fresh re-install

[Matthew Toseland]

On Thursday 22 October 2009 16:42:18 Matthew Toseland wrote:
> On Saturday 17 October 2009 18:16:20 Ian Clarke wrote:
> > Just did a fresh re-install, and I'm seeing these errors as soon as
> > I've finished the first-time wizard, not a good first impression!
> > 
> > Could not load plugin!
> > The plugin Library could not be loaded: error while fetching plugin:
> > Route not found - could not find enough nodes to be sure the data
> > doesn't exist for key
> > CHK at 
> > iWml5dwVavKmqZh6K6uUUEP8daBSxoQEE3WKpuHNSW0,K4xGaH3zRbVT6wVV7mGFespZMOzzg7rJF92Y-7NXnt0,AAIC--8/Library.jar
> > 
> > We are still trying to load the plugin over Freenet.
> > 
> > You can try again over Freenet, or you can fetch it over the web.
> > 
> > Could not load plugin!
> > The plugin ThawIndexBrowser could not be loaded: unexpected error
> > while plugin loading java.lang.UnsupportedClassVersionError: Bad
> > version number in .class file
> > 
> > We are still trying to load the plugin over Freenet.
> > 
> > You can try again over Freenet, or you can fetch it over the web.
> 
> All of these are fixed with the new installers released last week. Thanks!

Unfortunately they crept back in on the unix/mac installer for 1238. Fixed now.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-dev] Depency on emu

[Zero3]

Speaking of servers, how is our depency on emu looking? There was some 
discussion a while ago to move things off emu to save the monthly expense?

- Zero3

Matthew Toseland wrote:
 On Tuesday 20 October 2009 04:46:10 Ian Clarke wrote:
 I reported this bug a few days ago:

   https://bugs.freenetproject.org/view.php?id=3604

 Read it for the details, but basically I'm seeing an intermittent
 problem where Webstart can't download the installer.  Is anyone else
 seeing it?
 
 Reproduced the bug, fixed it by pointing the JWS installer to Google Code. 
 Which means that we now only use the mirror network for update.sh / 
 update.cmd. The freenet.jnlp itself is served from emu directly, so the JNLP 
 still says it's downloading from checksums.freenetproject.org.
 
 
 
 
 ___
 Devl mailing list
 Devl@freenetproject.org
 http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Update on getting rid of emu

[Matthew Toseland]

Emu:

A disk seems to be failing. Fortunately we have two in RAID1. I have taken a 
recent backup and should contact bytemark if we decide to keep it...

Emu costs us approx £80/month. Ian has been generously paying half of this, as 
it was used for his other projects (but isn't really atm), but I am not sure 
whether that is half of £80/month or half of £160/month.

Mirrors network:

The Java Web Start installer now uses Google Code (as do other downloads), so 
we now only use the mirror network for update.cmd/update.sh. That requires a 
fixed, secure URL to fetch the latest pointer and its sha1sum from, but could 
fetch the data from the mirrors network. That URL could be through Google App 
Engine or through some cheap hosting provider or possibly even sourceforge.

Website:

The website is currently all static. In future we will need language 
negotiation (which can be done statically), and possibly OS detection for the 
front page/download page (which can also be done statically although there may 
be reasons not to). Cheap/free hosting may not be fast enough when we get a 
slashdot, especially if we add some scripting but even if we don't (as they 
generally use apache so static content isn't necessarily dramatically faster). 
Google App Engine probably would be plenty fast, although we may need a paid 
account (which will probably only cost us when we are slashdotted).

Wiki:

We need to migrate the Wikka wiki to MediaWiki, because the latter is standard 
and we will be able to host it externally. E.g. sourceforge Hosted Apps allows 
for data import for MediaWiki.

Mailing lists:

There are a few free options e.g. sourceforge, berlios, also Google if we don't 
mind crappy I-am-not-an-Iranian click-throughs.

BUG TRACKER:

Basically it comes down to do we want to keep the existing bugs.

As I see it, bugs fall into 5 categories:
- Bugs abandoned by the end-user. These should be closed.
- Out of date bugs. These should *usually* be closed, but sometimes are in fact 
live bugs.
- Dev intelligence i.e. stuff people have said. If these are corroborated 
quickly they should be acted upon, else they should be closed.
- Live bugs.
- Feature requests, mostly by developers, often inter-linked with many other 
feature requests, linked to various end-user bug reports, and often with a lot 
of info on 1) consequences of implementation, and 2) how to implement.

If we keep the bug tracker:
- We need to find somewhere to host MANTIS. Probably we will have to pay for 
this.
- We need to keep it up to date ourselves, which is somewhat involved. It may 
not be as bad as nextgens implies though.
- Minimum immediate work.

If we don't keep the bug tracker:
- We can use any hosted bug tracker anywhere. E.g. Sourceforge Hosted Apps 
includes both Mantis and Trac. We will likely be able to avoid any fixed 
monthly payments.
- We can use any bug tracker: Mantis, Trac, etc. See below.
- We will need to do a spring clean: Keep the current bug tracker up for a 
while but read-only, *manually* migrate any important bugs and issues to the 
new tracker.
- This will be significant work.
- It will involve going over the bugs, dumping those which are out of date, 
abandoned etc, and rewriting those bugs and feature issues that are still 
valid. Trac's wiki functionality may be useful for this, although it loses the 
ability to link bugs formally.
- It may be a useful exercise in terms of prioritising and de-junking.

However, we have 20 weeks left of funding, so we have to ask whether spending a 
week de-junking is worth it?

An important related point: Relatively few end-users use the current bug 
tracker. It is on the Contribute menu on the website, but the main reason IMHO 
is it is not very newbie friendly. Uservoice is a reasonable solution for 
end-user feature suggestions and gauging public opinion, but because it does 
not force users to register their email addresses, it is worthless for solving 
individual reproducible bugs. It might reasonably be argued that we should have 
a separate issue tracker or forums system for end-user bug reports. Also, it 
may make sense for the developer-oriented bug tracker to be open source, 
whereas it matters less for the end-user tracker, because 1) end-users care 
less, and 2) long term stuff with detailed implementation notes is likely to be 
on the developer-oriented bug tracker.

If this line of reasoning is correct, we need to choose an end-user-oriented 
issue tracker or forums system (either way ideally gratis and hosted) to 
complement Uservoice. Suggestions?


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Update on getting rid of emu

[Ian Clarke]

On Fri, Oct 30, 2009 at 10:29 AM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 A disk seems to be failing. Fortunately we have two in RAID1. I have taken a 
 recent
 backup and should contact bytemark if we decide to keep it...

We should tell them anyway.  Doesn't cost us anything if they fix it,
and they'd probably want to know.

 The website is currently all static

I still think this is ridiculous, and pointlessly limiting.

 There are a few free options e.g. sourceforge, berlios, also Google if we 
 don't mind
 crappy I-am-not-an-Iranian click-throughs.

Google Groups has spam issues :-/

 BUG TRACKER:

I'm fond of Lighthouse, its got a simple clean flexible interface, and an API.

Ian.

-- 
Ian Clarke
CEO, Uprizer Labs
Email: i...@uprizer.com
Ph: +1 512 422 3588
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Update on getting rid of emu

[Evan Daniel]

On Fri, Oct 30, 2009 at 1:10 PM, Zero3 ze...@zerosplayground.dk wrote:
 Matthew Toseland wrote:
 We need to migrate the Wikka wiki to MediaWiki, because the latter is 
 standard and we will be able to host it externally. E.g. sourceforge Hosted 
 Apps allows for data import for MediaWiki.

 The wiki isn't really kept enough up-to-date, is it? I personally think
 the idea of something more integrated (like Trac) would be awesome.
 MediaWiki seems a bit over-do for Freenet?

I find MediaWiki far easier to use; it's overkill, but that doesn't
much matter.  If it's easier to use, it's easier to keep up to date.

Among other things, the current search sucks (making it hard to use
the wiki to answer a question), there aren't redirects (making it hard
to find the correct page), and we don't have categories and other
navigational aids.  Moving to MediaWiki would make it far easier for
users to get information from the wiki, and therefore a more
attractive place to keep documentation.

Evan Daniel
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Update on getting rid of emu

[Florent Daignière]

* Ian Clarke i...@locut.us [2009-10-30 10:57:41]:

 On Fri, Oct 30, 2009 at 10:29 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  A disk seems to be failing. Fortunately we have two in RAID1. I have taken 
  a recent
  backup and should contact bytemark if we decide to keep it...
 
 We should tell them anyway.  Doesn't cost us anything if they fix it,
 and they'd probably want to know.
 

It costs downtime and there is always a risk of loosing data/not being able to
reboot (raid1 is weak).

  The website is currently all static
 
 I still think this is ridiculous, and pointlessly limiting.
 
  There are a few free options e.g. sourceforge, berlios, also Google if we 
  don't mind
  crappy I-am-not-an-Iranian click-throughs.
 
 Google Groups has spam issues :-/
 
  BUG TRACKER:
 
 I'm fond of Lighthouse, its got a simple clean flexible interface, and an 
 API.
 

What about switching everything back to SF? The have both mantis (where
importing the current DB might be possible) and phpBB (forum for user
support; imho one of the most deployed).

NextGen$


signature.asc
Description: Digital signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Update on getting rid of emu

[Matthew Toseland]

On Friday 30 October 2009 19:13:18 Florent Daignière wrote:
 * Ian Clarke i...@locut.us [2009-10-30 10:57:41]:
 
  On Fri, Oct 30, 2009 at 10:29 AM, Matthew Toseland
  t...@amphibian.dyndns.org wrote:
   A disk seems to be failing. Fortunately we have two in RAID1. I have 
   taken a recent
   backup and should contact bytemark if we decide to keep it...
  
  We should tell them anyway.  Doesn't cost us anything if they fix it,
  and they'd probably want to know.
  
 
 It costs downtime and there is always a risk of loosing data/not being able to
 reboot (raid1 is weak).
 
   The website is currently all static
  
  I still think this is ridiculous, and pointlessly limiting.
  
   There are a few free options e.g. sourceforge, berlios, also Google if we 
   don't mind
   crappy I-am-not-an-Iranian click-throughs.
  
  Google Groups has spam issues :-/
  
   BUG TRACKER:
  
  I'm fond of Lighthouse, its got a simple clean flexible interface, and an 
  API.
 
 What about switching everything back to SF? The have both mantis (where
 importing the current DB might be possible) and phpBB (forum for user
 support; imho one of the most deployed).

Re data import on mantis, not likely afaics, they don't support data import now 
and when we asked they said ask for an enhancement.

Apart from that, you may have a point.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Update on getting rid of emu

[Matthew Toseland]

On Friday 30 October 2009 17:10:02 Zero3 wrote:
 Matthew Toseland wrote:
  We need to migrate the Wikka wiki to MediaWiki, because the latter is 
  standard and we will be able to host it externally. E.g. sourceforge Hosted 
  Apps allows for data import for MediaWiki.
 
 The wiki isn't really kept enough up-to-date, is it? I personally think 
 the idea of something more integrated (like Trac) would be awesome. 
 MediaWiki seems a bit over-do for Freenet?
 
  BUG TRACKER:
  - Dev intelligence i.e. stuff people have said. If these are corroborated 
  quickly they should be acted upon, else they should be closed.
 
 Something integrated (same user account, for starters) might help 
 motivating people to put this stuff in the wiki as well.

I agree it would help, but maybe not by very much? It would however make it a 
lot more readable.
 
  If we keep the bug tracker:
  - We need to find somewhere to host MANTIS. Probably we will have to pay 
  for this.
  - We need to keep it up to date ourselves, which is somewhat involved. It 
  may not be as bad as nextgens implies though.
  - Minimum immediate work.
 
 I personally think Mantis is *very* bad usability-wise. Trac, Launchpad, 
 and many other bug trackers are much easier to use. If we even have to 
 pay just to keep that thing running, I'd say find something else.

Is Trac easy enough *for end-users*?
 
  If we don't keep the bug tracker:
  - We can use any hosted bug tracker anywhere. E.g. Sourceforge Hosted Apps 
  includes both Mantis and Trac. We will likely be able to avoid any fixed 
  monthly payments.
  - We can use any bug tracker: Mantis, Trac, etc. See below.
  - We will need to do a spring clean: Keep the current bug tracker up for 
  a while but read-only, *manually* migrate any important bugs and issues to 
  the new tracker.
  - This will be significant work.
  - It will involve going over the bugs, dumping those which are out of date, 
  abandoned etc, and rewriting those bugs and feature issues that are still 
  valid. Trac's wiki functionality may be useful for this, although it loses 
  the ability to link bugs formally.
  - It may be a useful exercise in terms of prioritising and de-junking.
  
  However, we have 20 weeks left of funding, so we have to ask whether 
  spending a week de-junking is worth it?
 
 If it comes down to costing us £40-£80 per month... It quickly runs up. 
 I'm up for giving a hand and doing 5 'a day of the de-junking and 
 moving. I'm sure there are more people around willing to give a hand.

That would be helpful yeah. I can then subscribe to some sort of feed and 
review rather than having to do everything myself. :)
 
  An important related point: Relatively few end-users use the current bug 
  tracker. It is on the Contribute menu on the website, but the main reason 
  IMHO is it is not very newbie friendly. Uservoice is a reasonable solution 
  for end-user feature suggestions and gauging public opinion, but because it 
  does not force users to register their email addresses, it is worthless for 
  solving individual reproducible bugs. It might reasonably be argued that we 
  should have a separate issue tracker or forums system for end-user bug 
  reports. Also, it may make sense for the developer-oriented bug tracker to 
  be open source, whereas it matters less for the end-user tracker, because 
  1) end-users care less, and 2) long term stuff with detailed implementation 
  notes is likely to be on the developer-oriented bug tracker.
 
 Again, let's find something more user-friendly than Mantis :/
 
  If this line of reasoning is correct, we need to choose an 
  end-user-oriented issue tracker or forums system (either way ideally gratis 
  and hosted) to complement Uservoice. Suggestions?
 
 It would make sense to find a tracker that both users and devs can use. 
 Saves the overhead of moving things from e.g. a forums system to a bug 
 tracker.

Is it possible? Is Trac something that end users can use?


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Update on getting rid of emu

[xor]

On Friday 30 October 2009 16:29:57 Matthew Toseland wrote:

 If we don't keep the bug tracker:

 - We will need to do a spring clean: Keep the current bug tracker up for
 a while but read-only, *manually* migrate any important bugs and issues to
 the new tracker. - This will be significant work.
 - It will involve going over the bugs, dumping those which are out of date,
 abandoned etc, and rewriting those bugs and feature issues that are still
 valid. Trac's wiki functionality may be useful for this, although it loses
 the ability to link bugs formally. - It may be a useful exercise in terms
 of prioritising and de-junking.


Migrating it in read-only mode is insane because there would be no sane way 
for monitoring the migration progress: Which bugs have been reviewed  
migrated?

Instead, we should clean up mantis itself until ALL bugs there are suitable 
for migration and then migrate. 

Besides, as I've said numerous times, we MUST migrate all bugs because from a 
software engineering point of view the content of a bugtracker is the most 
valuable data of a project besides the soure code and the documentation.
Deleting any of it without reviewing it is just unprofessional, so we must 
make very sure that no issues in mantis get lost.

BTW: When did someone do the last backup of it's database? Does emu have a 
backup system?


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Update on getting rid of emu

[Juiceman]

On Fri, Oct 30, 2009 at 3:16 PM, Matthew Toseland t...@amphibian.dyndns.org
 wrote:

 On Friday 30 October 2009 19:13:18 Florent Daignière wrote:
  * Ian Clarke i...@locut.us [2009-10-30 10:57:41]:
 
   On Fri, Oct 30, 2009 at 10:29 AM, Matthew Toseland
   t...@amphibian.dyndns.org wrote:
A disk seems to be failing. Fortunately we have two in RAID1. I have
 taken a recent
backup and should contact bytemark if we decide to keep it...
  
   We should tell them anyway.  Doesn't cost us anything if they fix it,
   and they'd probably want to know.
  
 
  It costs downtime and there is always a risk of loosing data/not being
 able to
  reboot (raid1 is weak).
 
The website is currently all static
  
   I still think this is ridiculous, and pointlessly limiting.
  
There are a few free options e.g. sourceforge, berlios, also Google
 if we don't mind
crappy I-am-not-an-Iranian click-throughs.
  
   Google Groups has spam issues :-/
  
BUG TRACKER:
  
   I'm fond of Lighthouse, its got a simple clean flexible interface,
 and an API.
 
  What about switching everything back to SF? The have both mantis (where
  importing the current DB might be possible) and phpBB (forum for user
  support; imho one of the most deployed).

 Re data import on mantis, not likely afaics, they don't support data import
 now and when we asked they said ask for an enhancement.

 Apart from that, you may have a point.


Surely there are ways of importing a csv text file at least?  We could get
that out of Mantis, no?
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Current security measures may be harming performance; better measures may help

[Matthew Toseland]

Currently, requests are always routed the same way, but at high HTL we do not 
cache either replies to requests or incoming inserts.

Specifically, at HTL 18 and 17 we do not cache returned data from requests 
(though we do check the datastore), and at HTL 18, 17 and 16 we do not cache 
data from inserts. On average we spend 2 hops at HTL 18, including the 
originator, so on average for an insert it is 4 hops before we cache, with a 
minimum of 3 (or is it a minimum of 2? afaics we start at htl 18 and then we 
may decrement it when sending to the next hop, so a minimum of 3).

Decrement at HTL 18 is probabilistic, with a 50% probability.

Simulations suggest that the ideal node is likely found around HTL 14 to 15. 
So a significant proportion of requests and inserts will go past it while still 
in the no caching phase. This may partly explain poor data retention, which 
appears to affect some proportion of keys much more than the others.

Hence we might get better data retention if we e.g. random routed while in the 
no-cache phase.

But here is another reason for random routing while in the no-cache phase:

Lets assume that we only care about remote attackers. Generally they are much 
more scary. So we are talking about the mobile attacker source tracing attack. 
This means that a bad guy is a long way away, and he gets a few requests by 
chance which were part of the same splitfile insert or request originated by 
you. He is able to determine that they are part of the same, interesting, 
splitfile. For each request, he knows 1) that it was routed to him, and 2) its 
target location. He can thus determine where on the keyspace the request could 
have come from. This is a big vague due to backoff etc, but he can nonetheless 
identify an area where the originator is most likely present, starting at his 
location and extending in one direction or the other. In fact, he can identify 
the opposite end of it as the most likely location of the originator. So he 
then tries to get peers closer to this location, by announcement, path folding, 
changing his own location etc. If he is right, he will then get requests from 
this source much more quickly. And so he can keep on moving until he reaches 
the originator. It has been suggested that we could mark requests so that they 
will not be routed to new connections - the problem is this doesn't work for 
long-lived requests e.g. big inserts.

The number of samples the attacker gets is proportional to the number of hops 
from the originator to the ideal node, on average, since samples after the 
ideal are much less informative. It is also proportional to the number of 
requests sent, and inversely to the size of the network.

Random routing while the HTL is high, not to any specific location but to a 
random peer at each hop (subject to e.g. backoff), would make the pre-ideal 
samples much less useful, because they will each have effectively started at a 
random node - not a truly random node, especially if we route randomly at each 
hop, we won't have had enough hops for it to be a random node across the whole 
keyspace, but it will still mean the picture is much more vague, and the 
attacker will need a lot more samples. The post-ideal sample remains useless. 
If the request reaches the attacker while it is still in the random routing 
phase, this provides a useful sample to the attacker, but likely much less 
useful than in the routed stage.

So, just maybe, we could improve data persistence (if not necessarily overall 
performance), and maintain the current no-cache-at-high-htl, and improve 
security, by random routing as well as not caching while HTL is high. Worth 
simulating perhaps?

The next obvious solution is some form of bundling: Even if the bundle is not 
encrypted, routing a large bunch of requests together for some distance gives 
one sample instead of many. Short-lived bundles have the disadvantage that 
there are many of them so the attacker gets more samples if they happen to 
cross his path. However, we could do the don't-route-to-newbies trick with 
short-lived bundles, using a fixed path for the bundle's lifetime. 10 bundles 
each renewed once an hour beats hundreds of requests per hour! Long-lived 
bundles would probably have to automatically move to new nodes, and therefore 
could perhaps be traced back to source eventually - if the attacker managed to 
hook one, or more likely trace a stream of requests back to one.

Bundling is a lot more work, a lot more tuning, but of course more secure. It 
would replace the current no cache for a few hops, and would still check the 
local datastore.

Encrypted tunnels are a further evolution of bundling: We send out various 
randomly routed anchors, which rendezvous to create a tunnel, which is a 
short encrypted (using a shared secret scheme) path to a random start node. 
This has most of the same issues as bundling, although it doesn't check the 
local datastore, and it provides a reasonable degree of 

Re: [freenet-dev] Update on getting rid of emu

[Matthew Toseland]

On Friday 30 October 2009 19:21:51 xor wrote:
 On Friday 30 October 2009 16:29:57 Matthew Toseland wrote:
 
  If we don't keep the bug tracker:
 
  - We will need to do a spring clean: Keep the current bug tracker up for
  a while but read-only, *manually* migrate any important bugs and issues to
  the new tracker. - This will be significant work.
  - It will involve going over the bugs, dumping those which are out of date,
  abandoned etc, and rewriting those bugs and feature issues that are still
  valid. Trac's wiki functionality may be useful for this, although it loses
  the ability to link bugs formally. - It may be a useful exercise in terms
  of prioritising and de-junking.
 
 
 Migrating it in read-only mode is insane because there would be no sane way 
 for monitoring the migration progress: Which bugs have been reviewed  
 migrated?

Fair point. Devs should have the ability to close bugs after they have been 
migrated.
 
 Instead, we should clean up mantis itself until ALL bugs there are suitable 
 for migration and then migrate. 
 
 Besides, as I've said numerous times, we MUST migrate all bugs because from a 
 software engineering point of view the content of a bugtracker is the most 
 valuable data of a project besides the soure code and the documentation.
 Deleting any of it without reviewing it is just unprofessional, so we must 
 make very sure that no issues in mantis get lost.
 
 BTW: When did someone do the last backup of it's database? Does emu have a 
 backup system?


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl