[freenet-dev] Usability issues with new filter infrastructure was Re: New filter code: stuff that must be fixed before 1251

[Matthew Toseland]

On Friday 11 June 2010 23:58:31 Matthew Toseland wrote:
> On Friday 11 June 2010 14:43:09 Matthew Toseland wrote:
> > BEFORE RELEASING 1251, MUST FIX:
> > - Internal error on persistent download for mp3 with the check filter flag 
> > enabled. This should be an unfilterable content error.
> 
> My fault, fixed.
> 
> > - WoT
> > - Freetalk
> > - Spider
> > - XMLSpider
> > - FlogHelper
> 
> All done, thanks. Jars will be posted before 1251 is released.
> 
> > - ContentFilter charset detection code MUST use readFully, or a loop 
> > achieving similar goals. read() doesn't read the full buffer.
> 
> This is STILL not fixed. We use readFully() but we only read up to 
> available(). We should read the full buffer size specified by the 
> CharsetExtractor, up to the limit of the *SIZE OF THE BUCKET*. available() is 
> not very helpful here IMHO.

More issues that it would be good to fix before releasing a build with this 
code. This is all usability stuff which IMHO will make a difference to 
end-users' frustration levels (of course, filters for popular types, and 
straight-through decompression-to-filtering, will also help with usability; if 
any of these are big jobs we can postpone them but we must not forget them):

1) Error: Content sanitization failed: Unknown and potentially dangerous 
content type: application/octet-stream

(The title of the warning page). This is redundant, too many words. IMHO we 
should just show Error: Unknown and ...

2) When downloading a file to disk, with an unrecognised MIME type, with 
filtering enabled, it now fails the download with a message explaining that it 
is not supported; this message is visible on the queue page. However, it is a 
*really* *long* message. All of the other failures are short messages like "New 
URI" or "Invalid URI: Invalid crypto algorithm". Okay, those aren't terribly 
helpful, but IMHO it is useful for it to be short, maybe with a link to a 
longer explanation.

I'm not sure exactly how to handle this. IIRC GetFailed (the FCP mechanism 
which underlies the queue page) has an ExtraDescription field we could use for 
the long description. Then we just need a short description for the error 
itself? I think we already use this, assembling the error from the short 
description and then the long description after a colon ... this probably 
happens in FetchException ... hmmm. Worth thinking about anyway, if you can 
come up with a reasonably quick solution. I think we have short description and 
long description for specific error codes, this may be relevant?

Of course the real solution is to write filters for the most popular content 
types e.g. this is an MP3. :)

3) There should be some way to retry the download with filtering turned off. 
Probably this should take the form of another button in the first column, or a 
check box ([ ] Turn off filter). In future when the queue page has been cleaned 
up, the first column will be just checkboxes, and the actions will be at the 
bottom, at which point we can have a checkbox at the bottom; but that is way 
out of scope for your Summer of Code project and I'm hoping Bombe will get 
around to it eventually. But the lack of any obvious ability to retry with 
filtering disabled, given that it was caused by an unsupported MIME type, is 
IMHO a serious usability issue which will cause a lot of end-user frustration.

4) The filter flag is enabled by default for download to disk from within 
fproxy. Even if we know ahead of time that the MIME type is one we don't 
support, and even if security levels are all set to LOW. This will cause people 
to have the above frustration.

IMHO:
- If the page the download button is on is itself a MIME type warning, the 
filtered checkbox should default to OFF. IIRC if we have a nontrivial MIME type 
it cannot change later on because ClientMetadata.mergeNoOverwrite() does not 
allow us to change a nontrivial MIME type; so as long as the MIME type is not 
empty and not application/octet-stream (which is the same thing iirc), if we 
show the user the warning and they accept it, that is the MIME type that we 
will have at the end of the fetch. However, USKs may throw a spanner in the 
works here... so it may still be useful to be able to record that we've shown 
the user the warning for type T and not to fail *IF* the output is of that type.
- If the network security level and physical security level are set to LOW, the 
filtered checkbox should default to OFF.
- Ideally, if we know both the size and the MIME type, we should combine the 
two warnings into a single page. That's long term, file a bug and forget about 
it. A shorter term solution might be to show the MIME type warning before we 
show the size warning, once we are fairly sure we know the MIME type (i.e. we 
detect a non-null non-application/octet-stream type in the ExpectedMIME event 
which fproxy picks up).
> > 
> > Minor stuff:
> > 
> > 15b6283e017a26dec96f35820028b84992b5260f
> > - should pass the overridden mime type 

[freenet-dev] Usability issues with new filter infrastructure was Re: New filter code: stuff that must be fixed before 1251

[Matthew Toseland]

On Friday 11 June 2010 23:58:31 Matthew Toseland wrote:
 On Friday 11 June 2010 14:43:09 Matthew Toseland wrote:
  BEFORE RELEASING 1251, MUST FIX:
  - Internal error on persistent download for mp3 with the check filter flag 
  enabled. This should be an unfilterable content error.
 
 My fault, fixed.
 
  - WoT
  - Freetalk
  - Spider
  - XMLSpider
  - FlogHelper
 
 All done, thanks. Jars will be posted before 1251 is released.
 
  - ContentFilter charset detection code MUST use readFully, or a loop 
  achieving similar goals. read() doesn't read the full buffer.
 
 This is STILL not fixed. We use readFully() but we only read up to 
 available(). We should read the full buffer size specified by the 
 CharsetExtractor, up to the limit of the *SIZE OF THE BUCKET*. available() is 
 not very helpful here IMHO.

More issues that it would be good to fix before releasing a build with this 
code. This is all usability stuff which IMHO will make a difference to 
end-users' frustration levels (of course, filters for popular types, and 
straight-through decompression-to-filtering, will also help with usability; if 
any of these are big jobs we can postpone them but we must not forget them):

1) Error: Content sanitization failed: Unknown and potentially dangerous 
content type: application/octet-stream

(The title of the warning page). This is redundant, too many words. IMHO we 
should just show Error: Unknown and ...

2) When downloading a file to disk, with an unrecognised MIME type, with 
filtering enabled, it now fails the download with a message explaining that it 
is not supported; this message is visible on the queue page. However, it is a 
*really* *long* message. All of the other failures are short messages like New 
URI or Invalid URI: Invalid crypto algorithm. Okay, those aren't terribly 
helpful, but IMHO it is useful for it to be short, maybe with a link to a 
longer explanation.

I'm not sure exactly how to handle this. IIRC GetFailed (the FCP mechanism 
which underlies the queue page) has an ExtraDescription field we could use for 
the long description. Then we just need a short description for the error 
itself? I think we already use this, assembling the error from the short 
description and then the long description after a colon ... this probably 
happens in FetchException ... hmmm. Worth thinking about anyway, if you can 
come up with a reasonably quick solution. I think we have short description and 
long description for specific error codes, this may be relevant?

Of course the real solution is to write filters for the most popular content 
types e.g. this is an MP3. :)

3) There should be some way to retry the download with filtering turned off. 
Probably this should take the form of another button in the first column, or a 
check box ([ ] Turn off filter). In future when the queue page has been cleaned 
up, the first column will be just checkboxes, and the actions will be at the 
bottom, at which point we can have a checkbox at the bottom; but that is way 
out of scope for your Summer of Code project and I'm hoping Bombe will get 
around to it eventually. But the lack of any obvious ability to retry with 
filtering disabled, given that it was caused by an unsupported MIME type, is 
IMHO a serious usability issue which will cause a lot of end-user frustration.

4) The filter flag is enabled by default for download to disk from within 
fproxy. Even if we know ahead of time that the MIME type is one we don't 
support, and even if security levels are all set to LOW. This will cause people 
to have the above frustration.

IMHO:
- If the page the download button is on is itself a MIME type warning, the 
filtered checkbox should default to OFF. IIRC if we have a nontrivial MIME type 
it cannot change later on because ClientMetadata.mergeNoOverwrite() does not 
allow us to change a nontrivial MIME type; so as long as the MIME type is not 
empty and not application/octet-stream (which is the same thing iirc), if we 
show the user the warning and they accept it, that is the MIME type that we 
will have at the end of the fetch. However, USKs may throw a spanner in the 
works here... so it may still be useful to be able to record that we've shown 
the user the warning for type T and not to fail *IF* the output is of that type.
- If the network security level and physical security level are set to LOW, the 
filtered checkbox should default to OFF.
- Ideally, if we know both the size and the MIME type, we should combine the 
two warnings into a single page. That's long term, file a bug and forget about 
it. A shorter term solution might be to show the MIME type warning before we 
show the size warning, once we are fairly sure we know the MIME type (i.e. we 
detect a non-null non-application/octet-stream type in the ExpectedMIME event 
which fproxy picks up).
  
  Minor stuff:
  
  15b6283e017a26dec96f35820028b84992b5260f
  - should pass the overridden mime type along in places where we generate 
  expected 

[freenet-dev] Usability issues

[Matthew Toseland]

On Tuesday 06 May 2008 22:57, Ed Tomlinson wrote:
> On May 6, 2008, Matthew Toseland wrote:
> > On Tuesday 06 May 2008 15:50, Robert Hailey wrote:
> > > 
> > > On May 5, 2008, at 5:49 PM, Matthew Toseland wrote:
> > > > [...]
> > > > - Freenet is slow, especially in this case where it had an  
> > > > untweaked
> > > > browser. After a long delay, two sites loaded, after a longer delay,  
> > > > several
> > > > sites loaded at once; this strongly suggests imho that the problem  
> > > > is caused
> > > > by not finding the browser. User reports significant improvement in  
> > > > speed
> > > > after a few minutes.
> > > > - Connection limit is definitely a problem: many images which should  
> > > > be in the
> > > > same container not loading; same for the Potentially Dangerous Content
> > > > warning.
> > > 
> > > Surely the browser isn't the one which understands freenet containers.  
> > > Are we coalescing requests for the same container internally?
> > 
> > Not at the client level. There is room for further improvement here. But 
we do 
> > do some coalescing: we won't have multiple local requests in flight for 
the 
> > same key, for example.
> > 
> > Most of the above issues were caused by the browser not being found, this 
has 
> > been fixed in 90% of cases by the changes I made to browse.cmd.
> > 
> > However the fact remains that a newbie node is still rather 
disappointingly 
> > slow, even with a tweaked browser.
> 
> If the user is in the habit of stoping and starting the node and has a queue 
with a
> large number of blocks, browsing will be very slow (or impossible) until the 
rebuild
> if the queue completes.  See bug 2334.

True, that's not the case here: really new nodes are slow, bootstrapping is 
much slower than it ought to be. IMHO this is because of seednodes problems, 
which hopefully I will be able to fix soon.
> 
> Ed
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

[freenet-dev] Usability issues

[Ed Tomlinson]

On May 6, 2008, Matthew Toseland wrote:
> On Tuesday 06 May 2008 15:50, Robert Hailey wrote:
> > 
> > On May 5, 2008, at 5:49 PM, Matthew Toseland wrote:
> > > [...]
> > > - Freenet is slow, especially in this case where it had an  
> > > untweaked
> > > browser. After a long delay, two sites loaded, after a longer delay,  
> > > several
> > > sites loaded at once; this strongly suggests imho that the problem  
> > > is caused
> > > by not finding the browser. User reports significant improvement in  
> > > speed
> > > after a few minutes.
> > > - Connection limit is definitely a problem: many images which should  
> > > be in the
> > > same container not loading; same for the Potentially Dangerous Content
> > > warning.
> > 
> > Surely the browser isn't the one which understands freenet containers.  
> > Are we coalescing requests for the same container internally?
> 
> Not at the client level. There is room for further improvement here. But we 
> do 
> do some coalescing: we won't have multiple local requests in flight for the 
> same key, for example.
> 
> Most of the above issues were caused by the browser not being found, this has 
> been fixed in 90% of cases by the changes I made to browse.cmd.
> 
> However the fact remains that a newbie node is still rather disappointingly 
> slow, even with a tweaked browser.

If the user is in the habit of stoping and starting the node and has a queue 
with a
large number of blocks, browsing will be very slow (or impossible) until the 
rebuild
if the queue completes.  See bug 2334.

Ed

[freenet-dev] Usability issues

[Matthew Toseland]

On Tuesday 06 May 2008 15:50, Robert Hailey wrote:
> 
> On May 5, 2008, at 5:49 PM, Matthew Toseland wrote:
> > [...]
> > - Freenet is slow, especially in this case where it had an  
> > untweaked
> > browser. After a long delay, two sites loaded, after a longer delay,  
> > several
> > sites loaded at once; this strongly suggests imho that the problem  
> > is caused
> > by not finding the browser. User reports significant improvement in  
> > speed
> > after a few minutes.
> > - Connection limit is definitely a problem: many images which should  
> > be in the
> > same container not loading; same for the Potentially Dangerous Content
> > warning.
> 
> Surely the browser isn't the one which understands freenet containers.  
> Are we coalescing requests for the same container internally?

Not at the client level. There is room for further improvement here. But we do 
do some coalescing: we won't have multiple local requests in flight for the 
same key, for example.

Most of the above issues were caused by the browser not being found, this has 
been fixed in 90% of cases by the changes I made to browse.cmd.

However the fact remains that a newbie node is still rather disappointingly 
slow, even with a tweaked browser.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

[freenet-dev] Usability issues

[Robert Hailey]

On May 5, 2008, at 5:49 PM, Matthew Toseland wrote:
> [...]
> - Freenet is slow, especially in this case where it had an  
> untweaked
> browser. After a long delay, two sites loaded, after a longer delay,  
> several
> sites loaded at once; this strongly suggests imho that the problem  
> is caused
> by not finding the browser. User reports significant improvement in  
> speed
> after a few minutes.
> - Connection limit is definitely a problem: many images which should  
> be in the
> same container not loading; same for the Potentially Dangerous Content
> warning.

Surely the browser isn't the one which understands freenet containers.  
Are we coalescing requests for the same container internally?

--
Robert Hailey

[freenet-dev] Usability issues

[Matthew Toseland]

Okay, with further investigation ...

The user in question appears to have installed a clean copy of XP and is 
reusing the already-installed Firefox. Therefore App Paths in the registry is 
empty, and doesn't include firefox.exe. Therefore, we are not detecting 
Firefox.

However, this doesn't explain why we open a command line called welcome.html 
instead of IE/Firefox (there was a copy of FF running...) pointing to the 
welcome.html ... Further testing shows start http://127.0.0.1: launches a 
new firefox window pointing to the freenet homepage.

We should check popular locations for Firefox to deal with this eventuality. 
Obviously this won't work on some languages, but if it works on some it will 
significantly increase our chances of success (in an admittedly pathological 
but probably somewhat common scenario).

It is a pain that our fallbacks don't work, and I have no idea how to fix it, 
it looks like a windows bug, unless anyone has an alternative explanation. :(

On Monday 05 May 2008 23:49, Matthew Toseland wrote:
> Did another usability test. Problems:
> - The installer is signed by nextgens. We should change this. But this 
wasn't 
> reported by the user, it's just my view.
> - Freenet not only didn't find Firefox (2.0.0.14, english, custom theme), it 
> didn't find any web browser of any kind: it opened a command line with the 
> title "welcome.html" !! I told the user to go back to the download page, and 
> he followed the instruction to open http://127.0.0.1:/wizard/ , and from 
> then it was somewhat smooth...
> http://amphibian.dyndns.org/screenshot-freenet-command-line.bmp
> http://amphibian.dyndns.org/screenshot-freenet-bratnet.bmp
> - Freenet is slow, especially in this case where it had an untweaked 
> browser. After a long delay, two sites loaded, after a longer delay, several 
> sites loaded at once; this strongly suggests imho that the problem is caused 
> by not finding the browser. User reports significant improvement in speed 
> after a few minutes.
> - Connection limit is definitely a problem: many images which should be in 
the 
> same container not loading; same for the Potentially Dangerous Content 
> warning.
> - Freenet Activelink Index causing problems... in an untweaked browser, this 
> site is an utter disaster. Even in a tweaked browser it will DoS the rest 
for 
> a while. Maybe we shouldn't bundle it?
> 
> Solutions?
> 
> 
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

[freenet-dev] Usability issues

[Matthew Toseland]

Did another usability test. Problems:
- The installer is signed by nextgens. We should change this. But this wasn't 
reported by the user, it's just my view.
- Freenet not only didn't find Firefox (2.0.0.14, english, custom theme), it 
didn't find any web browser of any kind: it opened a command line with the 
title "welcome.html" !! I told the user to go back to the download page, and 
he followed the instruction to open http://127.0.0.1:/wizard/ , and from 
then it was somewhat smooth...
http://amphibian.dyndns.org/screenshot-freenet-command-line.bmp
http://amphibian.dyndns.org/screenshot-freenet-bratnet.bmp
- Freenet is slow, especially in this case where it had an untweaked 
browser. After a long delay, two sites loaded, after a longer delay, several 
sites loaded at once; this strongly suggests imho that the problem is caused 
by not finding the browser. User reports significant improvement in speed 
after a few minutes.
- Connection limit is definitely a problem: many images which should be in the 
same container not loading; same for the Potentially Dangerous Content 
warning.
- Freenet Activelink Index causing problems... in an untweaked browser, this 
site is an utter disaster. Even in a tweaked browser it will DoS the rest for 
a while. Maybe we shouldn't bundle it?

Solutions?

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

Re: [freenet-dev] Usability issues

[Robert Hailey]

On May 5, 2008, at 5:49 PM, Matthew Toseland wrote:
 [...]
 - Freenet is slow, especially in this case where it had an  
 untweaked
 browser. After a long delay, two sites loaded, after a longer delay,  
 several
 sites loaded at once; this strongly suggests imho that the problem  
 is caused
 by not finding the browser. User reports significant improvement in  
 speed
 after a few minutes.
 - Connection limit is definitely a problem: many images which should  
 be in the
 same container not loading; same for the Potentially Dangerous Content
 warning.

Surely the browser isn't the one which understands freenet containers.  
Are we coalescing requests for the same container internally?

--
Robert Hailey

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Usability issues

[Matthew Toseland]

On Tuesday 06 May 2008 15:50, Robert Hailey wrote:
 
 On May 5, 2008, at 5:49 PM, Matthew Toseland wrote:
  [...]
  - Freenet is slow, especially in this case where it had an  
  untweaked
  browser. After a long delay, two sites loaded, after a longer delay,  
  several
  sites loaded at once; this strongly suggests imho that the problem  
  is caused
  by not finding the browser. User reports significant improvement in  
  speed
  after a few minutes.
  - Connection limit is definitely a problem: many images which should  
  be in the
  same container not loading; same for the Potentially Dangerous Content
  warning.
 
 Surely the browser isn't the one which understands freenet containers.  
 Are we coalescing requests for the same container internally?

Not at the client level. There is room for further improvement here. But we do 
do some coalescing: we won't have multiple local requests in flight for the 
same key, for example.

Most of the above issues were caused by the browser not being found, this has 
been fixed in 90% of cases by the changes I made to browse.cmd.

However the fact remains that a newbie node is still rather disappointingly 
slow, even with a tweaked browser.


pgpD2Eyg5SKbf.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Usability issues

[Ed Tomlinson]

On May 6, 2008, Matthew Toseland wrote:
 On Tuesday 06 May 2008 15:50, Robert Hailey wrote:
  
  On May 5, 2008, at 5:49 PM, Matthew Toseland wrote:
   [...]
   - Freenet is slow, especially in this case where it had an  
   untweaked
   browser. After a long delay, two sites loaded, after a longer delay,  
   several
   sites loaded at once; this strongly suggests imho that the problem  
   is caused
   by not finding the browser. User reports significant improvement in  
   speed
   after a few minutes.
   - Connection limit is definitely a problem: many images which should  
   be in the
   same container not loading; same for the Potentially Dangerous Content
   warning.
  
  Surely the browser isn't the one which understands freenet containers.  
  Are we coalescing requests for the same container internally?
 
 Not at the client level. There is room for further improvement here. But we 
 do 
 do some coalescing: we won't have multiple local requests in flight for the 
 same key, for example.
 
 Most of the above issues were caused by the browser not being found, this has 
 been fixed in 90% of cases by the changes I made to browse.cmd.
 
 However the fact remains that a newbie node is still rather disappointingly 
 slow, even with a tweaked browser.

If the user is in the habit of stoping and starting the node and has a queue 
with a
large number of blocks, browsing will be very slow (or impossible) until the 
rebuild
if the queue completes.  See bug 2334.

Ed

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Usability issues

[Matthew Toseland]

On Tuesday 06 May 2008 22:57, Ed Tomlinson wrote:
 On May 6, 2008, Matthew Toseland wrote:
  On Tuesday 06 May 2008 15:50, Robert Hailey wrote:
   
   On May 5, 2008, at 5:49 PM, Matthew Toseland wrote:
[...]
- Freenet is slow, especially in this case where it had an  
untweaked
browser. After a long delay, two sites loaded, after a longer delay,  
several
sites loaded at once; this strongly suggests imho that the problem  
is caused
by not finding the browser. User reports significant improvement in  
speed
after a few minutes.
- Connection limit is definitely a problem: many images which should  
be in the
same container not loading; same for the Potentially Dangerous Content
warning.
   
   Surely the browser isn't the one which understands freenet containers.  
   Are we coalescing requests for the same container internally?
  
  Not at the client level. There is room for further improvement here. But 
we do 
  do some coalescing: we won't have multiple local requests in flight for 
the 
  same key, for example.
  
  Most of the above issues were caused by the browser not being found, this 
has 
  been fixed in 90% of cases by the changes I made to browse.cmd.
  
  However the fact remains that a newbie node is still rather 
disappointingly 
  slow, even with a tweaked browser.
 
 If the user is in the habit of stoping and starting the node and has a queue 
with a
 large number of blocks, browsing will be very slow (or impossible) until the 
rebuild
 if the queue completes.  See bug 2334.

True, that's not the case here: really new nodes are slow, bootstrapping is 
much slower than it ought to be. IMHO this is because of seednodes problems, 
which hopefully I will be able to fix soon.
 
 Ed


pgpWFRHyMBU2O.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Usability issues

[Matthew Toseland]

Did another usability test. Problems:
- The installer is signed by nextgens. We should change this. But this wasn't 
reported by the user, it's just my view.
- Freenet not only didn't find Firefox (2.0.0.14, english, custom theme), it 
didn't find any web browser of any kind: it opened a command line with the 
title welcome.html !! I told the user to go back to the download page, and 
he followed the instruction to open http://127.0.0.1:/wizard/ , and from 
then it was somewhat smooth...
http://amphibian.dyndns.org/screenshot-freenet-command-line.bmp
http://amphibian.dyndns.org/screenshot-freenet-bratnet.bmp
- Freenet is slow, especially in this case where it had an untweaked 
browser. After a long delay, two sites loaded, after a longer delay, several 
sites loaded at once; this strongly suggests imho that the problem is caused 
by not finding the browser. User reports significant improvement in speed 
after a few minutes.
- Connection limit is definitely a problem: many images which should be in the 
same container not loading; same for the Potentially Dangerous Content 
warning.
- Freenet Activelink Index causing problems... in an untweaked browser, this 
site is an utter disaster. Even in a tweaked browser it will DoS the rest for 
a while. Maybe we shouldn't bundle it?

Solutions?



pgpvWSLT2VZFw.pgp
Description: PGP signature
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl