[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Dennis Nezic]

On Sun, 3 May 2009 01:46:37 -0400, Juiceman wrote:
> On Sat, Apr 25, 2009 at 2:23 PM, Matthew Toseland
>  wrote:
> > On Friday 24 April 2009 17:46:09 freenetwork at web.de wrote:
> >> 1) CHK-keys are already long enough
> >
> > Long enough to be a PITA if they are longer? Or long enough to be
> > functional? I dispute the latter.
> >
> >> 2) why add something that tries to fix something broken (routing?)
> >> or contradicts the concept (caching of keys around the key
> >> location; unused content gets dropped)
> >
> > Routing is not broken. Data persistence is broken. The feedback I
> > have had is that frequently the problem with fetching a file is the
> > top block, which currently is not redundant. For example it might
> > take 3 weeks at 0% to find the top block and then make relatively
> > rapid progress after that. This is not your experience?
> >>
> >> if a) unwanted content is supposed to be dropped from the network
> >> to make space for fresh stuff and b) the top key is *needed* for
> >> *every* request of a ((larger) split-) file, how can the top key
> >> possibly fall off the network?
> >
> > If the key is not requested, it could well fall off the network,
> > *even though the rest of the file is still retrievable*. Because
> > the rest of the file is redundant and the top block is not. We have
> > very large datastores and generally very low input/output
> > bandwidth, so I would expect data to persist for a long time in the
> > current Freenet. Data that has not been recently requested will
> > only be in the stores, and not in the caches. But since routing
> > appears to work (and there is every theoretical reason to think it
> > does), there is a good chance of finding the data - IF the 3 nodes
> > which stored it to their datastores are online when you fetch and
> > there aren't any problems contacting them (e.g. on darknet they
> > might have swapped).
> >>
> >> IMHO I think this is making extra effort and adding
> >> YetJustAnotherKeyType for CreateAWorkaroundForSomethingDifferent
> >> for something that needs to be addressed elsewhere
> >
> > I don't see much evidence there is a fundamental problem with
> > routing in 0.7 on opennet. Do you have any evidence for this? I
> > would be very interested in any evidence that this is a routing
> > rather than a redundancy problem.
> >
> > Nodes' graphs usually show a fairly strong request specialisation,
> > the main symptom is that 2-3 weeks after inserting data, it is not
> > retrievable, if nobody has fetched it. And a lot of the time, that
> > specifically relates to the top block. This proposal would solve
> > the problem.
> >
> 
> If we have, say, 3 top CHK's or RHK (Redundant Hash Key is my vote)
> and we are able to find one of them, we should be able to recreate and
> reinsert the other two, right?

Yup. I'm pretty sure the redundant CHKs would be simple modifications to
the hashing math -- probably with specified constants added to it.

I'm still not crazy about introducing a new key though. Would it really
be better than simply sending more copies of the top block during the
initial insert (6 or 9 instead of the original 3?) and reinserting it
again a few times when another user downloads it?

> To prevent these reinserts giving away that we retrieved the file, we
> just need a queue of keys to be inserted (I assume we have this for
> FEC healing?).  Grab a random key from the queue each time we have a
> slot to insert.

Well, if anyone had that kind of access to your node (to the internal
queues, or even the datastore to a lesser extent), it would already be
too late, no?

> I would suggest implementing this before going after the shared bloom
> filters idea.  From a usability standpoint this would solve some of
> the glaring problems like unfetchable pages, pictures and files.

Agreed.

Re: [freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Dennis Nezic]

On Sun, 3 May 2009 01:46:37 -0400, Juiceman wrote:
 On Sat, Apr 25, 2009 at 2:23 PM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  On Friday 24 April 2009 17:46:09 freenetw...@web.de wrote:
  1) CHK-keys are already long enough
 
  Long enough to be a PITA if they are longer? Or long enough to be
  functional? I dispute the latter.
 
  2) why add something that tries to fix something broken (routing?)
  or contradicts the concept (caching of keys around the key
  location; unused content gets dropped)
 
  Routing is not broken. Data persistence is broken. The feedback I
  have had is that frequently the problem with fetching a file is the
  top block, which currently is not redundant. For example it might
  take 3 weeks at 0% to find the top block and then make relatively
  rapid progress after that. This is not your experience?
 
  if a) unwanted content is supposed to be dropped from the network
  to make space for fresh stuff and b) the top key is *needed* for
  *every* request of a ((larger) split-) file, how can the top key
  possibly fall off the network?
 
  If the key is not requested, it could well fall off the network,
  *even though the rest of the file is still retrievable*. Because
  the rest of the file is redundant and the top block is not. We have
  very large datastores and generally very low input/output
  bandwidth, so I would expect data to persist for a long time in the
  current Freenet. Data that has not been recently requested will
  only be in the stores, and not in the caches. But since routing
  appears to work (and there is every theoretical reason to think it
  does), there is a good chance of finding the data - IF the 3 nodes
  which stored it to their datastores are online when you fetch and
  there aren't any problems contacting them (e.g. on darknet they
  might have swapped).
 
  IMHO I think this is making extra effort and adding
  YetJustAnotherKeyType for CreateAWorkaroundForSomethingDifferent
  for something that needs to be addressed elsewhere
 
  I don't see much evidence there is a fundamental problem with
  routing in 0.7 on opennet. Do you have any evidence for this? I
  would be very interested in any evidence that this is a routing
  rather than a redundancy problem.
 
  Nodes' graphs usually show a fairly strong request specialisation,
  the main symptom is that 2-3 weeks after inserting data, it is not
  retrievable, if nobody has fetched it. And a lot of the time, that
  specifically relates to the top block. This proposal would solve
  the problem.
 
 
 If we have, say, 3 top CHK's or RHK (Redundant Hash Key is my vote)
 and we are able to find one of them, we should be able to recreate and
 reinsert the other two, right?

Yup. I'm pretty sure the redundant CHKs would be simple modifications to
the hashing math -- probably with specified constants added to it.

I'm still not crazy about introducing a new key though. Would it really
be better than simply sending more copies of the top block during the
initial insert (6 or 9 instead of the original 3?) and reinserting it
again a few times when another user downloads it?

 To prevent these reinserts giving away that we retrieved the file, we
 just need a queue of keys to be inserted (I assume we have this for
 FEC healing?).  Grab a random key from the queue each time we have a
 slot to insert.

Well, if anyone had that kind of access to your node (to the internal
queues, or even the datastore to a lesser extent), it would already be
too late, no?

 I would suggest implementing this before going after the shared bloom
 filters idea.  From a usability standpoint this would solve some of
 the glaring problems like unfetchable pages, pictures and files.

Agreed.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Juiceman]

On Sat, Apr 25, 2009 at 2:23 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 On Friday 24 April 2009 17:46:09 freenetw...@web.de wrote:
 1) CHK-keys are already long enough

 Long enough to be a PITA if they are longer? Or long enough to be functional?
 I dispute the latter.

 2) why add something that tries to fix something broken (routing?) or
 contradicts the concept (caching of keys around the key location; unused
 content gets dropped)

 Routing is not broken. Data persistence is broken. The feedback I have had is
 that frequently the problem with fetching a file is the top block, which
 currently is not redundant. For example it might take 3 weeks at 0% to find
 the top block and then make relatively rapid progress after that. This is not
 your experience?

 if a) unwanted content is supposed to be dropped from the network to
 make space for fresh stuff and b) the top key is *needed* for *every*
 request of a ((larger) split-) file, how can the top key possibly fall
 off the network?

 If the key is not requested, it could well fall off the network, *even though
 the rest of the file is still retrievable*. Because the rest of the file is
 redundant and the top block is not. We have very large datastores and
 generally very low input/output bandwidth, so I would expect data to persist
 for a long time in the current Freenet. Data that has not been recently
 requested will only be in the stores, and not in the caches. But since
 routing appears to work (and there is every theoretical reason to think it
 does), there is a good chance of finding the data - IF the 3 nodes which
 stored it to their datastores are online when you fetch and there aren't any
 problems contacting them (e.g. on darknet they might have swapped).

 IMHO I think this is making extra effort and adding
 YetJustAnotherKeyType for CreateAWorkaroundForSomethingDifferent for
 something that needs to be addressed elsewhere

 I don't see much evidence there is a fundamental problem with routing in 0.7
 on opennet. Do you have any evidence for this? I would be very interested in
 any evidence that this is a routing rather than a redundancy problem.

 Nodes' graphs usually show a fairly strong request specialisation, the main
 symptom is that 2-3 weeks after inserting data, it is not retrievable, if
 nobody has fetched it. And a lot of the time, that specifically relates to
 the top block. This proposal would solve the problem.

 ___
 Support mailing list
 Support@freenetproject.org
 http://news.gmane.org/gmane.network.freenet.support
 Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
 Or mailto:support-requ...@freenetproject.org?subject=unsubscribe


If we have, say, 3 top CHK's or RHK (Redundant Hash Key is my vote)
and we are able to find one of them, we should be able to recreate and
reinsert the other two, right?
To prevent these reinserts giving away that we retrieved the file, we
just need a queue of keys to be inserted (I assume we have this for
FEC healing?).  Grab a random key from the queue each time we have a
slot to insert.

I would suggest implementing this before going after the shared bloom
filters idea.  From a usability standpoint this would solve some of
the glaring problems like unfetchable pages, pictures and files.

-- 
I may disagree with what you have to say, but I shall defend, to the
death, your right to say it. - Voltaire
Those who would give up Liberty, to purchase temporary Safety, deserve
neither Liberty nor Safety. - Ben Franklin
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Matthew Toseland]

On Friday 24 April 2009 17:46:09 freenetwork at web.de wrote:
> 1) CHK-keys are already long enough

Long enough to be a PITA if they are longer? Or long enough to be functional? 
I dispute the latter.

> 2) why add something that tries to fix something broken (routing?) or
> contradicts the concept (caching of keys around the key location; unused
> content gets dropped)

Routing is not broken. Data persistence is broken. The feedback I have had is 
that frequently the problem with fetching a file is the top block, which 
currently is not redundant. For example it might take 3 weeks at 0% to find 
the top block and then make relatively rapid progress after that. This is not 
your experience?
> 
> if a) unwanted content is supposed to be dropped from the network to
> make space for fresh stuff and b) the top key is *needed* for *every*
> request of a ((larger) split-) file, how can the top key possibly fall
> off the network?

If the key is not requested, it could well fall off the network, *even though 
the rest of the file is still retrievable*. Because the rest of the file is 
redundant and the top block is not. We have very large datastores and 
generally very low input/output bandwidth, so I would expect data to persist 
for a long time in the current Freenet. Data that has not been recently 
requested will only be in the stores, and not in the caches. But since 
routing appears to work (and there is every theoretical reason to think it 
does), there is a good chance of finding the data - IF the 3 nodes which 
stored it to their datastores are online when you fetch and there aren't any 
problems contacting them (e.g. on darknet they might have swapped).
> 
> IMHO I think this is making extra effort and adding
> YetJustAnotherKeyType for CreateAWorkaroundForSomethingDifferent for
> something that needs to be addressed elsewhere

I don't see much evidence there is a fundamental problem with routing in 0.7 
on opennet. Do you have any evidence for this? I would be very interested in 
any evidence that this is a routing rather than a redundancy problem.

Nodes' graphs usually show a fairly strong request specialisation, the main 
symptom is that 2-3 weeks after inserting data, it is not retrievable, if 
nobody has fetched it. And a lot of the time, that specifically relates to 
the top block. This proposal would solve the problem.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Matthew Toseland]

On Thursday 23 April 2009 21:23:24 Jack T Mudge III wrote:
> On Thursday 23 April 2009 06:16:40 am Matthew Toseland wrote:
> > Anecdotal evidence suggests that right now at least one third of our
> > content persistence problems boil down to this one bug: "I added it 2 
weeks
> > ago and it still hasn't got past 0% (0/1)". A new key type, DHKs
> > (Duplicated Hash Keys), would solve the problem, but the new keys would be
> > twice as long as current CHKs. Is this a problem? I would really 
appreciate
> > input from users, particularly those who upload and download files:
> > - Is it a problem for the keys to be really long (twice as long as current
> > CHKs)? CHKs are copied and pasted, so maybe not a problem?
> > - Is it true that a great many downloads get stuck at 0% for a long time,
> > showing 0 blocks of 1 if you mouseover the percentage?
> >
> > Example:
> > 
CHK at 4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,97XjJekfSl8HxkJFYhj4cdo9n7s
> >0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3
> >
> > -> (something like)
> >
> > 
DHK at 97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,4~2FTXtBE2So8NZZIzneYrn5SOa
> 
>FFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5
> >NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3
> >
> > GORY DETAILS:
> >
> > Currently we use:
> > CHK@,,
> >
> > (Filenames afterwards are manifests, and therefore impact on the CHK)
> >
> > The new key type would be:
> >
> > DHK@,,, > 3>,/
> >
> > (A filename is mandatory, and is always ignored, so does not impact on the
> > rest of the key).
> >
> > We might allow any number of routing keys from 2 upwards, for more
> > redundancy at the cost of a longer URI, but IMHO 3 is a good default
> > number.
> >
> > You would get such a key when you insert a file as DHK at .
> >
> > Arguably nobody ever types CHKs even now, and copy and paste allows for
> > fairly long keys. Thoughts?
> 
> I doubt copy-pasted keys that are long would pose a problem in most 
> situations. But I can think of a couple of considerations:
> 
> 1. It seems that when keys are posted on FMS (not so much frost), they often 
> get chopped off at 80 characters, leaving the user to remove the newlines by 
> hand. If the keys get long enough that 2 lines isn't always enough, then it 
> might become hasslesome to undo the linebreaking (even if the linebreaking 
is 
> due to misuse of the news client). Maybe the freenet web interface should 
> notice extra newlines and remove them automatically?

Hmmm, because of NNTP. We might be able to do this...
> 
> 2. How much bandwidth do keys take? Do they get sent with every packet? (I 
> don't know much about Freenet internals yet, sorry!). If lots of users offer 
> very little bandwidth each, would the extra key size mean potentially 
slowing 
> things down?

No, this would just be the top level, the keys you quote on FMS or client on 
links. It would not affect lower levels.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Matthew Toseland]

On Thursday 23 April 2009 17:25:11 Dennis Nezic wrote:
> On Thu, 23 Apr 2009 14:16:40 +0100, Matthew Toseland wrote:
> > GORY DETAILS:
> > 
> > Currently we use:
> > CHK@,,
> > 
> > (Filenames afterwards are manifests, and therefore impact on the CHK)
> 
> Isn't the first part supposed to be the data hash, and not a routing
> key. And what is a routing key anyways? :P

The routing key is the hash of the encrypted, padded data of the block.
> 
> How does the filename impact the hash again?

Filenames are interpreted as manifests. This is necessary because a CHK does 
not have a filename. Having an optional filename would make it ambiguous 
whether that filename was a file in a CHK freesite, or whether it was a 
meaningless optional filename. A new key type can have a mandatory filename, 
and thus solve the problem.
> 
> > The new key type would be:
> > 
> > DHK@,,, > 3>,/
> > 
> > (A filename is mandatory, and is always ignored, so does not impact
> > on the rest of the key).
> 
> So the DHK filename part is mandatory, but is simply a recommendation?

Yes. Hence the same data with the same mime type and the same level of 
redundancy will always yield the same DHK (up to the slash).

> So, if the filenames here don't impact on the hash, this should help in
> the case where the same file gets uploaded with different filenames?

Yes, although such files would share the blocks anyway, it is more efficient 
if we always have the same key.

> (Ie., from your description of CHK filenames above, differently named
> files each have different manifests, and thus waste both disk space and
> key space?)

But the real gain is redundany in the top block. And the real cost is a longer 
URI. This is not a big problem?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Dennis Nezic]

On Sat, 25 Apr 2009 19:23:22 +0100, Matthew Toseland wrote:
>  ... IF the 3 nodes which stored it to their datastores are online
> when you fetch and there aren't any problems contacting them (e.g. on
> darknet they might have swapped).

I'm still a little confused about what a routing key is. You seem to be
using it interchangeably to mean either "the hash of the encrypted
padded data", or "the location/some-other-kind-of-id for a node". Why
are we even talking about actual node-ids (that were sent the original
data)? They may very well not even exist after a month or so. (Ie.
others request the blocks from them, and all three of their computers
blow up, or they abandon freenet.)

Why isn't the top block (FEC-) redundant again? If there's a
technical reason for it, then I think we should actively and
transparently cache these "special" blocks. Hacking in
soon-to-be-obsolete node references seems like an ugly bandaid solution
-- ie. it /might/ extend the key lifetimes by a bit -- maybe a few
weeks longer -- but still not nearly as long as the data blocks would
persist for. These "top blocks" perhaps should be placed in a separate
queue and regularly and actively spread to peers?

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Dennis Nezic]

On Sat, 25 Apr 2009 19:15:43 +0100, Matthew Toseland wrote:
> On Thursday 23 April 2009 21:23:24 Jack T Mudge III wrote:
> > 1. It seems that when keys are posted on FMS (not so much frost),
> > they often get chopped off at 80 characters, leaving the user to
> > remove the newlines by hand. If the keys get long enough that 2
> > lines isn't always enough, then it might become hasslesome to undo
> > the linebreaking (even if the linebreaking is due to misuse of the
> > news client). Maybe the freenet web interface should notice extra
> > newlines and remove them automatically?
> 
> Hmmm, because of NNTP. We might be able to do this...

Same problem with freemail  this should really get fixed in the
email/nntp clients though -- ie. they should be made aware of freenet
keys ... although it shouldn't be too hard to have the web interface be
a little more rugged here :b. I do foresee double the headaches with
this issue with DHKs.

Re: [freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Matthew Toseland]

On Thursday 23 April 2009 17:25:11 Dennis Nezic wrote:
 On Thu, 23 Apr 2009 14:16:40 +0100, Matthew Toseland wrote:
  GORY DETAILS:
  
  Currently we use:
  CHK@routing key,crypto key,extra
  
  (Filenames afterwards are manifests, and therefore impact on the CHK)
 
 Isn't the first part supposed to be the data hash, and not a routing
 key. And what is a routing key anyways? :P

The routing key is the hash of the encrypted, padded data of the block.
 
 How does the filename impact the hash again?

Filenames are interpreted as manifests. This is necessary because a CHK does 
not have a filename. Having an optional filename would make it ambiguous 
whether that filename was a file in a CHK freesite, or whether it was a 
meaningless optional filename. A new key type can have a mandatory filename, 
and thus solve the problem.
 
  The new key type would be:
  
  DHK@data hash,routing key 1,routing key 2,routing key 
  3,extra/ignore filename
  
  (A filename is mandatory, and is always ignored, so does not impact
  on the rest of the key).
 
 So the DHK filename part is mandatory, but is simply a recommendation?

Yes. Hence the same data with the same mime type and the same level of 
redundancy will always yield the same DHK (up to the slash).

 So, if the filenames here don't impact on the hash, this should help in
 the case where the same file gets uploaded with different filenames?

Yes, although such files would share the blocks anyway, it is more efficient 
if we always have the same key.

 (Ie., from your description of CHK filenames above, differently named
 files each have different manifests, and thus waste both disk space and
 key space?)

But the real gain is redundany in the top block. And the real cost is a longer 
URI. This is not a big problem?


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Matthew Toseland]

On Thursday 23 April 2009 21:23:24 Jack T Mudge III wrote:
 On Thursday 23 April 2009 06:16:40 am Matthew Toseland wrote:
  Anecdotal evidence suggests that right now at least one third of our
  content persistence problems boil down to this one bug: I added it 2 
weeks
  ago and it still hasn't got past 0% (0/1). A new key type, DHKs
  (Duplicated Hash Keys), would solve the problem, but the new keys would be
  twice as long as current CHKs. Is this a problem? I would really 
appreciate
  input from users, particularly those who upload and download files:
  - Is it a problem for the keys to be really long (twice as long as current
  CHKs)? CHKs are copied and pasted, so maybe not a problem?
  - Is it true that a great many downloads get stuck at 0% for a long time,
  showing 0 blocks of 1 if you mouseover the percentage?
 
  Example:
  
c...@4~2ftxtbe2so8nzzizneyrn5soaffk-hqsvjbhlc77a,97XjJekfSl8HxkJFYhj4cdo9n7s
 0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3
 
  - (something like)
 
  
d...@97xjjekfsl8hxkjfyhj4cdo9n7s0exhe-ewmr8zuvxm,4~2FTXtBE2So8NZZIzneYrn5SOa
 
FFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5
 NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3
 
  GORY DETAILS:
 
  Currently we use:
  CHK@routing key,crypto key,extra
 
  (Filenames afterwards are manifests, and therefore impact on the CHK)
 
  The new key type would be:
 
  DHK@data hash,routing key 1,routing key 2,routing key
  3,extra/ignore filename
 
  (A filename is mandatory, and is always ignored, so does not impact on the
  rest of the key).
 
  We might allow any number of routing keys from 2 upwards, for more
  redundancy at the cost of a longer URI, but IMHO 3 is a good default
  number.
 
  You would get such a key when you insert a file as d...@.
 
  Arguably nobody ever types CHKs even now, and copy and paste allows for
  fairly long keys. Thoughts?
 
 I doubt copy-pasted keys that are long would pose a problem in most 
 situations. But I can think of a couple of considerations:
 
 1. It seems that when keys are posted on FMS (not so much frost), they often 
 get chopped off at 80 characters, leaving the user to remove the newlines by 
 hand. If the keys get long enough that 2 lines isn't always enough, then it 
 might become hasslesome to undo the linebreaking (even if the linebreaking 
is 
 due to misuse of the news client). Maybe the freenet web interface should 
 notice extra newlines and remove them automatically?

Hmmm, because of NNTP. We might be able to do this...
 
 2. How much bandwidth do keys take? Do they get sent with every packet? (I 
 don't know much about Freenet internals yet, sorry!). If lots of users offer 
 very little bandwidth each, would the extra key size mean potentially 
slowing 
 things down?

No, this would just be the top level, the keys you quote on FMS or client on 
links. It would not affect lower levels.


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Dennis Nezic]

On Sat, 25 Apr 2009 19:15:43 +0100, Matthew Toseland wrote:
 On Thursday 23 April 2009 21:23:24 Jack T Mudge III wrote:
  1. It seems that when keys are posted on FMS (not so much frost),
  they often get chopped off at 80 characters, leaving the user to
  remove the newlines by hand. If the keys get long enough that 2
  lines isn't always enough, then it might become hasslesome to undo
  the linebreaking (even if the linebreaking is due to misuse of the
  news client). Maybe the freenet web interface should notice extra
  newlines and remove them automatically?
 
 Hmmm, because of NNTP. We might be able to do this...

Same problem with freemail  this should really get fixed in the
email/nntp clients though -- ie. they should be made aware of freenet
keys ... although it shouldn't be too hard to have the web interface be
a little more rugged here :b. I do foresee double the headaches with
this issue with DHKs.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Dennis Nezic]

On Sat, 25 Apr 2009 19:23:22 +0100, Matthew Toseland wrote:
  ... IF the 3 nodes which stored it to their datastores are online
 when you fetch and there aren't any problems contacting them (e.g. on
 darknet they might have swapped).

I'm still a little confused about what a routing key is. You seem to be
using it interchangeably to mean either the hash of the encrypted
padded data, or the location/some-other-kind-of-id for a node. Why
are we even talking about actual node-ids (that were sent the original
data)? They may very well not even exist after a month or so. (Ie.
others request the blocks from them, and all three of their computers
blow up, or they abandon freenet.)

Why isn't the top block (FEC-) redundant again? If there's a
technical reason for it, then I think we should actively and
transparently cache these special blocks. Hacking in
soon-to-be-obsolete node references seems like an ugly bandaid solution
-- ie. it /might/ extend the key lifetimes by a bit -- maybe a few
weeks longer -- but still not nearly as long as the data blocks would
persist for. These top blocks perhaps should be placed in a separate
queue and regularly and actively spread to peers?
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[freenetw...@web.de]

1) CHK-keys are already long enough
2) why add something that tries to fix something broken (routing?) or
contradicts the concept (caching of keys around the key location; unused
content gets dropped)

if a) unwanted content is supposed to be dropped from the network to
make space for fresh stuff and b) the top key is *needed* for *every*
request of a ((larger) split-) file, how can the top key possibly fall
off the network?

IMHO I think this is making extra effort and adding
YetJustAnotherKeyType for CreateAWorkaroundForSomethingDifferent for
something that needs to be addressed elsewhere


Matthew Toseland wrote:
> Anecdotal evidence suggests that right now at least one third of our content 
> persistence problems boil down to this one bug: "I added it 2 weeks ago and 
> it still hasn't got past 0% (0/1)". A new key type, DHKs (Duplicated Hash 
> Keys), would solve the problem, but the new keys would be twice as long as 
> current CHKs. Is this a problem? I would really appreciate input from users, 
> particularly those who upload and download files:
> - Is it a problem for the keys to be really long (twice as long as current 
> CHKs)? CHKs are copied and pasted, so maybe not a problem?
> - Is it true that a great many downloads get stuck at 0% for a long time, 
> showing 0 blocks of 1 if you mouseover the percentage?
>
> Example:
> CHK at 
> 4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3
>
> -> (something like)
>
> DHK at 
> 97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3
>
> GORY DETAILS:
>
> Currently we use:
> CHK@,,
>
> (Filenames afterwards are manifests, and therefore impact on the CHK)
>
> The new key type would be:
>
> DHK@,,, 3>,/
>
> (A filename is mandatory, and is always ignored, so does not impact on the 
> rest of the key).
>
> We might allow any number of routing keys from 2 upwards, for more redundancy 
> at the cost of a longer URI, but IMHO 3 is a good default number.
>
> You would get such a key when you insert a file as DHK at .
>
> Arguably nobody ever types CHKs even now, and copy and paste allows for 
> fairly 
> long keys. Thoughts?
>   
> 
>
> ___
> Support mailing list
> Support at freenetproject.org
> http://news.gmane.org/gmane.network.freenet.support
> Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
> Or mailto:support-request at freenetproject.org?subject=unsubscribe

Re: [freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[freenetw...@web.de]

1) CHK-keys are already long enough
2) why add something that tries to fix something broken (routing?) or
contradicts the concept (caching of keys around the key location; unused
content gets dropped)

if a) unwanted content is supposed to be dropped from the network to
make space for fresh stuff and b) the top key is *needed* for *every*
request of a ((larger) split-) file, how can the top key possibly fall
off the network?

IMHO I think this is making extra effort and adding
YetJustAnotherKeyType for CreateAWorkaroundForSomethingDifferent for
something that needs to be addressed elsewhere


Matthew Toseland wrote:
 Anecdotal evidence suggests that right now at least one third of our content 
 persistence problems boil down to this one bug: I added it 2 weeks ago and 
 it still hasn't got past 0% (0/1). A new key type, DHKs (Duplicated Hash 
 Keys), would solve the problem, but the new keys would be twice as long as 
 current CHKs. Is this a problem? I would really appreciate input from users, 
 particularly those who upload and download files:
 - Is it a problem for the keys to be really long (twice as long as current 
 CHKs)? CHKs are copied and pasted, so maybe not a problem?
 - Is it true that a great many downloads get stuck at 0% for a long time, 
 showing 0 blocks of 1 if you mouseover the percentage?

 Example:
 c...@4~2ftxtbe2so8nzzizneyrn5soaffk-hqsvjbhlc77a,97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3

 - (something like)

 d...@97xjjekfsl8hxkjfyhj4cdo9n7s0exhe-ewmr8zuvxm,4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3

 GORY DETAILS:

 Currently we use:
 CHK@routing key,crypto key,extra

 (Filenames afterwards are manifests, and therefore impact on the CHK)

 The new key type would be:

 DHK@data hash,routing key 1,routing key 2,routing key 
 3,extra/ignore filename

 (A filename is mandatory, and is always ignored, so does not impact on the 
 rest of the key).

 We might allow any number of routing keys from 2 upwards, for more redundancy 
 at the cost of a longer URI, but IMHO 3 is a good default number.

 You would get such a key when you insert a file as d...@.

 Arguably nobody ever types CHKs even now, and copy and paste allows for 
 fairly 
 long keys. Thoughts?
   
 

 ___
 Support mailing list
 Support@freenetproject.org
 http://news.gmane.org/gmane.network.freenet.support
 Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
 Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Matthew Toseland]

Anecdotal evidence suggests that right now at least one third of our content 
persistence problems boil down to this one bug: "I added it 2 weeks ago and 
it still hasn't got past 0% (0/1)". A new key type, DHKs (Duplicated Hash 
Keys), would solve the problem, but the new keys would be twice as long as 
current CHKs. Is this a problem? I would really appreciate input from users, 
particularly those who upload and download files:
- Is it a problem for the keys to be really long (twice as long as current 
CHKs)? CHKs are copied and pasted, so maybe not a problem?
- Is it true that a great many downloads get stuck at 0% for a long time, 
showing 0 blocks of 1 if you mouseover the percentage?

Example:
CHK at 
4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3

-> (something like)

DHK at 
97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3

GORY DETAILS:

Currently we use:
CHK@,,

(Filenames afterwards are manifests, and therefore impact on the CHK)

The new key type would be:

DHK@/

(A filename is mandatory, and is always ignored, so does not impact on the 
rest of the key).

We might allow any number of routing keys from 2 upwards, for more redundancy 
at the cost of a longer URI, but IMHO 3 is a good default number.

You would get such a key when you insert a file as DHK at .

Arguably nobody ever types CHKs even now, and copy and paste allows for fairly 
long keys. Thoughts?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Jack T Mudge III]

On Thursday 23 April 2009 06:16:40 am Matthew Toseland wrote:
> Anecdotal evidence suggests that right now at least one third of our
> content persistence problems boil down to this one bug: "I added it 2 weeks
> ago and it still hasn't got past 0% (0/1)". A new key type, DHKs
> (Duplicated Hash Keys), would solve the problem, but the new keys would be
> twice as long as current CHKs. Is this a problem? I would really appreciate
> input from users, particularly those who upload and download files:
> - Is it a problem for the keys to be really long (twice as long as current
> CHKs)? CHKs are copied and pasted, so maybe not a problem?
> - Is it true that a great many downloads get stuck at 0% for a long time,
> showing 0 blocks of 1 if you mouseover the percentage?
>
> Example:
> CHK at 4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,97XjJekfSl8HxkJFYhj4cdo9n7s
>0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3
>
> -> (something like)
>
> DHK at 97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,4~2FTXtBE2So8NZZIzneYrn5SOa
>FFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5
>NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3
>
> GORY DETAILS:
>
> Currently we use:
> CHK@,,
>
> (Filenames afterwards are manifests, and therefore impact on the CHK)
>
> The new key type would be:
>
> DHK@,,, 3>,/
>
> (A filename is mandatory, and is always ignored, so does not impact on the
> rest of the key).
>
> We might allow any number of routing keys from 2 upwards, for more
> redundancy at the cost of a longer URI, but IMHO 3 is a good default
> number.
>
> You would get such a key when you insert a file as DHK at .
>
> Arguably nobody ever types CHKs even now, and copy and paste allows for
> fairly long keys. Thoughts?

I doubt copy-pasted keys that are long would pose a problem in most 
situations. But I can think of a couple of considerations:

1. It seems that when keys are posted on FMS (not so much frost), they often 
get chopped off at 80 characters, leaving the user to remove the newlines by 
hand. If the keys get long enough that 2 lines isn't always enough, then it 
might become hasslesome to undo the linebreaking (even if the linebreaking is 
due to misuse of the news client). Maybe the freenet web interface should 
notice extra newlines and remove them automatically?

2. How much bandwidth do keys take? Do they get sent with every packet? (I 
don't know much about Freenet internals yet, sorry!). If lots of users offer 
very little bandwidth each, would the extra key size mean potentially slowing 
things down?



-- 
Sincerely,
Jack Mudge
jakykong at theanythingbox.com

[freenet-support] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Dennis Nezic]

On Thu, 23 Apr 2009 14:16:40 +0100, Matthew Toseland wrote:
> GORY DETAILS:
> 
> Currently we use:
> CHK@,,
> 
> (Filenames afterwards are manifests, and therefore impact on the CHK)

Isn't the first part supposed to be the data hash, and not a routing
key. And what is a routing key anyways? :P

How does the filename impact the hash again?

> The new key type would be:
> 
> DHK@,,, 3>,/
> 
> (A filename is mandatory, and is always ignored, so does not impact
> on the rest of the key).

So the DHK filename part is mandatory, but is simply a recommendation?
So, if the filenames here don't impact on the hash, this should help in
the case where the same file gets uploaded with different filenames?
(Ie., from your description of CHK filenames above, differently named
files each have different manifests, and thus waste both disk space and
key space?)

[freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Matthew Toseland]

Anecdotal evidence suggests that right now at least one third of our content 
persistence problems boil down to this one bug: I added it 2 weeks ago and 
it still hasn't got past 0% (0/1). A new key type, DHKs (Duplicated Hash 
Keys), would solve the problem, but the new keys would be twice as long as 
current CHKs. Is this a problem? I would really appreciate input from users, 
particularly those who upload and download files:
- Is it a problem for the keys to be really long (twice as long as current 
CHKs)? CHKs are copied and pasted, so maybe not a problem?
- Is it true that a great many downloads get stuck at 0% for a long time, 
showing 0 blocks of 1 if you mouseover the percentage?

Example:
c...@4~2ftxtbe2so8nzzizneyrn5soaffk-hqsvjbhlc77a,97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3

- (something like)

d...@97xjjekfsl8hxkjfyhj4cdo9n7s0exhe-ewmr8zuvxm,4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3

GORY DETAILS:

Currently we use:
CHK@routing key,crypto key,extra

(Filenames afterwards are manifests, and therefore impact on the CHK)

The new key type would be:

DHK@data hash,routing key 1,routing key 2,routing key 
3,extra/ignore filename

(A filename is mandatory, and is always ignored, so does not impact on the 
rest of the key).

We might allow any number of routing keys from 2 upwards, for more redundancy 
at the cost of a longer URI, but IMHO 3 is a good default number.

You would get such a key when you insert a file as d...@.

Arguably nobody ever types CHKs even now, and copy and paste allows for fairly 
long keys. Thoughts?


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Dennis Nezic]

On Thu, 23 Apr 2009 14:16:40 +0100, Matthew Toseland wrote:
 GORY DETAILS:
 
 Currently we use:
 CHK@routing key,crypto key,extra
 
 (Filenames afterwards are manifests, and therefore impact on the CHK)

Isn't the first part supposed to be the data hash, and not a routing
key. And what is a routing key anyways? :P

How does the filename impact the hash again?

 The new key type would be:
 
 DHK@data hash,routing key 1,routing key 2,routing key 
 3,extra/ignore filename
 
 (A filename is mandatory, and is always ignored, so does not impact
 on the rest of the key).

So the DHK filename part is mandatory, but is simply a recommendation?
So, if the filenames here don't impact on the hash, this should help in
the case where the same file gets uploaded with different filenames?
(Ie., from your description of CHK filenames above, differently named
files each have different manifests, and thus waste both disk space and
key space?)
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe