[freenet-dev] Priority of fixing bulk filesharing performance/disk issues? was Re: Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive

[Florent Daigniere]

On Sat, Apr 03, 2010 at 07:30:50PM -0400, Evan Daniel wrote:
> On Sat, Apr 3, 2010 at 6:01 PM, Matthew Toseland
>  wrote:
> > On Friday 02 April 2010 17:43:25 Evan Daniel wrote:
> >> On Fri, Apr 2, 2010 at 12:39 PM, Matthew Toseland
> >>  wrote:
> >> > On Friday 02 April 2010 17:31:13 Matthew Toseland wrote:
> >> >> On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote:
> >> >> > You should really send these to the support list; that's what it's 
> >> >> > for.
> >> >> >
> >> >> > You can change the physical security level setting independently of
> >> >> > the network seclevels -- see configuration -> security levels.
> >> >> >
> >> >> > I'm not sure what else to suggest at this point. ?You could try
> >> >> > increasing the amount of ram for temp buckets (configuration -> core
> >> >> > settings), but that's mostly a stab in the dark.
> >> >> >
> >> >> > I suspect you need to reduce the amount of stuff in your queue.
> >> >>
> >> >> Thanks Evan for helping Daniel. In theory it ought to be possible to 
> >> >> have a nearly unlimited number of downloads in the queue: That is 
> >> >> precisely why we decided to use a database to store the progress of 
> >> >> downloads. Unfortunately, in practice, disks are slow, and the more 
> >> >> stuff is queued, the less of it will be cached in RAM i.e. the more 
> >> >> reliant we are on slow disks.
> >> >>
> >> >> There are many options for optimising the code so that it uses the disk 
> >> >> less. But unfortunately they are all a significant amount of work.
> >> >>
> >> >> See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is 
> >> >> marked as related to.
> >> >
> >> > So I guess the real question here is, how important is it that we be 
> >> > able to queue 60 downloads and still have acceptable performance? How 
> >> > many users use Freenet filesharing in that sort of way?
> >>
> >> All of them, I suspect. ?If a file is mostly downloaded, but not
> >> complete, the natural response seems to be to leave it there in hopes
> >> it will complete, and add other files in the mean time. ?Combined with
> >> unretrievable files due to missing blocks, this will produce very
> >> large download queues.
> >
> > So this bug should be fairly high priority, despite its potentially being 
> > quite a lot of work?:
> >
> > https://bugs.freenetproject.org/view.php?id=4031
> 
> I think so.  I believe I've been saying client layer should be high
> priority for a while :)
> 
> Evan Daniel

Matthew, if it didn't change, you're the only one who understand the 
client-layer code of fred...
Yes, it sounds like very-high priority to me too.

Florent

Re: [freenet-dev] Priority of fixing bulk filesharing performance/disk issues? was Re: Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive

[Florent Daigniere]

On Sat, Apr 03, 2010 at 07:30:50PM -0400, Evan Daniel wrote:
> On Sat, Apr 3, 2010 at 6:01 PM, Matthew Toseland
>  wrote:
> > On Friday 02 April 2010 17:43:25 Evan Daniel wrote:
> >> On Fri, Apr 2, 2010 at 12:39 PM, Matthew Toseland
> >>  wrote:
> >> > On Friday 02 April 2010 17:31:13 Matthew Toseland wrote:
> >> >> On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote:
> >> >> > You should really send these to the support list; that's what it's 
> >> >> > for.
> >> >> >
> >> >> > You can change the physical security level setting independently of
> >> >> > the network seclevels -- see configuration -> security levels.
> >> >> >
> >> >> > I'm not sure what else to suggest at this point.  You could try
> >> >> > increasing the amount of ram for temp buckets (configuration -> core
> >> >> > settings), but that's mostly a stab in the dark.
> >> >> >
> >> >> > I suspect you need to reduce the amount of stuff in your queue.
> >> >>
> >> >> Thanks Evan for helping Daniel. In theory it ought to be possible to 
> >> >> have a nearly unlimited number of downloads in the queue: That is 
> >> >> precisely why we decided to use a database to store the progress of 
> >> >> downloads. Unfortunately, in practice, disks are slow, and the more 
> >> >> stuff is queued, the less of it will be cached in RAM i.e. the more 
> >> >> reliant we are on slow disks.
> >> >>
> >> >> There are many options for optimising the code so that it uses the disk 
> >> >> less. But unfortunately they are all a significant amount of work.
> >> >>
> >> >> See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is 
> >> >> marked as related to.
> >> >
> >> > So I guess the real question here is, how important is it that we be 
> >> > able to queue 60 downloads and still have acceptable performance? How 
> >> > many users use Freenet filesharing in that sort of way?
> >>
> >> All of them, I suspect.  If a file is mostly downloaded, but not
> >> complete, the natural response seems to be to leave it there in hopes
> >> it will complete, and add other files in the mean time.  Combined with
> >> unretrievable files due to missing blocks, this will produce very
> >> large download queues.
> >
> > So this bug should be fairly high priority, despite its potentially being 
> > quite a lot of work?:
> >
> > https://bugs.freenetproject.org/view.php?id=4031
> 
> I think so.  I believe I've been saying client layer should be high
> priority for a while :)
> 
> Evan Daniel

Matthew, if it didn't change, you're the only one who understand the 
client-layer code of fred...
Yes, it sounds like very-high priority to me too.

Florent
___
Devl mailing list
Devl@freenetproject.org
http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl

[freenet-dev] Priority of fixing bulk filesharing performance/disk issues? was Re: Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive

[Matthew Toseland]

On Friday 02 April 2010 17:43:25 Evan Daniel wrote:
> On Fri, Apr 2, 2010 at 12:39 PM, Matthew Toseland
>  wrote:
> > On Friday 02 April 2010 17:31:13 Matthew Toseland wrote:
> >> On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote:
> >> > You should really send these to the support list; that's what it's for.
> >> >
> >> > You can change the physical security level setting independently of
> >> > the network seclevels -- see configuration -> security levels.
> >> >
> >> > I'm not sure what else to suggest at this point. ?You could try
> >> > increasing the amount of ram for temp buckets (configuration -> core
> >> > settings), but that's mostly a stab in the dark.
> >> >
> >> > I suspect you need to reduce the amount of stuff in your queue.
> >>
> >> Thanks Evan for helping Daniel. In theory it ought to be possible to have 
> >> a nearly unlimited number of downloads in the queue: That is precisely why 
> >> we decided to use a database to store the progress of downloads. 
> >> Unfortunately, in practice, disks are slow, and the more stuff is queued, 
> >> the less of it will be cached in RAM i.e. the more reliant we are on slow 
> >> disks.
> >>
> >> There are many options for optimising the code so that it uses the disk 
> >> less. But unfortunately they are all a significant amount of work.
> >>
> >> See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is 
> >> marked as related to.
> >
> > So I guess the real question here is, how important is it that we be able 
> > to queue 60 downloads and still have acceptable performance? How many users 
> > use Freenet filesharing in that sort of way?
> 
> All of them, I suspect.  If a file is mostly downloaded, but not
> complete, the natural response seems to be to leave it there in hopes
> it will complete, and add other files in the mean time.  Combined with
> unretrievable files due to missing blocks, this will produce very
> large download queues.

So this bug should be fairly high priority, despite its potentially being quite 
a lot of work?:

https://bugs.freenetproject.org/view.php?id=4031
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-dev] Priority of fixing bulk filesharing performance/disk issues? was Re: Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive

[Evan Daniel]

On Sat, Apr 3, 2010 at 6:01 PM, Matthew Toseland
 wrote:
> On Friday 02 April 2010 17:43:25 Evan Daniel wrote:
>> On Fri, Apr 2, 2010 at 12:39 PM, Matthew Toseland
>>  wrote:
>> > On Friday 02 April 2010 17:31:13 Matthew Toseland wrote:
>> >> On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote:
>> >> > You should really send these to the support list; that's what it's for.
>> >> >
>> >> > You can change the physical security level setting independently of
>> >> > the network seclevels -- see configuration -> security levels.
>> >> >
>> >> > I'm not sure what else to suggest at this point. ?You could try
>> >> > increasing the amount of ram for temp buckets (configuration -> core
>> >> > settings), but that's mostly a stab in the dark.
>> >> >
>> >> > I suspect you need to reduce the amount of stuff in your queue.
>> >>
>> >> Thanks Evan for helping Daniel. In theory it ought to be possible to have 
>> >> a nearly unlimited number of downloads in the queue: That is precisely 
>> >> why we decided to use a database to store the progress of downloads. 
>> >> Unfortunately, in practice, disks are slow, and the more stuff is queued, 
>> >> the less of it will be cached in RAM i.e. the more reliant we are on slow 
>> >> disks.
>> >>
>> >> There are many options for optimising the code so that it uses the disk 
>> >> less. But unfortunately they are all a significant amount of work.
>> >>
>> >> See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is 
>> >> marked as related to.
>> >
>> > So I guess the real question here is, how important is it that we be able 
>> > to queue 60 downloads and still have acceptable performance? How many 
>> > users use Freenet filesharing in that sort of way?
>>
>> All of them, I suspect. ?If a file is mostly downloaded, but not
>> complete, the natural response seems to be to leave it there in hopes
>> it will complete, and add other files in the mean time. ?Combined with
>> unretrievable files due to missing blocks, this will produce very
>> large download queues.
>
> So this bug should be fairly high priority, despite its potentially being 
> quite a lot of work?:
>
> https://bugs.freenetproject.org/view.php?id=4031

I think so.  I believe I've been saying client layer should be high
priority for a while :)

Evan Daniel

Re: [freenet-dev] Priority of fixing bulk filesharing performance/disk issues? was Re: Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive

[Evan Daniel]

On Sat, Apr 3, 2010 at 6:01 PM, Matthew Toseland
 wrote:
> On Friday 02 April 2010 17:43:25 Evan Daniel wrote:
>> On Fri, Apr 2, 2010 at 12:39 PM, Matthew Toseland
>>  wrote:
>> > On Friday 02 April 2010 17:31:13 Matthew Toseland wrote:
>> >> On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote:
>> >> > You should really send these to the support list; that's what it's for.
>> >> >
>> >> > You can change the physical security level setting independently of
>> >> > the network seclevels -- see configuration -> security levels.
>> >> >
>> >> > I'm not sure what else to suggest at this point.  You could try
>> >> > increasing the amount of ram for temp buckets (configuration -> core
>> >> > settings), but that's mostly a stab in the dark.
>> >> >
>> >> > I suspect you need to reduce the amount of stuff in your queue.
>> >>
>> >> Thanks Evan for helping Daniel. In theory it ought to be possible to have 
>> >> a nearly unlimited number of downloads in the queue: That is precisely 
>> >> why we decided to use a database to store the progress of downloads. 
>> >> Unfortunately, in practice, disks are slow, and the more stuff is queued, 
>> >> the less of it will be cached in RAM i.e. the more reliant we are on slow 
>> >> disks.
>> >>
>> >> There are many options for optimising the code so that it uses the disk 
>> >> less. But unfortunately they are all a significant amount of work.
>> >>
>> >> See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is 
>> >> marked as related to.
>> >
>> > So I guess the real question here is, how important is it that we be able 
>> > to queue 60 downloads and still have acceptable performance? How many 
>> > users use Freenet filesharing in that sort of way?
>>
>> All of them, I suspect.  If a file is mostly downloaded, but not
>> complete, the natural response seems to be to leave it there in hopes
>> it will complete, and add other files in the mean time.  Combined with
>> unretrievable files due to missing blocks, this will produce very
>> large download queues.
>
> So this bug should be fairly high priority, despite its potentially being 
> quite a lot of work?:
>
> https://bugs.freenetproject.org/view.php?id=4031

I think so.  I believe I've been saying client layer should be high
priority for a while :)

Evan Daniel
___
Devl mailing list
Devl@freenetproject.org
http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl

[freenet-dev] Priority of fixing bulk filesharing performance/disk issues? was Re: Another way Freenet sucks for filesharing was Re: [freenet-support] major problems - stuck at 100%, nonresponsive

[Matthew Toseland]

On Friday 02 April 2010 17:43:25 Evan Daniel wrote:
> On Fri, Apr 2, 2010 at 12:39 PM, Matthew Toseland
>  wrote:
> > On Friday 02 April 2010 17:31:13 Matthew Toseland wrote:
> >> On Tuesday 09 March 2010 04:27:24 Evan Daniel wrote:
> >> > You should really send these to the support list; that's what it's for.
> >> >
> >> > You can change the physical security level setting independently of
> >> > the network seclevels -- see configuration -> security levels.
> >> >
> >> > I'm not sure what else to suggest at this point.  You could try
> >> > increasing the amount of ram for temp buckets (configuration -> core
> >> > settings), but that's mostly a stab in the dark.
> >> >
> >> > I suspect you need to reduce the amount of stuff in your queue.
> >>
> >> Thanks Evan for helping Daniel. In theory it ought to be possible to have 
> >> a nearly unlimited number of downloads in the queue: That is precisely why 
> >> we decided to use a database to store the progress of downloads. 
> >> Unfortunately, in practice, disks are slow, and the more stuff is queued, 
> >> the less of it will be cached in RAM i.e. the more reliant we are on slow 
> >> disks.
> >>
> >> There are many options for optimising the code so that it uses the disk 
> >> less. But unfortunately they are all a significant amount of work.
> >>
> >> See https://bugs.freenetproject.org/view.php?id=4031 and the bugs it is 
> >> marked as related to.
> >
> > So I guess the real question here is, how important is it that we be able 
> > to queue 60 downloads and still have acceptable performance? How many users 
> > use Freenet filesharing in that sort of way?
> 
> All of them, I suspect.  If a file is mostly downloaded, but not
> complete, the natural response seems to be to leave it there in hopes
> it will complete, and add other files in the mean time.  Combined with
> unretrievable files due to missing blocks, this will produce very
> large download queues.

So this bug should be fairly high priority, despite its potentially being quite 
a lot of work?:

https://bugs.freenetproject.org/view.php?id=4031


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://osprey.vm.bytemark.co.uk/cgi-bin/mailman/listinfo/devl