On Tue, Oct 21, 2008 at 11:24 PM, wrote:
> Author: nextgens
> Date: 2008-10-21 15:24:47 + (Tue, 21 Oct 2008)
> New Revision: 23014
>
> Modified:
> trunk/freenet/src/freenet/client/ArchiveManager.java
> trunk/freenet/src/freenet/client/ArchiveStoreContext.java
> trunk/freenet/src/freenet
On Thu, Oct 23, 2008 at 9:02 PM, Florent Daigniere
wrote:
> Florent Daigniere wrote:
>> Matthew Toseland wrote:
>>> On Thursday 23 October 2008 10:39, NextGen$ wrote:
* Daniel Cheng [2008-10-23 08:12:14]:
> On Thu, Oct 23, 2008 at 6:49 AM, NextGen$
>>> wrote:
>> * Matthew Tosel
On Thu, Oct 23, 2008 at 9:51 PM, Florent Daigniere
<[EMAIL PROTECTED]> wrote:
> Daniel Cheng wrote:
>> On Thu, Oct 23, 2008 at 9:02 PM, Florent Daigniere
>> <[EMAIL PROTECTED]> wrote:
>>> Florent Daigniere wrote:
Matthew Toseland wrote:
> On Thursday 23 October 2008 10:39, NextGen$ wrote:
Matthew Toseland wrote:
> On Thursday 23 October 2008 14:02, Florent Daigniere wrote:
>> Florent Daigniere wrote:
>>> Matthew Toseland wrote:
On Thursday 23 October 2008 10:39, NextGen$ wrote:
> * Daniel Cheng [2008-10-23 08:12:14]:
>
>> On Thu, Oct 23, 2008 at 6:49 AM, NextGen$ >
Daniel Cheng wrote:
> On Thu, Oct 23, 2008 at 9:02 PM, Florent Daigniere
> wrote:
>> Florent Daigniere wrote:
>>> Matthew Toseland wrote:
On Thursday 23 October 2008 10:39, NextGen$ wrote:
> * Daniel Cheng [2008-10-23 08:12:14]:
>
>> On Thu, Oct 23, 2008 at 6:49 AM, NextGen$
efficiency of a given
> > compression algorithm but I am not sure they are worth implementing.
> >
>
> Here is some more representative data on a dual-core system:
>
> real24m55.472s
> user23m4.947s
> sys 0m10.633s
> 1884544 iso.lzma
>
> real13m32.442s
> user12m6.937s
> sys 0m7.784s
> 1934324 iso.bz2
> My implementation of BZIP2 uses only one of the two cores
>
> real3m19.066s
> user2m11.332s
> sys 0m6.284s
> 1935056 iso.gz
>
> And the original :
> 2026416 iso
>
> So, we have:
> 63325 blocks for the original
> 60470 blocks with GZIP (4.5% gain)
> 60447 blocks with BZIP2 (4.5% gain)
> 58892 blocks with LZMA (7% gain)
>
> Of course those don't include the FEC blocks: So to sum up, yes I think
> it's worth spending half an hour of CPU time to "win" 4433*2=8866
> blocks. And that's still true on a single core system where we would
> spend 1 hour.
Okay, then the current trunk code is fine. Lzma would be great, if you can
solve the DoS issues; we'd probably use -5, I'm definitely not comfortable
with -7:
-1 2 MB 1 MB
-2 12 MB 2 MB
-3 12 MB 1 MB
-4 16 MB 2 MB
-5 26 MB 3 MB
-6 45 MB 5 MB
-7 83 MB 9 MB
-8159 MB 17 MB
-9311 MB 33 MB
Eventually we should show a progress bar within compression. In the short
term, it would be reasonably easy for fproxy to show "Compressing" when it is
compressing, instead of just having no progress bar. After that it should
move to "Starting", and after that show a progress bar. If you want me to
deal with that I'll get around to it eventually; should I file a bug? The UI
changes are needed for 0.8 but not for 1166.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: not available
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20081023/c5d84d38/attachment.pgp>
Florent Daigniere wrote:
> Matthew Toseland wrote:
>> On Thursday 23 October 2008 10:39, NextGen$ wrote:
>>> * Daniel Cheng [2008-10-23 08:12:14]:
>>>
On Thu, Oct 23, 2008 at 6:49 AM, NextGen$
>> wrote:
> * Matthew Toseland [2008-10-22 20:48:24]:
>
>> On Wednesday 22 October 20
Matthew Toseland wrote:
> On Thursday 23 October 2008 10:39, NextGen$ wrote:
>> * Daniel Cheng [2008-10-23 08:12:14]:
>>
>>> On Thu, Oct 23, 2008 at 6:49 AM, NextGen$
> wrote:
* Matthew Toseland [2008-10-22 20:48:24]:
> On Wednesday 22 October 2008 01:09, NextGen$ wrote:
>> *
t;private void tryCompress() throws InsertException {
> > + synchronized(compressorSync) {
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: not available
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20081023/88bc5d1b/attachment.pgp>
ingle
> segment: it's just too small.
>
> > As soon as we get enough compressed data for FEC, we can insert them.
> > This would be a great preformance improvement for large file on SMP.
> >
>
> That would involve rewritting most of the client-layer.
>
> > It this doable without changing the data format?
> >
>
> It's not about the data format; we insert the manifest at the end unless not
> told to by the earlyEncode parameter.
>
> IMHO we are debating for no real reason here: the real-time taken by the
> compression phase is insignificant compared to the time taken by the
> insertion process. Sure, trunk will take at least 3 times longer than
current
> stable before it starts inserting anything; but is that a big deal? You will
> need real numbers to convince me here.
I'd like some numbers ... iirc it takes around 2 days to insert a CD-sized
ISO? How long does it take to bzip2 it?
Also from a usability point of view, having Freenet apparently not doing
anything with an insert for hours is *bad*. We will need a compression
progress monitor.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: not available
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20081023/aa1c07f0/attachment.pgp>
are... we
could
> > do something similar with inserts in compression.
> >
>
> Have a look to what I have commited. From now on the compression is fully
> serialized... We have one mutex, and only one compression job (just like we
> do for FEC encoding in fact) which means a even higher latency.
For FEC we can run up to min(3, # cores) jobs. The limit of 3 can be gotten
rid of if/when we solve the too many fd's problem (each encode uses 256 fd's
at the moment); I have a partial fix on the db4o branch but it needs
optimising and extending to non-persistent ...
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: not available
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20081023/953e8554/attachment.pgp>
On Thu, Oct 23, 2008 at 6:40 AM, wrote:
> Author: nextgens
> Date: 2008-10-22 22:40:31 + (Wed, 22 Oct 2008)
> New Revision: 23041
>
> Modified:
> trunk/freenet/src/freenet/client/async/SingleFileInserter.java
> Log:
> serialize compression so that we don't run out of memory
>
> Modified:
* Daniel Cheng [2008-10-23 08:12:14]:
> On Thu, Oct 23, 2008 at 6:49 AM, NextGen$
> wrote:
> > * Matthew Toseland [2008-10-22 20:48:24]:
> >
> >> On Wednesday 22 October 2008 01:09, NextGen$ wrote:
> >> > * Matthew Toseland [2008-10-21 20:53:51]:
> >> >
> >> > > On Tuesday 21 October 2008 16:
On Tue, Oct 21, 2008 at 11:24 PM, <[EMAIL PROTECTED]> wrote:
> Author: nextgens
> Date: 2008-10-21 15:24:47 + (Tue, 21 Oct 2008)
> New Revision: 23014
>
> Modified:
> trunk/freenet/src/freenet/client/ArchiveManager.java
> trunk/freenet/src/freenet/client/ArchiveStoreContext.java
> trunk/
On Thu, Oct 23, 2008 at 6:49 AM, NextGen$
wrote:
> * Matthew Toseland [2008-10-22 20:48:24]:
>
>> On Wednesday 22 October 2008 01:09, NextGen$ wrote:
>> > * Matthew Toseland [2008-10-21 20:53:51]:
>> >
>> > > On Tuesday 21 October 2008 16:24, nextgens at freenetproject.org wrote:
>> > > > Autho
Matthew Toseland wrote:
> On Thursday 23 October 2008 14:02, Florent Daigniere wrote:
>> Florent Daigniere wrote:
>>> Matthew Toseland wrote:
On Thursday 23 October 2008 10:39, NextGen$ wrote:
> * Daniel Cheng <[EMAIL PROTECTED]> [2008-10-23 08:12:14]:
>
>> On Thu, Oct 23, 2008 at
Daniel Cheng wrote:
> On Thu, Oct 23, 2008 at 9:02 PM, Florent Daigniere
> <[EMAIL PROTECTED]> wrote:
>> Florent Daigniere wrote:
>>> Matthew Toseland wrote:
On Thursday 23 October 2008 10:39, NextGen$ wrote:
> * Daniel Cheng <[EMAIL PROTECTED]> [2008-10-23 08:12:14]:
>
>> On Thu,
On Thursday 23 October 2008 14:02, Florent Daigniere wrote:
> Florent Daigniere wrote:
> > Matthew Toseland wrote:
> >> On Thursday 23 October 2008 10:39, NextGen$ wrote:
> >>> * Daniel Cheng <[EMAIL PROTECTED]> [2008-10-23 08:12:14]:
> >>>
> On Thu, Oct 23, 2008 at 6:49 AM, NextGen$ <[EMAIL P
On Thu, Oct 23, 2008 at 9:02 PM, Florent Daigniere
<[EMAIL PROTECTED]> wrote:
> Florent Daigniere wrote:
>> Matthew Toseland wrote:
>>> On Thursday 23 October 2008 10:39, NextGen$ wrote:
* Daniel Cheng <[EMAIL PROTECTED]> [2008-10-23 08:12:14]:
> On Thu, Oct 23, 2008 at 6:49 AM, NextG
Florent Daigniere wrote:
> Matthew Toseland wrote:
>> On Thursday 23 October 2008 10:39, NextGen$ wrote:
>>> * Daniel Cheng <[EMAIL PROTECTED]> [2008-10-23 08:12:14]:
>>>
On Thu, Oct 23, 2008 at 6:49 AM, NextGen$ <[EMAIL PROTECTED]>
>> wrote:
> * Matthew Toseland <[EMAIL PROTECTED]> [2008
Matthew Toseland wrote:
> On Thursday 23 October 2008 10:39, NextGen$ wrote:
>> * Daniel Cheng <[EMAIL PROTECTED]> [2008-10-23 08:12:14]:
>>
>>> On Thu, Oct 23, 2008 at 6:49 AM, NextGen$ <[EMAIL PROTECTED]>
> wrote:
* Matthew Toseland <[EMAIL PROTECTED]> [2008-10-22 20:48:24]:
> On W
On Thursday 23 October 2008 04:28, Daniel Cheng wrote:
> On Thu, Oct 23, 2008 at 6:40 AM, <[EMAIL PROTECTED]> wrote:
> > Author: nextgens
> > Date: 2008-10-22 22:40:31 + (Wed, 22 Oct 2008)
> > New Revision: 23041
> >
> > Modified:
> > trunk/freenet/src/freenet/client/async/SingleFileInserter
On Thursday 23 October 2008 10:39, NextGen$ wrote:
> * Daniel Cheng <[EMAIL PROTECTED]> [2008-10-23 08:12:14]:
>
> > On Thu, Oct 23, 2008 at 6:49 AM, NextGen$ <[EMAIL PROTECTED]>
wrote:
> > > * Matthew Toseland <[EMAIL PROTECTED]> [2008-10-22 20:48:24]:
> > >
> > >> On Wednesday 22 October 2008 0
On Wednesday 22 October 2008 23:49, NextGen$ wrote:
> * Matthew Toseland <[EMAIL PROTECTED]> [2008-10-22 20:48:24]:
>
> > On Wednesday 22 October 2008 01:09, NextGen$ wrote:
> > > * Matthew Toseland <[EMAIL PROTECTED]> [2008-10-21 20:53:51]:
> > >
> > > > On Tuesday 21 October 2008 16:24, [EMAIL
* Daniel Cheng <[EMAIL PROTECTED]> [2008-10-23 08:12:14]:
> On Thu, Oct 23, 2008 at 6:49 AM, NextGen$ <[EMAIL PROTECTED]> wrote:
> > * Matthew Toseland <[EMAIL PROTECTED]> [2008-10-22 20:48:24]:
> >
> >> On Wednesday 22 October 2008 01:09, NextGen$ wrote:
> >> > * Matthew Toseland <[EMAIL PROTECTE
25 matches
Mail list logo
