* Daniel Cheng <j16sdiz+freenet at gmail.com> [2008-10-23 08:12:14]: > On Thu, Oct 23, 2008 at 6:49 AM, NextGen$ <nextgens at freenetproject.org> > wrote: > > * Matthew Toseland <toad at amphibian.dyndns.org> [2008-10-22 20:48:24]: > > > >> On Wednesday 22 October 2008 01:09, NextGen$ wrote: > >> > * Matthew Toseland <toad at amphibian.dyndns.org> [2008-10-21 20:53:51]: > >> > > >> > > On Tuesday 21 October 2008 16:24, nextgens at freenetproject.org wrote: > >> > > > Author: nextgens > >> > > > Date: 2008-10-21 15:24:47 +0000 (Tue, 21 Oct 2008) > >> > > > New Revision: 23014 > >> > > > > >> > > > Modified: > >> > > > trunk/freenet/src/freenet/client/ArchiveManager.java > >> > > > trunk/freenet/src/freenet/client/ArchiveStoreContext.java > >> > > > trunk/freenet/src/freenet/client/ClientMetadata.java > >> > > > trunk/freenet/src/freenet/client/HighLevelSimpleClientImpl.java > >> > > > trunk/freenet/src/freenet/client/Metadata.java > >> > > > trunk/freenet/src/freenet/client/async/ClientPutter.java > >> > > > trunk/freenet/src/freenet/client/async/SimpleManifestPutter.java > >> > > > trunk/freenet/src/freenet/client/async/SingleFileFetcher.java > >> > > > trunk/freenet/src/freenet/client/async/SingleFileInserter.java > >> > > > trunk/freenet/src/freenet/client/async/SplitFileInserter.java > >> > > > trunk/freenet/src/freenet/clients/http/WelcomeToadlet.java > >> > > > trunk/freenet/src/freenet/frost/message/FrostMessage.java > >> > > > trunk/freenet/src/freenet/node/NodeARKInserter.java > >> > > > trunk/freenet/src/freenet/node/TextModeClientInterface.java > >> > > > trunk/freenet/src/freenet/node/fcp/ClientPut.java > >> > > > trunk/freenet/src/freenet/node/fcp/DirPutFile.java > >> > > > > >> > > > trunk/freenet/src/freenet/node/simulator/BootstrapPushPullTest.java > >> > > > Log: > >> > > > more work on bug #71: *** IT NEEDS TESTING! *** > >> > > > It's still not backward compatible with stable but should be > >> > > forward-compatible ;) > >> > > >> > > >> > [...] see r23023 > >> > > >> > > Do we attempt to compress all files with bzip2 as well as gzip now? > >> Shouldn't > >> > > there be a max size configuration above which we don't try bzip2, > >> > > perhaps > >> > > unless asked to via FCP? bzip2'ing ISOs could take a really long time > >> > > ... > >> > > >> > I don't think we need one. Big files will take long to compress but will > >> take > >> > long to insert too. I think it's worth spending a few more CPU cycles to > >> > spare the insertion of a few blocks (plus their FEC blocks). > >> > >> I'm not convinced that this is acceptable from a usability point of view. > >> Maybe we can provide a progress bar within the compression phase? On the > >> new > >> UI it is proposed to separate downloads which are not yet finalised (i.e. > >> haven't fetched the last lot of metadata) from downloads that are... we > >> could > >> do something similar with inserts in compression. > >> > > > > Have a look to what I have commited. From now on the compression is fully > > serialized... We have one mutex, and only one compression job (just like we > > do for FEC encoding in fact) which means a even higher latency. > > It is feasible to insert some blocks of data while compressing? > Gzip, bzip2 and lzma all support streams. We can collect the output data > as we feed data to them. >
Right now we attempt to compress the full data using all the compression algorithms and we keep the smallest resulting bucket. How do you plan to chose the best-performing algorithm before actually compressing the data? I don't think that we can evaluate how well algorithms compress over a single segment: it's just too small. > As soon as we get enough compressed data for FEC, we can insert them. > This would be a great preformance improvement for large file on SMP. > That would involve rewritting most of the client-layer. > It this doable without changing the data format? > It's not about the data format; we insert the manifest at the end unless not told to by the earlyEncode parameter. IMHO we are debating for no real reason here: the real-time taken by the compression phase is insignificant compared to the time taken by the insertion process. Sure, trunk will take at least 3 times longer than current stable before it starts inserting anything; but is that a big deal? You will need real numbers to convince me here.
