On 11/11/05, Matthew Toseland <toad at amphibian.dyndns.org> wrote: > A FEC code, such as the one we use for redundant splitfiles, is a code > which takes k blocks of input data and produces n blocks of output data > (the first k of which are normally the input data). > > Onion FEC's CPU usage is roughly quadratic with k. > > It takes 3.4 seconds on my box (XP 2800+) to encode one segment (after > setup; 4MB input, 6MB output), with the current 128/192 code. This is > the same as we used in 0.5. > > In 0.5, for a long time all encoding was done in pure java, which is > many times slower, and all encoding had to be done at the start of the > splitfile insert. In the 0.7 code, we start inserting the data blocks > immediately while running the splitfile encode, segment by segment, in > parallel, and adding the new check blocks when they are done (starting > with the last, shortest, segment). > > Therefore I suggest that it may be sensible to consider some other > configurations. > > Issues: > - Speed reduces by roughly a factor of 4 if either n or k is over 256, > because we have to use a 16-bit code. > - At present we use 50% redundancy. This is probably not necessary. > > PROJECTED ENCODE RATES > n k segsize time rate > 128 192 4MB 3.48s 1200kB/sec > 170 256 5.4MB 6.14s 880kB/sec > 192 256 6MB 7.83s 780kB/sec > 256 384 8MB 55s ~= 1m 147kB/sec > 512 768 16MB 222s ~= 3m 74kB/sec > 1024 1536 32MB 890s ~= 15m 37kB/sec > 2048 3072 64MB 3560s ~= 1h 18kB/sec > 4096 6144 128MB 14250s ~= 4h 9.2kB/sec > 8192 12288 256MB 57016s ~= 16h 4.6kB/sec > 16384 24576 512MB 228ks ~= 63h 2.3kB/sec > > Encode rates are relative to the original data. > Since we start inserting the original data blocks immediately, a codec > will be acceptable if its encode rate is over our expected insert rate. > Actually we have some leeway beyond that for the later segments, as we > have to insert the check blocks too, but lets assume single segment of > the maximum size. > > Note that the memory usage is 1/8th of the segment size with 4kB > stripes. > > IMHO very few nodes will be able to insert at much over 37kB/sec. But > some will, and there is CPU usage to consider too... do we want fast > encodes so that people can run games etc at the same time as inserts? > Realistically people are going to turn Freenet off while gaming... at > least the hardcore gamers are. Even if they had loads of RAM and could > nice it, they'd want to have really low bandwidth usage. Which buys us > time, because it slows down the insertion of blocks, and therefore we > don't need to have such a fast encode. > > 16 or 32MB segments seem to make sense to me. Does this make sense? I > can make segment size a parameter otherwise... > -- > Matthew J Toseland - toad at amphibian.dyndns.org > Freenet Project Official Codemonkey - http://freenetproject.org/ > ICTHUS - Nothing is impossible. Our Boss says so. > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.1 (GNU/Linux) > > iD8DBQFDdPm7HzsuOmVUoi0RAiTsAJ9WAvpigwzkUSmkf+uTJsaztE5kawCeMZKA > hONUnA8ivl6JAzHGPWFaXB0= > =LwCz > -----END PGP SIGNATURE----- > > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > >
What are the pro's and con's of using a small segment size (4/6 MB) verses 16/32 MB? Does using a small segment size limit our maximum file insertion size? -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire
