Re: Re: allocating 14KB memory per packet compression/decompression results in v
On 2005-11-04 11:14, Sergey Babkin <[EMAIL PROTECTED]> wrote: >Giorgos Keramidas <[EMAIL PROTECTED]> wrote: >>On 2005-11-03 22:56, kamal kc <[EMAIL PROTECTED]> wrote: >>> since i am using the adaptive LZW compression scheme it >>> requires construction of string table for >>> compression/decompression. So an ip packet of size 1500 bytes >>> requires a table of size (4KB + 4KB + 2KB = 12KB). >> >> I may be stating the obvious or something totally wrong, but >> couldn't the string table be constructed once instead of each >> time a packet goes down? It is my intuition that this would >> perform much much better than re-doing the work of the string >> table each time a packet goes out. > > No, the table changes as data is compressed. It records the > knowledge about the strings that have already occured in the > data. > > Keeping the table between the packets would improve the > compression but the packets would have to be transmitted > through a reliable medium since to decompress a packet you > would have to decompress all the preceding packets first > (essentially you get a stream compression). Ah, yes, I see now. You're right of course. I was thinking of something resembling a "compressed tunnel" when I wrote the reply, but that doesn't work with IP very well, unless some other sort of encapsulation is at work. > To keep the packets separate, the compression state > must be reset between them. > > But of course resetting the compression state does not > mean that the memory should be deallocated. Very true :) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Re: allocating 14KB memory per packet compression/decompression results in v
>From: Giorgos Keramidas <[EMAIL PROTECTED]> >On 2005-11-03 22:56, kamal kc <[EMAIL PROTECTED]> wrote: >> since i am using the adaptive LZW compression scheme it >> requires construction of string table for >> compression/decompression. So an ip packet of size 1500 bytes >> requires a table of size (4KB + 4KB + 2KB = 12KB). > >I may be stating the obvious or something totally wrong, but >couldn't the string table be constructed once instead of each >time a packet goes down? It is my intuition that this would >perform much much better than re-doing the work of the string >table each time a packet goes out. No, the table changes as data is compressed. It records the knowledge about the strings that have already occured in the data. Keeping the table between the packets would improve the compression but the packets would have to be transmitted through a reliable medium since to decompress a packet you would have to decompress all the preceding packets first (essentially you get a stream compression). To keep the packets separate, the compression state must be reset between them. But of course resetting the compression state does not mean that the memory should be deallocated. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Re: allocating 14KB memory per packet compression/decompression results in v
>From: kamal kc <[EMAIL PROTECTED]> >since i am using the adaptive LZW >compression scheme it requires construction of string >table for compression/decompression. So an ip packet > of size 1500 bytes requires a table of size (4KB + > 4KB + 2KB =12KB). > >further still i copy the ip packet > data in another data buffer (about 1.4KB) and >then compress it. > >So all this adds up to about 14KB. > >Right now i can't do with less than 14KB. > >as i said before the compression/decompression works >fine. but soon the kernel would panic with one >of the vm_fault: error message. Most likely you overrun your buffer somewhere and damage some unrelated memory area. >what would be the best possible way to >allocate/deallocate 14KB memory per packet without >causing vm_faults ?? The best possible way is to not do it at all. Allocate you 14KB buffer once and then reuse it for every packet. Obviously, your code would have to be either single-threaded, or synchronize the access to the buffer, or use a separate buffer per CPU. >is there anything i am missing ?? Also an extra memory-to-memory copy is a bad idea. It hurts the performance. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
