memory pool, rfc
Hello: I have some free time and want to do an memory pool. The idea is to have a memory zone of N KB (or several MB) compressed in memory. I have fast compression algorithms now that can release under BSD licence that are faster than hd i/o, so it take less compress/decompress a memory zone than read/write it to disk. I don't know if it already exist for FreeBSD, so if it's already done i'll try to improve it. - Each memory chunk is compressed separately, so i can decompress and use one without decompress anyother one. - In 4KB chunks of text i get 50-60 % compression (avg 2 - 1.6 KB result) - In 4KB chunks of binary (application code) i get 30-40 % compression (avg 2.8 - 2.4 KB result) - In both cases, results may vary depending on data type and chunk size, greater implies better compression - Speed once implemented will be very fast. Current speed hogs a PATA 133 disk. For what can be used? - Memory pools in applications (like malloc) - Ram disks - Disk Cache (permit bigger disk cache) - 'On the fly' filesystem compression (and it takes less read/write compressed data than non-compressed) - Perhaps add it as Virtual Memory swap cache? - Other Don't point me to zlib or libbzip2, they are on another league and are much slower than my code. Thanks ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
I have some free time and want to do an memory pool. The idea is to have a memory zone of N KB (or several MB) compressed in memory. I have fast compression algorithms now that can release under BSD licence that are faster than hd i/o, so it take less compress/decompress a memory zone than read/write it to disk. I don't know if it already exist for FreeBSD, so if it's already done i'll try to improve it. Perhaps you should try the freebsd-hackers list... this list is generally for user operational questions. Don't point me to zlib or libbzip2, they are on another league and are much slower than my code. Ouch! ...you are not trying to sell anything are you? It may be in your best interest if you proceed to the hackers list, to initiate conversation in a way that explains how your code will benefit a cause, not slam other people (and their work) that are already established! Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
At 00:38 01/11/2007, you wrote: Ouch! ...you are not trying to sell anything are you? It may be in your best interest if you proceed to the hackers list, to initiate conversation in a way that explains how your code will benefit a cause, not slam other people (and their work) that are already established! Steve No, i don't want to sell anything to anyone. I'm already on hackers list but has very low traffic (9 messages last 5 days) and this is a question list no ? ;-) I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? answer. If other developers thinks that they need my rfc i'll add my code to FreeBSD. The point to zlib / libbzip2 is because when i talk about compression *always* someone says Why don't use zlib? Short answer No, Medium Answer: deflate is 15 years old and there are better and faster algs now. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
On 11/1/07, Eduardo Morras [EMAIL PROTECTED] wrote: At 00:38 01/11/2007, you wrote: Ouch! ...you are not trying to sell anything are you? It may be in your best interest if you proceed to the hackers list, to initiate conversation in a way that explains how your code will benefit a cause, not slam other people (and their work) that are already established! Steve No, i don't want to sell anything to anyone. I'm already on hackers list but has very low traffic (9 messages last 5 days) and this is a question list no ? ;-) I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? Even though you might get a little critized for being slightly off topic (at least until you add your code) try -current ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? answer. If other developers thinks that they need my rfc i'll add my code to FreeBSD. Agreed, so could it be added as a port, or can you license the code with the BSD license and post a link to it? To be honest, from what you say about your application, it sounds beneficial. I personally would be willing to try it on one of the boxes that I boot from removable USB disk and run the entire OS in memory, with no hard disk whatsoever. The point to zlib / libbzip2 is because when i talk about compression *always* someone says Why don't use zlib? Short answer No, Medium Answer: deflate is 15 years old and there are better and faster algs now. Understood. Can you provide a compilable product with reasonable documentation that can be tested? Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
And is it better than bzip? This is in essence why I tried to lead this thread off of this list. The OP stated nothing of being 'better'. On top of that, the OP was referencing libraries, not applications. The OP is trying to get his own code under the BSD license and that is great. Asking whether his app is 'better' than something else is a VERY relative question. 'Better' or 'faster' is something that you will have to answer for yourself after you've implemented it and given it appropriate benchmarks within your particular environment. Just like everything else in FreeBSD, there are always multiple ways to do it. This, as anything else, could possibly prove to be a way to do it 'better' for what you need it for. This truly was not meant to be a flame war. Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
No, i don't want to sell anything to anyone. I'm already on hackers list but has very low traffic (9 messages last 5 days) and this is a question list no ? ;-) I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? Even though you might get a little critized for being slightly off topic (at least until you add your code) try -current I apologize if I came across as critical, I did not intend it that way. Perhaps if the OP goes to -current with it, this thread could be referenced (in addition to the inclusion of code). Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
On 2007-11-01 01:00, Eduardo Morras [EMAIL PROTECTED] wrote: I'm already on hackers list but has very low traffic (9 messages last 5 days) and this is a question list no ? ;-) This is a 'general' questions list, with a fairly high level of traffic. Many FreeBSD committers and team affiliates are subscribed, but there are still a lot of knowledgeable people who are respected team members, but are *not* subcribed here. Maybe in a more appropriate list you can get a greater amount of visibility with people who can help your contributions to be integrated with the main source code tree of FreeBSD. I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? answer. If other developers thinks that they need my rfc i'll add my code to FreeBSD. It's hard to tell in advance, without actually *seeing* something in the form of a proposal, or at least an implementation. The FreeBSD team works in an open manner, and we take pride in the fairly well established process of development we use. Everyone is invited to support, extend, fix bugs, implement new features, or otherwise improve FreeBSD, as long as they are prepared to put in some effort to fit within the team. Having said that, all you would have to show us would be at least one of: * A detailed proposal of the changes, optionally including a justifiction of why you think the changes are necessary and how they will improve the user experience of the people who comprise our userbase. * A pilot implementation of the new features you described. It doesn't have to be complete; it doesn't have to precisely match the existing style of the code; it doesn't have to be performing blazingly fast, if there are plans for tuning; it doesn't even have to be compilable; as long as you are prepared to help us fix all the details. Then, I'm sure that if the improvements you are working on look interesting, you stand a better chance of soliciting comments, feedback related to the way the changes can go into FreeBSD, help with the testing process of the source integration, and so on. This means that the next reasonable step would be to make available at least a proposal or source patch, and post it to one of the developer lists of FreeBSD. The freebsd-hackers list would be nice. The point to zlib / libbzip2 is because when i talk about compression *always* someone says Why don't use zlib? Short answer No, Medium Answer: deflate is 15 years old and there are better and faster algs now. That's one way to describe zlib. Perhaps not the only one, but still a valid opinion to hold, so let's go down that path. It doesn't sound like a productive use of everyone's time, unless we have a solid plan for a better zlib replacement in place :-) Giorgos pgpHhBKCvTiOW.pgp Description: PGP signature
Re: memory pool, rfc
At 01:38 01/11/2007, you wrote: This is a 'general' questions list, with a fairly high level of traffic. Many FreeBSD committers and team affiliates are subscribed, but there are still a lot of knowledgeable people who are respected team members, but are *not* subcribed here. Maybe in a more appropriate list you can get a greater amount of visibility with people who can help your contributions to be integrated with the main source code tree of FreeBSD. Understood I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? answer. If other developers thinks that they need my rfc i'll add my code to FreeBSD. It's hard to tell in advance, without actually *seeing* something in the form of a proposal, or at least an implementation. . testing process of the source integration, and so on. This means that the next reasonable step would be to make available at least a proposal or source patch, and post it to one of the developer lists of FreeBSD. The freebsd-hackers list would be nice. I can prepare a malloc-like pilot implementation and detailed paper. It doesn't sound like a productive use of everyone's time, unless we have a solid plan for a better zlib replacement in place :-) You can check http://www.maximumcompression.com/index.html and http://cs.fit.edu/~mmahoney/compression/text.html for the state of the art lossless compressors. Giorgos Thanks ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
At 01:45 01/11/2007, you wrote: And is it better than bzip? Depends on your concept of better. It doesn't compress more, it compress/decompress faster, it's designed for memory to memory compression. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
On 2007-11-01 02:06, Eduardo Morras [EMAIL PROTECTED] wrote: At 01:38 01/11/2007, you wrote: This is a 'general' questions list, with a fairly high level of traffic. Many FreeBSD committers and team affiliates are subscribed, but there are still a lot of knowledgeable people who are respected team members, but are *not* subcribed here. Maybe in a more appropriate list you can get a greater amount of visibility with people who can help your contributions to be integrated with the main source code tree of FreeBSD. Understood I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? answer. If other developers thinks that they need my rfc i'll add my code to FreeBSD. It's hard to tell in advance, without actually *seeing* something in the form of a proposal, or at least an implementation. . testing process of the source integration, and so on. This means that the next reasonable step would be to make available at least a proposal or source patch, and post it to one of the developer lists of FreeBSD. The freebsd-hackers list would be nice. I can prepare a malloc-like pilot implementation and detailed paper. That would be fantastic! It doesn't sound like a productive use of everyone's time, unless we have a solid plan for a better zlib replacement in place :-) You can check http://www.maximumcompression.com/index.html and http://cs.fit.edu/~mmahoney/compression/text.html for the state of the art lossless compressors. I will, thanks for the references; they look interesting :) pgp4BbuIrO69j.pgp Description: PGP signature
Re: memory pool, rfc
On 10/31/07, Steve Bertrand [EMAIL PROTECTED] wrote: I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? answer. If other developers thinks that they need my rfc i'll add my code to FreeBSD. Agreed, so could it be added as a port, or can you license the code with the BSD license and post a link to it? To be honest, from what you say about your application, it sounds beneficial. I personally would be willing to try it on one of the boxes that I boot from removable USB disk and run the entire OS in memory, with no hard disk whatsoever. The point to zlib / libbzip2 is because when i talk about compression *always* someone says Why don't use zlib? Short answer No, Medium Answer: deflate is 15 years old and there are better and faster algs now. Understood. Can you provide a compilable product with reasonable documentation that can be tested? Steve And is it better than bzip? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory pool, rfc
At 01:46 01/11/2007, you wrote: No, i don't want to sell anything to anyone. I'm already on hackers list but has very low traffic (9 messages last 5 days) and this is a question list no ? ;-) I think that in this list are FreeBSD *gurus*/hacks too which could say a try it or a are you crazy? Even though you might get a little critized for being slightly off topic (at least until you add your code) try -current I apologize if I came across as critical, I did not intend it that way. I asked because i want to ear critics and new points of view. Steve Thanks ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
