I did that Paul and proved that, for accessing user information and
adding users using a keyvalue store it reduces the load from the
database. The mysql server was using 30-60-90% of the cpu load
processing everything, while memcache only used 3%. So memcache is a
good system for storing small amounts of data. With the user
information implemented it was time to try and store messages.
Progress on the messages is, I can store messages into the keyvalue
store. Now I need to retrieve them.
But one of my problems was I was allocating too much memory for the
operations, then the memory needed for messages is far greater than
the memory needed for user data. so instead of having a user-data
buffer and a message buffer, I needed to make the original buffer more
flexible. I cant just keep redoing the same redundant things over and
over. Like the first time I had 3-4 functions that initiated
libmemcache, now I have one. That worked great for small key values,
but for messages, not at all.because messages can be from 2kb -
20-50mbs, wheres the small keys are usually no more than 20-50bytes
Maybe the 3 buffer structs linked to the arg struct is not good I
could do flexible arrays with the argument pointers pointing back to
the memory addresses of the character arrays. or just put a type def
around the three structs so they are defined and have there own types
so they are easily, noticed and used.
Creating a buffer that's
char buffer[65535][1800] right off the bat, doesn't feel like a good
idea that's 117963000 bytes when we only need 3 keys that total 5400
bytes.
creating a buffer that's
char buffer*=malloc((sizeof(char)*TOTAL_KEYS)*KEY_SIZES) I feel makes
more sense. Only allocate the space needed at the time of the function
call.
hmppph, thanks Paul.
On 04/08/2012 at 11:41 AM, Paul J Stevens wrote:On 04/08/2012 03:33
PM, skr...@hushmail.com wrote:
> Is it getting better? are am I going backwards?
I think you really need to step back now and then and ask yourself:
1) what is it I'm trying to do, really. Try to keep the scope of your
work narrow and focused.
2) how can I make my code cleaner, easier to read. Refactoring code is
a
great way to allow your mind to think about the first question, while
improving the overall design of your code
I know you think you can improve DBMail and make it scale by
leveraging
a key-value store. You maybe right, you maybe wrong. Of course I
understand you're still learning about the basics, but very often
getting the basics right will clear the way for more difficult stuff.
If I would build a key-value interface I would first setup an
interface
with really limited functionality. Ignore difficult stuff like
compression and chunking. Get it right with small data blocks first
that
don't need additional sit-ups to push and retrieve them from the KV
store. Keeping things simple and clean will force you to rethink now
and
when, but it will also keep things easier to understand and use.
Setting up such a basic interface will allow you to start thinking
about
practical application quickly and leverage the interface in the
overall
dbmail code-base to prove the added value of the new interface. If you
see results, this will affirm your intuition that your barking up the
right tree.
And last but not least: use unit-tests to prove the correctness of
your
code, and prevent future regressions. Look in the dbmail/test/
directory
for examples how to set them up. You will need to install 'check'
http://check.sourceforge.net/ to use them.
--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin
* Premium Hosting Services and Web Application Consultancy *
www.nfg.nl/i...@nfg.nl/+31.85.877.99.97
________________________________________________________________
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev_______________________________________________
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev
