I think one of the first design parameters you need is how much data. If
the amount of data is small, then you can get away with xml. If it starts
slowing down you need to move to a different format such as a custom binary.
XML - quick to code, easy, the customer can own their own data and work it
out themselves if needed. Can be slow as data increases.
Binary - slower to code, needs a lot of rules up front, speedy in use, needs
custom export as xml functions (perhaps)
In both cases you end up loading all the data into memory, so there's a real
upper limit. If you get into paging it gets too complex and it's time for a
real data provider.
XML suffers from having to parse the file, looking for special characters,
then a lookup match for each field for each record.
Binary on the other hand you can write the length of each record as the
record prefix so you just read in chunks of bytes, splitting each field
either at fixed length (really quick) or having variable length prefixed
with their length.
Eg
<User>
<FirstName>Greg</FirstName>
<LastName>Keogh</LastName>
<User>
Becomes
30 4 Greg 5 Keogh
Where the 30 is the first dword (4 bytes), 4 is the next (4 bytes), then
parse the string which is 4 Char (wide is 8 bytes) next 4 bytes is the value
of 5, then parse the 5 chars (10 bytes)
The problem with doing binary is if they change the schema, it's a real PITA
to change the code.
|-----Original Message-----
|From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-
|boun...@ozdotnet.com] On Behalf Of Greg Keogh
|Sent: Tuesday, 4 June 2013 12:25 PM
|To: ozDotNet
|Subject: Lightweight database
|
|Folks, an app has just grown with a new feature that needs to store of
"users",
|"jobs" and "reports" and the joins between them, If I was using SQLite it
would be
|3 tables with joins. However, rather than use SQLite this time I'd like to
consider
|an alternative that's even more lightweight to setup and use. The app does
not
|currently use any database technology and the guys managing the project are
|actually scared of them.
|
|Can anyone recommend an in-process database (not necessarily relational!)
that
|is has a friendly managed API, small footprint, not too complicated and is
easy to
|get going? I know this is a lot to ask, but there may be some NoSQL options
|around that I'm not aware of. The most important issues for me are: (1)
Minimal
|dependencies (2) Simple managed API.
|
|I'm running a few web searches now for such things, and I can see Redis,
Mongo,
|Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on and
on.
|There are too many choices and it would take many days of hard slog to work
out
|which one would be suitable. So perhaps someone has already been through
this
|process?!
|
|I've been tempted many times over the last 10 years to write a pure managed
|single-file database with indexes, and nothing much else (no transactions,
no
|client-server, no schemas, etc). However, I decided to leave it to the
experts, and
|it looks like there are too many of them, and they all over-engineer their
works.
|
|Cheers,
|Greg K
