Patrick Galbraith wrote:
Wow,
I turn away from my mail for a day or two and come back to find a
deluge of all these passionate emails and fervor. You'd think this was
a discussion on religion or politics. I suppose in a sense it is.
That's the worst thing about writing a book is that I often miss these
discussions!
Jim, if you had a magic wand, what would be the best way to store
BLOBs-- say if you were to be writing a DB app from scratch-- vs. the
way it is now implemented (pointer in server record) ? What is the
"right" way, and how do you deal with all the issues that people bring
up such as backup, etc, that you contend are issues only because they
were implemented wrongly in the first place?
I sometimes wonder about the issue of whether your (one's) RDBMS is
good, it should eat it's own dogfood by using itself for it's own
information as is the case with information schema. The age-old
question of whether to store BLOBs in the DB or not seems to be in
this category of philosophical arguments.
Good to hear from you, Patrick. I understand we'll see you this weekend.
Having gotten somewhat a professional rut of writing rdms after rdms
after rdms, I faced that question in Rdb/Eln, Interbase,
Netfrastructure/Falcon, and now Nimbus. This is how my thinking has
evolved.
In Rdb/ELN and Interbase/Firebird, small blobs and large blob headers
were intermingled table rows on data pages, preferably on the same
page. For large blobs, the header was either a vector of page numbers
of the blob pages, or a vector of pointer pages to data pages. The
theory was that when the blob was reasonable small, it could be fetched
for free.
The actual record contained a blob id (record number) -- essentially a
pointer to the blob. When the user requested a blob, he actually got
the blob id. When the blob was referenced (Rdb and Interbase/Falcon,
like Java, have a corner of the API dedicated to blob handling to enable
handling of blobs too large to fit in memory).
The counter argument is that intermingling blobs reduced the density of
rows on page, slowing down large scans that were unlikely to touch to
blobs at all. Netfrastructure (from which Falcon derived) was first and
foremost a Web application platform for which blobs figured prominently
as jpegs, Java classes, Word documents, and the like. On the other hand,
there were other mechanisms that came into play:
* Images (jpegs, pngs, etc.) and static downloadable things (PDFs)
were fetched and cached by the Web server module.
* Things like Word documents (used by humans to manage content)
needed to be kept, but were translated to wholesome HTML on upload
* Java classes were referenced at most once per server instance
This translates into many frequently referenced tables containing blobs
that themselves were infrequently referenced. A good example worth
looking at is the Images table. During upload, images were extracted
from documents, stored in the image table, and the image replaced with a
reference to the image table. During page generation, an image
reference was replaces with a file reference to the image on the Web
server, and the image name appended to a custom http header. The Web
server module would run through the header, determine which
images/downloads were already resident on the Web server, and do a
database query to fetch the ones that weren't. This guaranteed that the
Web server would fetch an image blob at most once. On the other hand,
it meant that page generator made more than a few references to Images
table, but never actually references the image blob itself. A little
complex, perhaps, but it worked like a charm. Image names were assigned
by a trigger (a popular drizzle non-features) from a sequence.
So Netfrastructure and Falcon have two independent data sections per
table -- on for rows and one for blobs. They tend to be co-linear by
happenstance, but nothing goes out of its way to ensure that. From a
on-disk structure perspective, the row and blob data sections were
identical, though Falon uses different code paths to achieve critical
de-optimization (oh, well).
I should probably note that all over the above systems -- Rdb/Eln,
Interbase, Firebird, Netfrastructure, and Falcon -- are careful during
update operations to leave unmodified blobs in place. Only when a
record or record version is garbage collected are unloved blobs
deleted. In specific, this means that multiple record versions can (and
almost always do) point to a single blob.
Nimbus retains the Falcon philosophy is separate data spaces for rows
and blobs. Unlike the precursors, however, the two types of data
section are handled differently. The primary reason that that records
have versions, format version number, and subtypes (Nimbus implements
the semantic data model). Blobs have none of these, and benefit from a
simpler structure.
I'd be happy to go on at even greater length at conference this weekend,
particularly is someone else is buying. Paul, do you remember who
picked up the last round at the UC?
I think it's sad that drizzle is opting out of the Web, but database
guys have been rather thick from the beginning. It's just so hard to
think past the card reader...
029 anyone?
--
Jim Starkey
President, NimbusDB, Inc.
978 526-1376
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
