Dear Matthew,
The difference between a relational database and a hierarchical one (or,
for that matter, a polymorphic one like FileMan) has to do with how it
models the world.
All databases exist to record an abstract model of pieces of the world.
Databases are usually structured as files (or tables or classes), each
of which lists entities of a similar kind, such as patients, or drugs,
or visits. Just as the file represents a category of entities, so each
record (or entry or row or object or instance) in that file represents a
specific entity, such as a specific patient, a specific drug, or a
specific visit. Databases, files, and records are not the real things
they represent, only abstract representations of them. An entry in the
patient file is not a real patient, but an abstraction of a patient, a
metaphor for that patient. Very much as with poetry, the more closely
that metaphor matches the important parts of the real thing it
represents, the more powerful the metaphor, the more meaningful, and
from the perspective of medical informatics, the more likely it is to
assist in improving patient health. Whether you have the right
information and whether you have organized it into the right metaphor is
largely dictated by how that information will be used--that tells you
which operations can be inefficient and which need to be efficient,
which tells you how to balance the tradeoffs that are always involved.
A good database designer chooses apt metaphors that match well the kinds
of information the clients need to record. The strategic part of that
choice involves selecting the right database paradigm; the tactical part
is using that paradigm effectively. WHICH data a file records is up to
the file designer, but HOW that data is stored is up to the database
paradigm you choose (relational, hierarchical, network, polymorphic,
object-oriented, etc.). As with successful adaptation in nature, the
secret to success lies not with rigid orthodoxy but with responsive
flexibility, varying your approach to let each problem dictate its own
best solution.
Relational databases represent files as tables, records as rows, fields
as columns. This is the spreadsheet's view of the world. In truth,
spreadsheets are excellent for certain categories of problems (e.g.,
inventories of parts), and terrible for others (e.g., Beethoven's
Pastorale piano sonata). The relational database is no more the perfect
solution to every problem than is the hammer. Neither is it the wrong
solution for every problem. It must be used appropriately, to solve
those problems for which it is well-suited.
Relational purists, those who insist every database problem must be
solved relationally, love the reductionist simplicity of having only a
single metaphor for all problems, and argue that by reducing all
problems to this common form you increase interoperability. This
assertion is false. When used universally and rigidly, the relational
metaphor becomes a Procrustean bed, stretching short people and cutting
off the legs of tall people so they all fit "perfectly" in the same size
bed. A relational solution is most apt for atomic information bound by
simple keys, or for groups of such entities likely to be sorted
different ways at different times, in which there is no clear way they
are usually organized. It is weak at representing asymmetrical
relationships, and does not represent behavior at all (making it
incompatible with the object-oriented approach to modeling the world has
increasingly favored over the last quarter century). The things left
out of the relational model are just as important to creating a good
model as are the things it includes, and often turn what may appear to
be the trivial problem of making two relational systems share data into
a nightmare of hidden assumptions and missing relationships.
Monomaniacal devotion to the relational model (or any one technology) is
based on the common but erroneous belief that most software problems are
technological problems, that using a common technology removes the most
important barriers. In actuality, software is a human communication and
relationship problem far more than a technical one, consisting of
struggling to understand and communicate our needs, first to each other,
then to the computer and whoever maintains the code over the decades
that follow. Arguing that common use of a relational model solves
interoperability is like arguing that world peace has been solved by
having all human beings share the same DNA; the proposed solution does
not match the context of the problem.
FileMan is a polymorphic database management system. You can use a
relational approach when that is appropriate, hierarchical when that is,
and so on. It does not fully implement any of these paradigms, but it
does enough of most of them to give us a reasonable solution to most the
database problems we have faced over the last twenty-eight years.
In VistA, one place we use a relational model is for information
relating to visits, because the priority of the relationships among the
many visit-related entities is roughly symmetrical--you are just as
likely to look up visits by date as you are by patient or by doctor or
by any of a number of other things. Since no one of these entities is
the clear majority choice for how to organize the rest, we flatten them
all into a collection of separate V files that all point to the shared
Visit file. As is often the case with relational solutions, this is as
flexible as possible, but using the resulting files is then
labor-intensive, since the software or users must correctly combine the
files they need to get the right results. This was a good use of the
relational model by Indian Health Service in RPMS, which the VA had the
good sense to adopt into VistA.
One place we use a hierarchical model is for information relating to the
menus a user can choose from. To simplify a bit from reality, in VistA
all options are collected into menus, which may themselves be collected
into other menus and so on; the resulting structure looks something like
a tree. To simplify a bit more, each user is assigned a primary menu
that contains all the options she needs to do her job. The software
that uses this option information is called MenuMan, and it uses this
file to solve several different problems that actually require it to use
more than one database paradigm for the same data.
When a user is signed into VistA, MenuMan has to make sure she can reach
all the options in her primary menu tree, but only those options. For
example, if a user tries to jump to an option, MenuMan has to read the
option she wants to jump to and then decide whether that option is
indeed somewhere among the options she can reach from her primary menu
(and its children, and their children, and so on). This is very much a
tree operation, not a table operation, and storing this in relational
tables would make MenuMan slow, since it would have to recompute all the
interrelationships among the options every time it tried to solve this
problem. Instead, MenuMan builds a tree for each primary menu that
contains all the interrelationships, and builds an index on that tree to
quickly identify 1) which options are contained within each tree, and 2)
if so, how do you navigate through the tree to reach those options. A
partially hierarchical model is exactly right for this problem, since it
is about traversing trees, so that is what VistA uses in its compiled
menu trees in the ^XUTL global.
On the other hand, options are also managed individually--locked or
unlocked, changes to the menu text, sending out new versions if they are
patched, and so on. For this kind of management, an option's location
within the compiled menu trees is unimportant. All that matters is
being able to quickly and independently find and manipulate each
option. For this a relational approach is appropriate, so VistA stores
options partially relationally in the Option file. Indeed, although all
of the information about each option is stored in the Option file, some
of it is stored a second time in the menu trees that VistA copies out of
the Option file and reorganizes--the same information is stored both
relationally and hierarchically because MenuMan needs it one way for
some operations but not others. This is more or less the right solution
to MenuMan's problems, but the software industry's warring camps would
instead choose to twist the solution one way or the other to make it fit
their ideological loyalties. The VistA way is not to take one side or
the other nor even to remain neutral, but to advocate an end to the war
so the two sides can get on with intermarrying to produce healthier,
prettier, and hopefully wiser children.
The software industry has a huge failure rate. It is in the midst of an
ongoing, growing, forty-year problem called The Software Crisis, in
which as the costs of hardware plummet and the power of that hardware
increases geometrically, the costs of software are rising, as is the
failure rate. No one has solved or shows any immediate signs of solving
this industry-wide dilemma, but everyone who admits to the existence of
this crisis claims to have a solution, and all of the solutions are
technological. Use SQL and all your troubles will go away. No, use
C#. No, use brand X instead. These approaches cannot solve the crisis
because none of them deal with the fundamental issues. They all propose
that The Software Crisis is a werewolf, so all you need is a magic
bullet, which by coincidence they just happen to have with them and are
willing to part with in exchange for your savings account. An expensive
hammer can indeed solve every problem; it just solves most of them
badly, and so do these many technologies-du-jour. Each has a small
number of proper uses and a large number of inappropriate uses, and each
is proposed to be the only solution you need for entire classes of problems.
When an entire industry is this deluded, you can expect crazy behavior.
For example, if the American College of Physician Executives says the
state of medical informatics is a disaster, except for VistA, which is
great, you can expect the industry to complain as one that no, that
cannot be so, for look: VistA is not written in brand X. No matter how
bad most medical software is, VistA's success cannot be as interesting
as which technologies it uses. Failure according to the prescribed
technology is the only permitted option. Succeeding with a technology
that everyone agrees cannot work must be a mirage. Look away, before
the VistA heathens corrupt you with their non-brand-X ways.
So let me play Ariadne to your Theseus and offer you the golden clew
with which to escape the labyrinth: ignore technology; focus on
features. The short answer to whether you should choose a relational or
hierarchical model is: don't. Forget about technology. Leave that to
the programmers. Focus instead on the problems you are trying to solve,
the features you want, the kinds of information you need. That is by
far the harder problem, and something the technologist cannot do for
you. If instead you focus on the technology, not only are you
artificially constraining the solution (thus increasing the chances of
failure) but also neglecting the harder problem that only you can do
(thus ensuring failure).
Yours truly,
Rick
PS: "Short" answers to your other questions:
1) Is Mumps a serious limitation to complete EHR functionality, code
maintainence, HL7, or PMS interfacing?
How many completely functional EHRs can you name that are not written in
MUMPS, whose functionality even comes close to VistA's? As far as I
know, MUMPS is the only programming system designed specifically for
medical systems development. Standard MUMPS cannot be used to generate
sophisticated graphical interfaces, but it can be used to communicate
with programming languages that do. No language does all things, nor
should, nor can. Programming requires mastery of multiple languages,
and the core language must be carefully chosen. The best reason for
using Standard MUMPS as VistA's core language is that it is decades too
late to do anything else. VistA is already written in Standard MUMPS,
and it took twenty-eight years to get this far. Replacing Standard
MUMPS at this point is an irresponsible waste of resources that could
instead be used to easily extend VistA to save lives. It is like
arguing that brick is passe, so we should shut down New York City for
fifty years so we can remove all the brick and replace it with something
more popular. Replacing Standard MUMPS to improve code maintenance (for
example) is like replacing my DNA so I can learn to play the
flute--unnecessary and irrelevant. Standard MUMPS is VistA's DNA.
Honestly, though, why does anyone who is not a programmer care what it
was written in? What is Mac OS X written in? How about Microsoft
Word? Google? Quicken? The Sims? Do you feel competent to evaluate
which programming language is ideal for a given problem domain? After
twenty-one years of programming practice and study, I do not know beyond
my chosen field of medical software. I certainly do not feel competent
to choose among surgical instruments. I could spend time trying to
teach nontechnical people how to evaluate programming systems enough to
understand why VistA had to be written in Standard MUMPS (something even
most programmers evidently do not understand), or they could spend a
fraction of that time teaching me what they need VistA to do for them.
If I can get VistA to do all those things for them, then in the end who
cares what language it is written in?
3)How hard is it for non-Mumps IT personnel to learn Mumps/VistA and are
there enough experienced VistA programmers (or former VistA programmers)
to consult or be hired to non-VA projects?
It is easy to learn Standard MUMPS, but impossible to master VistA.
Like the art of medicine itself, VistA is complex beyond human
comprehension--no, I am not kidding or exaggerating--and no one person
will ever know it all anymore. I have been programming with VistA for
twenty-one years; there are a couple areas of the code I know better
than anyone, several I know as well as the other experts, and a dozen or
so I know reasonably well--out of 80-120 software packages, depending on
where you draw the lines. Most of VistA I know by its patterns and
common structures, and maybe a few basic architectural features per
package, but for most of VistA I am the wrong person to go to. There
are whole packages I know only by name, whose purpose I can only guess
at. So it is with all the VistA gurus. None of us pretends to know it
all or even most of it. We work together as a community, sharing out
the vast scope of work that is VistA among ourselves.
To address your central concern, our tradition is to grow our own new
Standard MUMPS and VistA programmers from among its users, because we
have discovered it is far easier to teach a nurse to program than to
teach a programmer to practice medicine; the nurse has already mastered
the difficult part. It takes mere minutes to start writing Standard
MUMPS code, as with any programming system worth discussing, but a day
or two to introduce the basics, a week or two to introduce them fully,
and a month or two to become fully comfortable with the programming
system. It takes experienced programmers longer to learn Standard
MUMPS, because it is not like most other programming systems, and they
spend years whining about it instead of buckling down and coming to
grips with it on its own terms. Learning Standard MUMPS is like taking
small steps over very deep crevices; it is easy but unnerving for some.
Learning to program with VistA takes longer, and should happen in two
phases. First the programmer needs to learn our programming standards
and conventions and common calls. Then the programmer needs to pick a
package and focus on it for a long time, moving from simple assignments
to more complex ones. It is best if the student began as a user of that
package, then graduated to being an application coordinator (a kind of
super-user) for it, before learning to program with it. Becoming an
expert user of any reasonably sophisticated VistA package takes years.
Once an application coordinator starts training to become an information
manager, starts working on supporting and extending a package at a site,
every year they keep at it improves their skills with the package
measurably. Those who have worked with a package for ten years or more
are easy to tell apart from those who have only been doing it a few years.
There is a lot more I could tell you about the expertise lifecycle, how
it is structured, where to find VistA experts and how to grow your own,
but I am trying to keep this postscript "short."
4) What other concerns should we have regarding adopting VistA?
Above all, it is a serious commitment. It is free as in freedom, not
free beer. It will save lives, not time. You cannot do this alone; you
may think you can, but sooner or later you will run aground without
help. To succeed with VistA you need the community in ways you cannot
imagine, but the good news is they will welcome and help you, and it
will not be long before you can return the favor. The big hurdle with
VistA is that anyone used to making medical informatics decisions has
learned how to prepare for defeat, to choose vendors on the basis of how
deep their pockets are (so you can sue them when the project collapses
in failure) instead of on the basis of their expertise and customer
service, to choose based on marketing flair and reputation. VistA is
the real deal, and for that very reason may be difficult to recognize as
such by customers who have learned only how to choose among flashy
failures. VistA requires the "customer" to become a partner, a
collaborator, words that have all but lost their meaning in the modern
marketplace; when you become fully engaged with the VistA lifecycle, you
will come to understand that if VistA is broken it is your fault as much
as ours, that it is your responsibility to hold the developers and the
software to a high enough standard to meet your needs. Unlike with most
commercial software products, you will rewarded instead of punished for
engaging in the critique, review, and even development of VistA.
There is an endless amount to learn about it, and it is under continuous
development, continuously patched, continuously changing, as it must
be. The static details of VistA are less important than the living
process by which its users drive development through their continuous
stream of suggestions and complaints. You will not and cannot
appreciate how true that is until after you have been involved with
VistA for years. The secret to its success is no secret, not specific
features, nor the technology used. It is the dynamic, hyperactive
software lifecycle that engages the creativity of tens of thousands of
users to mold the software over and over so that the longer you wait the
better it gets, as opposed to most software which is static by design,
updated at best occasionally, and obsolesces with time.
That inversion of the norm, emphasizing change instead of stasis, is a
repudiation of the core beliefs of the software industry and much of
academia, and as such you must expect to hear an endless stream of
irrelevant complaints lodged against VistA, usually by those who have
not used it. From those who do use VistA, you will generally find an
endless stream of relevant complaints together with strong loyalty.
Like any experienced VistA professional, I can and will criticize VistA
up one side and down the other for hours on end. We like it in part
because we know its faults, and so we try to prioritize the work most
important to us; when the lifecycle is healthy, if something is broken
it's because we think it's more important for the developers to work on
something else first. Those who fully engage in the VistA lifecycle
know VistA's flaws far more intimately than its critics, but they know
its strengths, too, and they can point to features in the software that
they personally first brought to the attention of the VistA development
community. In a way most software will never be, VistA really does
belong to its users, and they know it.
So when I echo other writers in saying VistA is free as in freedom, I
mean as in the responsibility that comes with true freedom, the
expectation to interact with it as an adult, taking responsibility for
what we need from it, and helping to chart our own future. To be blunt,
many people do not want that, prefer the simplicity of having limited
choices imposed upon them by someone more powerful, to recreate the
false security of childhood. Others do not mind the responsibility, but
cannot spare the time to learn a fully featured medical informatics
system, not even just the few parts of the few packages they would use
in their medical roles. In general, we find the truth about VistA
properly screens our potential clients; the right people respond well to
the challenge and possibilities of VistA, and enrich the community and
its software when they engage with us.
PPS: If I sound opinionated in the above, it is because experience has
made me so. For the first half of my career with VistA, I thought it
was probably nothing special, that every hospital system must have
something comparable. My disgust with the state of the art and
appreciation for VistA has accumulated over the years through a series
of disillusioning exposures to how most medical software works--or
doesn't. I do not know if there is even one feature in VistA that it
does better than any other system, but no other system seems to be able
to combine them all into such an integrated architecture driven by such
a potentially responsive software lifecycle. The experience of patients
in New Orleans in the wake of Hurricane Katrina would seem to be the
latest dramatic illustration. I have gradually arrived at the
surprising conclusion that my friends and I are working on something
unusually cool. Who knew?
Aylesworth Marc A Ctr AFRL/IFSE wrote:
Relational DBs always reserve the total space for a record for faster
searching. That is why when creating a db you have to tell the DB what type
and size of each field is.
Thanks
Marc Aylesworth
C3I Associates
AFRL/IFSE
Joint Battlespace Infosphere Team
525 Brooks Rd
Rome, NY 13441-4505
Tel:315.330.2422
Fax:315.330.7009
Email: [EMAIL PROTECTED]
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Gregory
Woodhouse
Sent: Monday, September 19, 2005 3:06 PM
To: hardhats-members@lists.sourceforge.net
Subject: Re: [Hardhats-members] Re: Wiki additions: [was: web evaluation
tool]
This is actually not true. flat tables are the abstraction presented
to the user (or programmer) with relational databases, but the
underlying storage model is, more likely than not, based on B-trees,
just as it is in MUMPS. This can be confusing, because the basic
abstraction presented to the user in MUMPS is multi-branching trees,
but that abstraction is not the same as the underlying storage model.
===
Gregory Woodhouse
[EMAIL PROTECTED]
"The most incomprehensible thing about
the world is that it is at all comprehensible."
--Albert Einstein (1879-1955)
On Sep 19, 2005, at 11:56 AM, Kevin Toppenberg wrote:
2) What is the fundamental difference between a relational database
and
a hierarchical database and how does that effect the end-user? (Should
we even care? If so why?)
Another difference is the way the data is stored. M data is stored in
b-trees, as compared to flat tables (I believe). This leads to faster
data acess, and less CPU power needed.
Also, the database in M is called by some a "sparce array." This
means that there are no "blank spaces" left for data to be later
filled into. So with M, if there is no data present, then no space is
wasted. I find this to lead to many many fields being defined for a
given file. With a traditional database, having all these fields with
empty/wasted space, would lead to huge database files. But with M,
one can can store years of patient information on a relatively small
disk.
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Hardhats-members mailing list
Hardhats-members@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/hardhats-members
