mARK bLOORE wrote:

i had a design where i was essentially duplicating a folder tree by
storing the folder's path in a string, and the files in it in a
subview:

stuff[folderpath:S,files[id:I,name:S]]
(and other data, of course.  note that it is not fully heirarchical,
just two levels.)

i found that when there were many rows (say 100,000) it was very slow
to open.  it seems to go through them all to get some metadata on the
subviews.

Yes. With 100,000 subviews things can bog down. The empty case is treated specially, but still. A number of performance tweaks have been made over the past year to address this. More are needed, there is some over-zealous memory buffer allocation going on as well.


(a sidenote:  it looks like metakit couldn't do a real folder tree,
since there  is no type which says "this is an arbitrary, dynamically
allocated, metakit type".  i'm not sure such flexibility would be a
good thing anyway, though the "store by columns" design would probably
make it fairly easy.  hmm, maybe not such a bad idea, that could
extend metakit from a relational database into the realm of
object-oriented DBs.)

Yes, arbitrary subview structures is up there on my list. I even have a notation for it ("view[*]") and have worked out most details w.r.t. restructuring: restructuring needs to be allowed on views, not just entire storages. Doing so automatically redefines the parent view as having type "*", while restructuring the parent becomes a way to bring heterogeneous subview structures back into homogenous shape.


But there is a way out for simple nesting. A relatively little known feature of MK is the ability to define self-recursive structures:
blah[foo:S,yes[^],bar:S]
The ^ stands for "same structure as parent", so this can handle homogenous trees. It's not necessarily always most convenient thing to do, a row (or tree node in this case) will need a variable amount of nested indexing to reach from the root.


Another efficient way to store a tree in a view, as used in CatFish and in Starkits, is to store either the parent or the child relationships explicitly:
dirs[parent:I,name:S,files[name:S,date:I]]
Where "parent" is a row index into dirs, or -1 for the root. This structure turns out to scale surprisingly well for filesystem-like data collections. In this approach, <parent,name> is unique.


The general guideline is to look for a balance between # of rows in each view/subview and the total number of data columns present in a storage, since columns are the basic objects managed by MK.

-jcw

_____________________________________________
Metakit mailing list  -  [EMAIL PROTECTED]
http://www.equi4.com/mailman/listinfo/metakit

Reply via email to