I think we started out as a bunch of relation-heads, modeling things
after the system catalogs found in our favorite RDBMSs. Their record
types are open AFAIK - but I think Steven was saying that it feels like
this is a use case whose field(s) should be predeclared (and using the
term "closed field" to mean predeclared field). I agree. I think to
the extent that we can anticipate the need for a given bit of metadata,
it's useful to have it be there in the known schema; but, when we move
into the era when we cannot make changes to the known schema part of the
catalogs, because we have dependent customers who'd be hurt by that, we
will start adding more fields that are not predeclared but only
documented. The schema for the metadata is really there more as
documentation than as being "truly necessary"...
When we first started out we were just not thinking in an "open-minded"
way on the catalog side of things..... :-)
Cheers,
Mike
On 12/14/15 11:53 AM, Till Westmann wrote:
Let me prefix this with that statement that I haven’t looked into our
metadata storage implementation.
But looking at the relatively frequent changes to the metadata format
it seems that we should either
a) make the types for metadata records open or
b) be very careful in designing the next (and future) revision(s) of it.
I don’t know why the metadata storage was done the way it is now and
what the considerations were when we did that.
Does anybody remember those (or is there even a document that contains
the design and rationale)?
Thanks,
Till
On 14 Dec 2015, at 11:35, Steven Jacobs wrote:
It could easily be done as reverse-compatible, but my thinking was that
this is the "wrong" choice. I can easily make the datatype dataverse an
open field for the metadata. The question is, why do we have open vs
closed
fields for metadata at all? If it is okay for them to be open, should we
get rid of the schema entirely? if it's not okay, then shouldn't this
field
be closed? From a design standpoint it seems that if reverse
compatibility
were not an issue than the field should be closed.
Steven
On Mon, Dec 14, 2015 at 11:30 AM, Ildar Absalyamov <
[email protected]> wrote:
If fix for 2) will break the backwards-compatibility 1) might do
that as
well and be submitted together.
Now 2) was a long overdue problem, I don’t think there is any reason
even
to try make changes backwards-compatible, because it was broken in the
first place.
On Dec 14, 2015, at 11:16, Steven Jacobs <[email protected]> wrote:
It's a new attribute, but it's a closed field, which means it isn't
backwards compatible.
Steven
On Mon, Dec 14, 2015 at 11:13 AM, Ian Maxon <[email protected]> wrote:
For 1), I guess the question is whether it would be a backwards
compatible change. Since it's just a new attribute (right?...),
and it
is also sort of a new feature rather than a fix for something that
was
critically broken, I would tend toward putting it on master. If it's
not backwards compatible though maybe it needs more careful
consideration.
-Ian
On Mon, Dec 14, 2015 at 10:12 AM, Steven Jacobs <[email protected]>
wrote:
Hi all,
I'm implementing a change so that datasets can use datatypes from
alternate
data verses (previously the type and set had to be from the same
dataverse). Unfortunately this means another change for Dataset
Metadata
(which will now store the dataverse for its type).
As such, I had a couple of questions:
1) Should this change be thrown into the release branch, as it is
another
Metadata change?
2) In implementing this change, I've been looking at the Metadata
secondary
indexes. I had a discussion with Ildar, and it seems the thread on
Metadata
secondary indexes being "hacked" has been lost. Is this also
something
that
should get into the release? Is there anyone currently looking at
it?
Steven
Best regards,
Ildar
