I think the spec needs to be further extended. This was just draft 1
with the hope to spark some thought.
I'll continue refining over time.
The thought was to make it a generic "catalogue" which could potentially
drive all type definitions, regardless of serialization mechanism.
--Udo
On 12/22/16 16:32, Darrel Schneider wrote:
You proposal seems to be only handle types for JSON. I do not see that you
support all the pdx field types.
Also you have things like List and "subType" which currently have no
support explicit support in the pdx type system.
So do you intend this proposal to be specific to JSON? If so the gfsh and
apis need to make this clear. If not then your proposal should make sure it
supports all the existing pdx types.
On Thu, Dec 22, 2016 at 4:27 PM, Darrel Schneider <dschnei...@pivotal.io>
wrote:
Something I did not see in your proposal was the rules that would be used
when a JSON document uses "@typeId" to determine if that type is valid for
the current document.
For example I think you want to allow the type to have a field that does
not exist in the document.
I think you also want to say that if the document has a field that does
not exist in the type then an exception is thrown.
You may also have exceptions for when the document's field data can not be
represented in the type's field. For example the type may say the field is
Boolean but the document may have a String whose value is "foobar". Before
the field type was derived from the actual value in the document but now
you can have a mismatch.
On Thu, Dec 22, 2016 at 4:16 PM, Darrel Schneider <dschnei...@pivotal.io>
wrote:
One danger of this solution is users may think they can modify a
previously defined type. Since they specify the type they may think they
can just edit the file and reload the types with modified definitions. In
most cases if data has already been serialized using the old type then
modifying the type will lead to data that can no longer be deserialized.
Are you thinking that these new user defined types would be loaded into
the PDX registry and remembered? If you later tried to reload the same type
and it differs then the reload fails? If so then I think this would keep
users from making illegal changes.
On Thu, Dec 22, 2016 at 4:11 PM, Darrel Schneider <dschnei...@pivotal.io>
wrote:
When generating a pdx type for a JSON document couldn't we sort the
field names from the JSON document so that field order would not generated
different pdx types?
Also when choosing a pdx field type if we always picked a "wider" type
then it would reduce the number of types generated because of different
field types.
On Thu, Dec 22, 2016 at 10:02 AM, Udo Kohlmeyer <ukohlme...@pivotal.io>
wrote:
Hi there Dan,
You are correct, the thought is there to add a flag to the registry to
indicate that a definition is custom and thus should not conflict with the
existing ids. Even if they types were to be stored with the current Pdx
type definitions, upon loading/registration of the custom type definitions,
any conflict will be reported and the custom set will not be registered
until all issues were addressed.
I also had the opinion of the "if they can provide me a typeId, then
surely they can provide me with a fully populated JSON document".
Referencing the example document from the wiki, an user can be created with
just a first and surname. It is not required to provide currentAddress,
previousAddresses, dob,etc... Whilst one could force the client to provide
all fields in the JSON document, it is not always possible nor feasible to
do so. In the POJO world we have a structured data definition and the
generation of a type definition is simple. This done because from a
serialization perspective we always make sure that all fields are
serialized. BUT if we were to change the serialization, i.e not serialize a
field because it is null, the type definition behavior would be exactly the
same as JSON. Only, in this case, because we changed the type definition
for the 'com.demo.User' object (at runtime) the deserialization step for
previous versions would fail.
I believe that if we were to be able to describe WHAT the structure of
a JSON document should be and define the type according to that definition,
we could improve performance (as we don't have to determine type
definitions for every JSON document), be more flexible in consuming JSON
documents that are only partially populated and lastly not potentially
cause a vast amount of JSON-based type definitions to be generated.
In addition to just the JSON benefits, having a formal way of
describing the type definitions will allow us to better maintain the
current registered type definitions. In addition to this, it would allow
customers/clients to create type definitions, by hand, if they were to have
lost their type registry.
As final thought, the addition of the external type registration
process is not meant replace the current behavior. But rather enhance its
capabilities. If no external types will have been defined OR the client
does not provide a '@typeId' tag, the current JSON type definition behavior
will stay the same.
--Udo
On 12/21/16 18:20, Dan Smith wrote:
I'm assuming the type ids here are a different set than the type ids
used
with regular PDX serialization so they won't conflict if the pdx
registry
assigns 1 to some class and a user puts @typeId: 1 in their json?
I'm concerned that this won't really address the type explosion issue.
Users that are able to go to the effort of adding these typeIds to all
of
their json are probably users that can produce consistently formatted
json
in the first place. Users that have inconsistently formatted json are
probably not going to want or be able to add these type ids.
It might be better for us to pursue a way to store arbitrary documents
that
are self describing. Our current approach for json documents is
assuming
that the documents are all consistently formatted. We are infer a
schema
for their documents store the field names in the type registry and the
field values in the serialized data. If we give people the option to
store
and query self describing values, then users with inconsistent json
could
just use that option and pay the extra storage cost.
-Dan
On Tue, Dec 20, 2016 at 4:53 PM, Udo Kohlmeyer <ukohlme...@gmail.com>
wrote:
Hey there,
I've just completed a new proposal on the wiki for a new mechanism
that
could be used to define a type definition for an object.
https://cwiki.apache.org/confluence/display/GEODE/Custom+
External+Type+Definition+Proposal+for+JSON
Primarily the new type definition proposal will hopefully help with
the
"structuring" of JSON document definitions in a manner that will allow
users to submit JSON documents for data types without the need to
provide
every field of the whole domain object type.
Please review and comment as required.
--Udo
