A little late to the game here but I want to go back to Dan's idea of storing the JSON or other self describing objects as a first class object in Geode. As it stands right now an entry can be a POJO, Java serialized object, or PDX, so why not other types? Seems perfectly reasonable to allow first class storage of JSON (or BSON) without sacrificing any features that we support today with PDX or POJO. In fact one could even implement a version of PdxInstance that wraps a JSON document.
It seems to me that any attempt to add structure to something without structure is going to give us way more heartache than trying to add support for truly unstructured data in Geode. We are part of the way there with PDX since get/set field operations hide any real formal structure. In my last deep dive into PDX over a year ago it seemed very doable to me at the time to add support for something like this. -Jake On Wed, Dec 28, 2016 at 9:52 AM Udo Kohlmeyer <ukohlme...@pivotal.io> wrote: > You are correct here. Ordering the fields would be a simple solution IF > the only problem was that fields were incorrectly ordered. In most cases > not all fields are provided thus causing an explosion of type > definitions that would be generated. > > > On 12/22/16 16:11, Darrel Schneider wrote: > > When generating a pdx type for a JSON document couldn't we sort the field > > names from the JSON document so that field order would not generated > > different pdx types? > > Also when choosing a pdx field type if we always picked a "wider" type > then > > it would reduce the number of types generated because of different field > > types. > > > > > > On Thu, Dec 22, 2016 at 10:02 AM, Udo Kohlmeyer <ukohlme...@pivotal.io> > > wrote: > > > >> Hi there Dan, > >> > >> You are correct, the thought is there to add a flag to the registry to > >> indicate that a definition is custom and thus should not conflict with > the > >> existing ids. Even if they types were to be stored with the current Pdx > >> type definitions, upon loading/registration of the custom type > definitions, > >> any conflict will be reported and the custom set will not be registered > >> until all issues were addressed. > >> > >> I also had the opinion of the "if they can provide me a typeId, then > >> surely they can provide me with a fully populated JSON document". > >> Referencing the example document from the wiki, an user can be created > with > >> just a first and surname. It is not required to provide currentAddress, > >> previousAddresses, dob,etc... Whilst one could force the client to > provide > >> all fields in the JSON document, it is not always possible nor feasible > to > >> do so. In the POJO world we have a structured data definition and the > >> generation of a type definition is simple. This done because from a > >> serialization perspective we always make sure that all fields are > >> serialized. BUT if we were to change the serialization, i.e not > serialize a > >> field because it is null, the type definition behavior would be exactly > the > >> same as JSON. Only, in this case, because we changed the type definition > >> for the 'com.demo.User' object (at runtime) the deserialization step for > >> previous versions would fail. > >> > >> I believe that if we were to be able to describe WHAT the structure of a > >> JSON document should be and define the type according to that > definition, > >> we could improve performance (as we don't have to determine type > >> definitions for every JSON document), be more flexible in consuming JSON > >> documents that are only partially populated and lastly not potentially > >> cause a vast amount of JSON-based type definitions to be generated. > >> > >> In addition to just the JSON benefits, having a formal way of describing > >> the type definitions will allow us to better maintain the current > >> registered type definitions. In addition to this, it would allow > >> customers/clients to create type definitions, by hand, if they were to > have > >> lost their type registry. > >> > >> As final thought, the addition of the external type registration > process > >> is not meant replace the current behavior. But rather enhance its > >> capabilities. If no external types will have been defined OR the client > >> does not provide a '@typeId' tag, the current JSON type definition > behavior > >> will stay the same. > >> > >> --Udo > >> > >> > >> On 12/21/16 18:20, Dan Smith wrote: > >> > >>> I'm assuming the type ids here are a different set than the type ids > used > >>> with regular PDX serialization so they won't conflict if the pdx > registry > >>> assigns 1 to some class and a user puts @typeId: 1 in their json? > >>> > >>> I'm concerned that this won't really address the type explosion issue. > >>> Users that are able to go to the effort of adding these typeIds to all > of > >>> their json are probably users that can produce consistently formatted > json > >>> in the first place. Users that have inconsistently formatted json are > >>> probably not going to want or be able to add these type ids. > >>> > >>> It might be better for us to pursue a way to store arbitrary documents > >>> that > >>> are self describing. Our current approach for json documents is > assuming > >>> that the documents are all consistently formatted. We are infer a > schema > >>> for their documents store the field names in the type registry and the > >>> field values in the serialized data. If we give people the option to > store > >>> and query self describing values, then users with inconsistent json > could > >>> just use that option and pay the extra storage cost. > >>> > >>> -Dan > >>> > >>> On Tue, Dec 20, 2016 at 4:53 PM, Udo Kohlmeyer <ukohlme...@gmail.com> > >>> wrote: > >>> > >>> Hey there, > >>>> I've just completed a new proposal on the wiki for a new mechanism > that > >>>> could be used to define a type definition for an object. > >>>> https://cwiki.apache.org/confluence/display/GEODE/Custom+ > >>>> External+Type+Definition+Proposal+for+JSON > >>>> > >>>> Primarily the new type definition proposal will hopefully help with > the > >>>> "structuring" of JSON document definitions in a manner that will allow > >>>> users to submit JSON documents for data types without the need to > provide > >>>> every field of the whole domain object type. > >>>> > >>>> Please review and comment as required. > >>>> > >>>> --Udo > >>>> > >>>> > >>>> > >
