I think I understand better now. Thank you for the explanation as I am still trying to get an understanding of DTO's and the converter and how they can be used in my application.
David Daniel On Tue, Aug 16, 2016 at 8:29 AM, David Leangen <o...@leangen.net> wrote: > > Hi David D., > > Thanks for your input. > > What you write makes sense, but I don’t think it really applies to what I > am trying to suggest. > > All I am asking for is an introspective view on the DTO data tree > structure, nothing more. Since this is being parsed anyway, this code > provides the perfect opportunity to open up this view of the data. Anything > beyond that would be out of scope, but would be able to interact nicely and > easily with the data tree exposed by this new interface. > > This is in no way a persistence structure, or any kind of database, > especially not relational. Having a parsable, manipulatable view of the > data structure would, however, allow much easier processing of the data to > map to a database, whether it be relational, key/value, graph, etc. > > Aa an example, a few years back, I had to write exactly this type of > functionality in order to do a deep storage of a data structure in Berkeley > DB. The top-level objects “shared” other objects, so it was decided that it > would be wasteful to have millions of duplicate copies of those mid-level > objects. When storing and retrieving data, I had to introspect the data > structure and dejoin the mid-level objects, store them in their own > key/value store, then join them again when the data was retrieved. Document > stores and graph DBs also need instructions as to how to store the data. > Not just RDBs. > > I agree with what you write, that the relationships are too complex to > deduce from the DTO structure. Objects that are embedded in the data do not > define the relationships; there is not enough information for that. That > type of information would be beyond the scope of what I am proposing. > However, the annotations (for instance) that you would use to mark the > relationships would work very well with this data structure. With that > access to the data tree, I could easily kick off a process against the tree > that looks for these annotations so that I can interface with my > persistence layer. > > > That is for persistence. If you want to work with, for instance, Lucene, > for indexing the data, I think that this type of introspective access to > the data tree would also be very useful. > > All I am suggesting is that we provide access to the tree, which would > _enable_ many types of processing against it, not that we actually include > anything new. > > I do understand that it is a step away from a simple “Converter”, but the > parsing is essentially the same. Since the hard work is already being done, > why not take advantage of it here? Even if this tree view ends up being a > completely different service, the same code base could easily serve the two. > > > Cheers, > =David > > > > On Aug 16, 2016, at 8:35 PM, David Daniel <david.daniel.1...@gmail.com> > wrote: > > > > In my opinion with the scenarios you described you are no longer > describing > > the data but a persistence structure. Indexes are about duplicating the > > data so it can be retrieved faster given a storage structure and some > idea > > of that structure is needed to create them. If the persisting side could > > determine the structure solely off the presence of the key then I think > it > > is not bad to have in but it is impossible for a key to cover every area. > > I think I just don't like the idea of tying the structure into the > > definition without value types in java. I am also wary about pulling in > > object relationships because that is getting into type/category theory > > which java does not have a standard way of doing. With relational data > if > > you have a key then you can just have a limited number relationships by > > default like 1 to 1 or 1 to many or many to many but I don't like the > idea > > of having DTO's as relational defined objects while most things are > moving > > away from that. I also think you cannot accurately define most > > relationships in a small structure. How do you define multiple elements > as > > sum types or product types. How do you show one DTO has an inherited > > relationship vs a compositional one. I think there are many functional > > languages that run on the jvm like scala, clojure, kotlin and hazkel that > > do a good job of allowing programmers to create their own relationships > > that can be defined on data and I dont think the spec should be creating > > another. I think having DTO's as they are and then the programmer can > use > > the converter to convert to other objects that have more meaningful > > capabilities is enough for the standard. I think the things discussed > > above should be in a separate library that adds functionality onto DTO > and > > Converter similar to cats in scala. > > > > My 2 cents, > > David Daniel > > > > On Tue, Aug 16, 2016 at 2:00 AM, David Leangen <o...@leangen.net> wrote: > > > >> > >> Hi! > >> > >> The Converter service does a lot of introspection and parsing of the DTO > >> data structure. In many cases, a DTO is a very simple object structure. > >> However, it can also be a very complex structure, too. > >> > >> According to my understanding of the objectives of the Converter, one > >> important goal is to be able to persist data. The idea is that the DTO > >> describes the data, the whole data, and nothing but the data, so help me > >> Uncle. Thus, it is the ideal way to ship the state of the system off to > >> PersistenceLand. > >> > >> I can buy into this vision. > >> > >> If we do buy into this vision, then we may be missing out on a few great > >> opportunities here. When data gets persisted, we often need to > understand > >> the relationships between the embedded objects. Or, we may want to be > able > >> to create an index on the data. These are a few of the reasons why we > would > >> want to have some kind of x-ray vision on the data structure. Since we > >> already go through all the trouble of parsing the data structure in > order > >> to convert it, and since this is ~95% of the work, it would be really > nice > >> to provide access to this information in order to easily link in > services > >> that require this intimate knowledge. Otherwise, all the parsing would > have > >> to be done over and over again for each service. > >> > >> I believe that it would only take a few methods to be able to leverage > all > >> the parsing work done by the Converter. I can think of: > >> > >> DataTree Converter.toTree(DTO dto); // DataTre gives a tree view of the > >> structure > >> Object tree.valueAt(DTO dto, String path); // Dot-separated path value > >> within the tree structure > >> void tree.set(DTO dto, String path, Object value); // Set the value at > >> the given location in the tree structure > >> void process(DTO dto, Consumer<?> function); // Visit each node for > some > >> kind of processing > >> > >> Those are just some examples. Perhaps a new API would be necessary, but > my > >> main point here is that since we are going through all this work of > >> implementing a parser, this is the IDEAL time to create this type of > view > >> on the data. > >> > >> > >> wdyt? > >> > >> I can explain further the idea if you like. For now, I just wanted to > get > >> a quick feedback to see if there is any openness to this kind of thing. > >> > >> > >> =David > >> > >> > >> > >
