Sorry guys, I was off today. Happy to be part of the party too ;) Regards JB
On 02/04/2018 06:19 PM, Reuven Lax wrote: > Romain, since you're interested maybe the two of us should put together a > proposal for how to set this things (hints, schema) on PCollections? I don't > think it'll be hard - the previous list thread on hints already agreed on a > general approach, and we would just need to flesh it out. > > BTW in the past when I looked, Json schemas seemed to have some odd > limitations > inherited from Javascript (e.g. no distinction between integer and > floating-point types). Is that still true? > > Reuven > > On Sun, Feb 4, 2018 at 9:12 AM, Romain Manni-Bucau <rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com>> wrote: > > > > 2018-02-04 17:53 GMT+01:00 Reuven Lax <re...@google.com > <mailto:re...@google.com>>: > > > > On Sun, Feb 4, 2018 at 8:42 AM, Romain Manni-Bucau > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: > > > 2018-02-04 17:37 GMT+01:00 Reuven Lax <re...@google.com > <mailto:re...@google.com>>: > > I'm not sure where proto comes from here. Proto is one example > of a type that has a schema, but only one example. > > 1. In the initial prototype I want to avoid modifying the > PCollection API. So I think it's best to create a special > SchemaCoder, and pass the schema into this coder. Later we > might > targeted APIs for this instead of going through a coder. > 1.a I don't see what hints have to do with this? > > > Hints are a way to replace the new API and unify the way to pass > metadata in beam instead of adding a new custom way each time. > > > I don't think schema is a hint. But I hear what your saying - hint is > a > type of PCollection metadata as is schema, and we should have a > unified > API for setting such metadata. > > > :), Ismael pointed me out earlier this week that "hint" had an old meaning > in beam. My usage is purely the one done in most EE spec (your "metadata" > in > previous answer). But guess we are aligned on the meaning now, just wanted > to be sure. > > > > > > > > 2. BeamSQL already has a generic record type which fits this > use > case very well (though we might modify it). However as > mentioned > in the doc, the user is never forced to use this generic > record > type. > > > Well yes and not. A type already exists but 1. it is very strictly > limited (flat/columns only which is very few of what big data SQL > can do) and 2. it must be aligned on the converge of generic data > the schema will bring (really read "aligned" as "dropped in favor > of" - deprecated being a smooth way to do it). > > > As I said the existing class needs to be modified and extended, and > not > just for this schema us was. It was meant to represent Calcite SQL > rows, > but doesn't quite even do that yet (Calcite supports nested rows). > However I think it's the right basis to start from. > > > Agree on the state. Current impl issues I hit (additionally to the nested > support which would require by itself a kind of visitor solution) are the > fact to own the schema in the record and handle field by field the > serialization instead of as a whole which is how it would be handled with > a > schema IMHO. > > Concretely what I don't want is to do a PoC which works - they all work > right? and integrate to beam without thinking to a global solution for > this > generic record issue and its schema standardization. This is where > Json(-P) > has a lot of value IMHO but requires a bit more love than just adding > schema > in the model. > > > > > > So long story short the main work of this schema track is not only > on using schema in runners and other ways but also starting to > make > beam consistent with itself which is probably the most important > outcome since it is the user facing side of this work. > > > > On Sun, Feb 4, 2018 at 12:22 AM, Romain Manni-Bucau > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: > > @Reuven: is the proto only about passing schema or also > the > generic type? > > There are 2.5 topics to solve this issue: > > 1. How to pass schema > 1.a. hints? > 2. What is the generic record type associated to a schema > and how to express a schema relatively to it > > I would be happy to help on 1.a and 2 somehow if you need. > > Le 4 févr. 2018 03:30, "Reuven Lax" <re...@google.com > <mailto:re...@google.com>> a écrit : > > One more thing. If anyone here has experience with > various OSS metadata stores (e.g. Kafka Schema > Registry > is one example), would you like to collaborate on > implementation? I want to make sure that source > schemas > can be stored in a variety of OSS metadata stores, and > be easily pulled into a Beam pipeline. > > Reuven > > On Sat, Feb 3, 2018 at 6:28 PM, Reuven Lax > <re...@google.com <mailto:re...@google.com>> wrote: > > Hi all, > > If there are no concerns, I would like to start > working on a prototype. It's just a prototype, so > I > don't think it will have the final API (e.g. for > the > prototype I'm going to avoid change the API of > PCollection, and use a "special" Coder instead). > Also even once we go beyond prototype, it will be > @Experimental for some time, so the API will not > be > fixed in stone. > > Any more comments on this approach before we start > implementing a prototype? > > Reuven > > On Wed, Jan 31, 2018 at 1:12 PM, Romain > Manni-Bucau > <rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com>> wrote: > > If you need help on the json part I'm happy to > help. To give a few hints on what is very > doable: we can add an avro module to johnzon > (asf json{p,b} impl) to back jsonp by avro > (guess it will be one of the first to be > asked) > for instance. > > > Romain Manni-Bucau > @rmannibucau > <https://twitter.com/rmannibucau> | > Blog <https://rmannibucau.metawerx.net/> | > Old > Blog <http://rmannibucau.wordpress.com> | > Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> > > 2018-01-31 22:06 GMT+01:00 Reuven Lax > <re...@google.com <mailto:re...@google.com>>: > > Agree. The initial implementation will be > a > prototype. > > On Wed, Jan 31, 2018 at 12:21 PM, > Jean-Baptiste Onofré <j...@nanthrax.net > <mailto:j...@nanthrax.net>> wrote: > > Hi Reuven, > > Agree to be able to describe the > schema > with different format. The good point > about json schemas is that they are > described by a spec. My point is also > to > avoid the reinvent the wheel. Just an > abstract to be able to use Avro, Json, > Calcite, custom schema descriptors > would > be great. > > Using coder to describe a schema > sounds > like a smart move to implement > quickly. > However, it has to be clear in term of > documentation to avoid "side effect". > I > still think PCollection.setSchema() is > better: it should be metadata (or hint > ;))) on the PCollection. > > Regards > JB > > On 31/01/2018 20:16, Reuven Lax wrote: > > As to the question of how a schema > should be specified, I want to > support several common schema > formats. So if a user has a Json > schema, or an Avro schema, or a > Calcite schema, etc. there should > be > adapters that allow setting a > schema > from any of them. I don't think we > should prefer one over the other. > While Romain is right that many > people know Json, I think far > fewer > people know Json schemas. > > Agree, schemas should not be > enforced (for one thing, that > wouldn't be backwards > compatible!). > I think for the initial prototype > I > will probably use a special coder > to > represent the schema (with > setSchema > an option on the coder), largely > because it doesn't require > modifying > PCollection. However I think > longer > term a schema should be an > optional > piece of metadata on the > PCollection > object. Similar to the previous > discussion about "hints," I think > this can be set on the producing > PTransform, and a SetSchema > PTransform will allow attaching a > schema to any PCollection (i.e. > pc.apply(SetSchema.of(schema))). > This part isn't designed yet, but > I > think schema should be similar to > hints, it's just another piece of > metadata on the PCollection > (though > something interpreted by the > model, > where hints are interpreted by the > runner) > > Reuven > > On Tue, Jan 30, 2018 at 1:37 AM, > Jean-Baptiste Onofré > <j...@nanthrax.net > <mailto:j...@nanthrax.net> > <mailto:j...@nanthrax.net > <mailto:j...@nanthrax.net>>> > wrote: > > Hi, > > I think we should avoid to mix > two things in the discussion (and > so > the document): > > 1. The element of the > collection > and the schema itself are two > different things. > By essence, Beam should not > enforce any schema. That's why I > think > it's a good > idea to set the schema > optionally on the PCollection > (pcollection.setSchema()). > > 2. From point 1 comes two > questions: how do we represent a > schema ? > How can we > leverage the schema to > simplify > the serialization of the element > in the > PCollection and query ? These > two questions are not directly > related. > > 2.1 How do we represent the > schema > Json Schema is a very > interesting idea. It could be an > abstract and > other > providers, like Avro, can be > bind on it. It's part of the json > processing spec > (javax). > > 2.2. How do we leverage the > schema for query and serialization > Also in the spec, json pointer > is interesting for the querying. > Regarding the > serialization, jackson or > other > data binder can be used. > > It's still rough ideas in my > mind, but I like Romain's idea > about > json-p usage. > > Once 2.3.0 release is out, I > will start to update the document > with > those ideas, > and PoC. > > Thanks ! > Regards > JB > > On 01/30/2018 08:42 AM, Romain > Manni-Bucau wrote: > > > > > > Le 30 janv. 2018 01:09, > "Reuven Lax" <re...@google.com > <mailto:re...@google.com> > <mailto:re...@google.com > <mailto:re...@google.com>> > > <mailto:re...@google.com > <mailto:re...@google.com> > <mailto:re...@google.com > <mailto:re...@google.com>>>> a > écrit : > > > > > > > > On Mon, Jan 29, 2018 at > 12:17 PM, Romain Manni-Bucau > <rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com> > <mailto:rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com>> > > > <mailto:rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com>>>> > wrote: > > > > Hi > > > > I have some > questions > on this: how hierarchic schemas > would work? Seems > > it is not really > supported by the ecosystem (out of > custom stuff) :(. > > How would it > integrate smoothly with other > generic record > types - N bridges? > > > > > > Do you mean nested > schemas? What do you mean here? > > > > > > Yes, sorry - wrote the mail > too late ;). Was hierarchic data > and > nested schemas. > > > > > > Concretely I wonder > if using json API couldnt be > beneficial: json-p is a > > nice generic > abstraction with a built in > querying > mecanism (jsonpointer) > > but no actual > serialization (even if json and > binary json > are very > > natural). The big > advantage is to have a well known > ecosystem - who > > doesnt know json > today? - that beam can reuse for > free: > JsonObject > > (guess we dont want > JsonValue abstraction) for the > record > type, > > jsonschema standard > for the schema, jsonpointer for > the > > > delection/projection > etc... It doesnt enforce the > actual > serialization > > (json, smile, avro, > ...) but provide an expressive and > alread known API > > so i see it as a > big > win-win for users (no need to > learn > a new API and > > use N bridges in > all > ways) and beam (impls are here and > API design > > already thought). > > > > > > I assume you're talking > about the API for setting schemas, > not using them. > > Json has many downsides > and I'm not sure it's true that > everyone knows it; > > there are also > competing > schema APIs, such as Avro etc.. > However I think we > > should give Json a fair > evaluation before dismissing it. > > > > > > It is a wider topic than > schema. Actually schema are not > the > first citizen but a > > generic data representation > is. That is where json hits almost > any other API. > > Then, when it comes to > schema, json has a standard for > that > so we > are all good. > > > > Also json has a good > indexing > API compared to alternatives which > are sometimes a > > bit faster - for noop > transforms - but are hardly usable > or make > the code not > > that readable. > > > > Avro is a nice competitor > but > it is compatible - actually avro > is > json driven by > > design - but its API is far > to be that easy due to its schema > enforcement which > > is heavvvyyy and worse is > you > cant work with avro without a > schema. Json would > > allow to reconciliate the > dynamic and static cases since > the job > wouldnt change > > except the setschema. > > > > That is why I think json > is a > good compromise and having a > standard API for it > > allow to fully customize > the > imol as will if needed - even > using > avro or protobuf. > > > > Side note on beam api: i > dont > think it is good to use a main API > for runner > > optimization. It enforces > something to be shared on all > runners > but not widely > > usable. It is also > misleading > for users. Would you set a flink > pipeline option > > with dataflow? My proposal > here is to use hints - properties > - > instead of > > something hardly defined in > the API then standardize it if all > runners support it. > > > > > > > > Wdyt? > > > > Le 29 janv. 2018 > 06:24, "Jean-Baptiste Onofré" > <j...@nanthrax.net > <mailto:j...@nanthrax.net> > <mailto:j...@nanthrax.net > <mailto:j...@nanthrax.net>> > > > <mailto:j...@nanthrax.net > <mailto:j...@nanthrax.net> > <mailto:j...@nanthrax.net > <mailto:j...@nanthrax.net>>>> a > écrit : > > > > > Hi Reuven, > > > > Thanks for the > update ! As I'm working with you > on > this, I fully > > agree and great > > doc gathering > the > ideas. > > > > It's clearly > something we have to add asap in > Beam, > because it would > > allow new > > use cases for > our > users (in a simple way) and open > new areas for the > > runners > > (for instance > dataframe support in the Spark > runner). > > > > By the way, > while > ago, I created BEAM-3437 to track > the PoC/PR > > around this. > > > > Thanks ! > > > > Regards > > JB > > > > On 01/29/2018 > 02:08 AM, Reuven Lax wrote: > > > Previously I > submitted a proposal for adding > schemas as a > > first-class > concept on > > > Beam > PCollections. The proposal > engendered quite a > bit of > > discussion > from the > > > community - > more discussion than I've seen > from > almost any of our > > proposals to > > > date! > > > > > > Based on the > feedback and comments, I reworked > the > proposal > > document quite > a > > > bit. It now > talks more explicitly about the > different between > > dynamic schemas > > > (where the > schema is not fully not know at > graph-creation time), > > and static > > > schemas > (which > are fully know at graph-creation > time). Proposed > > APIs are more > > > fleshed out > now > (again thanks to feedback from > community members), > > and the > > > document > talks > in more detail about evolving > schemas in > > long-running > streaming > > > pipelines. > > > > > > Please take a > look. I think this will be very > valuable to Beam, > > and welcome any > > > feedback. > > > > > > > > > > > https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit# > > <https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit#> > > > <https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit# > > <https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit#>> > > > > <https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit# > > <https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit#> > > <https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit# > > <https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit#>>> > > > > > > Reuven > > > > -- > > Jean-Baptiste > Onofré > > jbono...@apache.org > <mailto:jbono...@apache.org> > <mailto:jbono...@apache.org > <mailto:jbono...@apache.org>> > <mailto:jbono...@apache.org > <mailto:jbono...@apache.org> > <mailto:jbono...@apache.org > <mailto:jbono...@apache.org>>> > > http://blog.nanthrax.net > > Talend - > http://www.talend.com > > > > > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > <mailto:jbono...@apache.org> > <mailto:jbono...@apache.org > <mailto:jbono...@apache.org>> > http://blog.nanthrax.net > Talend - http://www.talend.com > > > > > > > > > > > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
