Done Regards JB
On 02/04/2018 09:14 PM, Romain Manni-Bucau wrote: > Works for me. So a jira with target version = 3. > > Can someone with the karma check we have a 3.0.0 in jira system please? > > Le 4 févr. 2018 20:46, "Reuven Lax" <re...@google.com > <mailto:re...@google.com>> > a écrit : > > Seems fine to me. At some point we might want to do an audit of existing > Jira issues, because I suspect there are issues that should be targeted to > 3.0 but are not yet tagged. > > On Sun, Feb 4, 2018 at 11:41 AM, Jean-Baptiste Onofré <j...@nanthrax.net > <mailto:j...@nanthrax.net>> wrote: > > I would prefer to use Jira, with "wish"/"ideas", and adding Beam 3.0.0 > version. > > WDYT ? > > Regards > JB > > On 02/04/2018 07:55 PM, Reuven Lax wrote: > > Do we have a good place to track the items for Beam 3.0, or is Jira > the best > > place? Romain has a good point - if this gets forgotten when we do > Beam 3.0, > > then we're stuck waiting around till Beam 4.0. > > > > Reuven > > > > On Sun, Feb 4, 2018 at 9:27 AM, Jean-Baptiste Onofré > <j...@nanthrax.net <mailto:j...@nanthrax.net> > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote: > > > > That's a good point. In the roadmap for Beam 3, I think it makes > sense to add a > > point about this. > > > > Regards > > JB > > > > On 02/04/2018 06:18 PM, Eugene Kirpichov wrote: > > > I think doing a change that would break pipeline update for > every single user of > > > Flink and Dataflow needs to be postponed until a next major > version. Pipeline > > > update is a very frequently used feature, especially by the > largest users. We've > > > had those users get significantly upset even when we > accidentally broke update > > > compatibility for some special cases of individual transforms; > breaking it > > > intentionally and project-wide is too extreme to be justified > by > the benefits of > > > the current change. > > > > > > That said, I think concerns about coder APIs are reasonable, > and > it is > > > unfortunate that we effectively can't make changes to them > right > now. It would > > > be great if in the next major version we were better prepared > for evolution of > > > coders, e.g. by having coders support a version marker or > something like that, > > > with an API for detecting the version of data on wire and > reading or writing > > > data of an old version. Such a change (introducing versioning) > would also, of > > > course, be incompatible and would need to be postponed until a > major version - > > > but, at least, subsequent changes wouldn't. > > > > > > ...And as I was typing this email, seems that this is what the > thread already > > > came to! > > > > > > On Sun, Feb 4, 2018 at 9:16 AM Romain Manni-Bucau > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>> wrote: > > > > > > I like this idea of migration support at coder level. It > would require to > > > add a metadata in all outputs which would represent the > version then coders > > > can handle the logic properly depending the version - we > can assume a coder > > > dev upgrade the version when he breaks the representation > I hope ;). > > > With this: no runner impact at all :). > > > > > > > > > Romain Manni-Bucau > > > @rmannibucau <https://twitter.com/rmannibucau > <https://twitter.com/rmannibucau> > > <https://twitter.com/rmannibucau > <https://twitter.com/rmannibucau>>> | Blog > > > <https://rmannibucau.metawerx.net/ > <https://rmannibucau.metawerx.net/> > > <https://rmannibucau.metawerx.net/ > <https://rmannibucau.metawerx.net/>>> | Old Blog > > > <http://rmannibucau.wordpress.com > <http://rmannibucau.wordpress.com> <http://rmannibucau.wordpress.com > <http://rmannibucau.wordpress.com>>> > > | Github > > > <https://github.com/rmannibucau > <https://github.com/rmannibucau> <https://github.com/rmannibucau > <https://github.com/rmannibucau>>> | > > LinkedIn > > > <https://www.linkedin.com/in/rmannibucau > <https://www.linkedin.com/in/rmannibucau> > > <https://www.linkedin.com/in/rmannibucau > <https://www.linkedin.com/in/rmannibucau>>> | Book > > > > > > > <https://www.packtpub.com/application-development/java-ee-8-high-performance > > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > <https://www.packtpub.com/application-development/java-ee-8-high-performance > > <https://www.packtpub.com/application-development/java-ee-8-high-performance>>> > > > > > > 2018-02-04 18:09 GMT+01:00 Reuven Lax <re...@google.com > <mailto:re...@google.com> <mailto:re...@google.com > <mailto:re...@google.com>> > > > <mailto:re...@google.com <mailto:re...@google.com> > <mailto:re...@google.com <mailto:re...@google.com>>>>: > > > > > > It would already break quite a number of users at > this point. > > > > > > I think what we should be doing is moving forward on > the snapshot/update > > > proposal. That proposal actually provides a way > forward when coders > > > change (it proposes a way to map an old snapshot to > one using the new > > > coder, so changes to coders in the future will be > much easier to make. > > > However much of the implementation for this will > likely be at the runner > > > level, not the SDK level. > > > > > > Reuven > > > > > > On Sun, Feb 4, 2018 at 9:04 AM, Romain Manni-Bucau > > > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>> wrote: > > > > > > I fully understand that, and this is one of the > reason managing to > > > solve these issues is very important and ASAP. My > conclusion is that > > > we must break it now to avoid to do it later when > usage will be way > > > more developped - I would be very happy to be > wrong on that point - > > > so I started this PR and this thread. We can > postpone it but it > > > would break later so for probably more users. > > > > > > > > > Romain Manni-Bucau > > > @rmannibucau <https://twitter.com/rmannibucau > <https://twitter.com/rmannibucau> > > <https://twitter.com/rmannibucau > <https://twitter.com/rmannibucau>>> | Blog > > > <https://rmannibucau.metawerx.net/ > <https://rmannibucau.metawerx.net/> > > <https://rmannibucau.metawerx.net/ > <https://rmannibucau.metawerx.net/>>> | Old Blog > > > <http://rmannibucau.wordpress.com > <http://rmannibucau.wordpress.com> > > <http://rmannibucau.wordpress.com > <http://rmannibucau.wordpress.com>>> > | Github > > > <https://github.com/rmannibucau > <https://github.com/rmannibucau> > > <https://github.com/rmannibucau > <https://github.com/rmannibucau>>> | LinkedIn > > > <https://www.linkedin.com/in/rmannibucau > <https://www.linkedin.com/in/rmannibucau> > > <https://www.linkedin.com/in/rmannibucau > <https://www.linkedin.com/in/rmannibucau>>> | Book > > > > > > > <https://www.packtpub.com/application-development/java-ee-8-high-performance > > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > <https://www.packtpub.com/application-development/java-ee-8-high-performance > > <https://www.packtpub.com/application-development/java-ee-8-high-performance>>> > > > > > > 2018-02-04 17:49 GMT+01:00 Reuven Lax > <re...@google.com <mailto:re...@google.com> <mailto:re...@google.com > <mailto:re...@google.com>> > > > <mailto:re...@google.com <mailto:re...@google.com> > <mailto:re...@google.com <mailto:re...@google.com>>>>: > > > > > > Unfortunately several runners (at least Flink > and Dataflow) > > > support in-place update of streaming > pipelines as a key feature, > > > and changing coder format breaks this. This > is a very important > > > feature of both runners, and we should > endeavor not to break them. > > > > > > In-place snapshot and update is also a > top-level Beam proposal > > > that was received positively, though neither > of those runners > > > yet implement the proposed interface. > > > > > > Reuven > > > > > > On Sun, Feb 4, 2018 at 8:44 AM, Romain > Manni-Bucau > > > <rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com> > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>> wrote: > > > > > > Sadly yes, and why the PR is actually > WIP. As mentionned it > > > modifies it and requires some updates in > other languages and > > > the standard_coders.yml file (I didn't > find how this file > > > was generated). > > > Since coders must be about volatile data > I don't think it is > > > a big deal to change it though. > > > > > > > > > Romain Manni-Bucau > > > @rmannibucau > <https://twitter.com/rmannibucau <https://twitter.com/rmannibucau> > > <https://twitter.com/rmannibucau > <https://twitter.com/rmannibucau>>> | Blog > > > <https://rmannibucau.metawerx.net/ > <https://rmannibucau.metawerx.net/> > > <https://rmannibucau.metawerx.net/ > <https://rmannibucau.metawerx.net/>>> | Old Blog > > > <http://rmannibucau.wordpress.com > <http://rmannibucau.wordpress.com> > > <http://rmannibucau.wordpress.com > <http://rmannibucau.wordpress.com>>> > | Github > > > <https://github.com/rmannibucau > <https://github.com/rmannibucau> > > <https://github.com/rmannibucau > <https://github.com/rmannibucau>>> | LinkedIn > > > <https://www.linkedin.com/in/rmannibucau > <https://www.linkedin.com/in/rmannibucau> > > <https://www.linkedin.com/in/rmannibucau > <https://www.linkedin.com/in/rmannibucau>>> | Book > > > > > > > <https://www.packtpub.com/application-development/java-ee-8-high-performance > > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > <https://www.packtpub.com/application-development/java-ee-8-high-performance > > <https://www.packtpub.com/application-development/java-ee-8-high-performance>>> > > > > > > 2018-02-04 17:34 GMT+01:00 Reuven Lax > <re...@google.com <mailto:re...@google.com> <mailto:re...@google.com > <mailto:re...@google.com>> > > > <mailto:re...@google.com > <mailto:re...@google.com> <mailto:re...@google.com > <mailto:re...@google.com>>>>: > > > > > > One question - does this change the > actual byte encoding > > > of elements? We've tried hard not to > do that so far for > > > reasons of compatibility. > > > > > > Reuven > > > > > > On Sun, Feb 4, 2018 at 6:44 AM, > Romain Manni-Bucau > > > <rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>> > > > wrote: > > > > > > Hi guys, > > > > > > I submitted a PR on coders to > enhance 1. the user > > > experience 2. the determinism and > handling of > > coders. > > > > > > 1. the user experience is linked > to > what i > > sent some > > > days ago: close handling of the > streams from a > > coder > > > code. Long story short I add a > SkipCloseCoder > > which > > > can decorate a coder and just > wraps > the stream > > > (input or output) in flavors > skipping close() > > calls. > > > This avoids to do it by default > (which had my > > > preference if you read the related > thread but not > > > the one of everybody) but also > makes > the usage > > of a > > > coder with this issue easy since > the > of() of the > > > coder just wraps itself in this > delagating coder. > > > > > > 2. this one is more nasty and > mainly > concerns > > > IterableLikeCoders. These ones use > this kind of > > > algorithm (keep in mind they work > on > a list): > > > > > > writeSize() > > > for all element e { > > > elementCoder.write(e) > > > } > > > writeMagicNumber() // this one > depends the size > > > > > > The decoding is symmetric so I > bypass it here. > > > > > > Indeed all these writes (reads) > are > done on > > the same > > > stream. Therefore it assumes you > read as much > > bytes > > > than you write...which is a huge > assumption for a > > > coder which should by contract > assume it can read > > > the stream...as a stream (until > -1). > > > > > > The idea of the fix is to change > this encoding to > > > this kind of algorithm: > > > > > > writeSize() > > > for all element e { > > > writeElementByteCount(e) > > > elementCoder.write(e) > > > } > > > writeMagicNumber() // still > optionally > > > > > > This way on the decode size you > can > wrap the > > stream > > > by element to enforce the > limitation > of the > > byte count. > > > > > > Side note: this indeed enforce a > limitation due to > > > java byte limitation but if you > check coder > > code it > > > is already here at the higher > level > so it is not a > > > big deal for now. > > > > > > In terms of implementation it > uses a > > > LengthAwareCoder which delegates > to > another coder > > > the encoding and just adds the > byte > count > > before the > > > actual serialization. Not perfect > but should > > be more > > > than enough in terms of support > and > perf for > > beam if > > > you think real pipelines (we try > to > avoid > > > serializations or it is done on > some > well known > > > points where this algo should be > > enough...worse case > > > it is not a huge overhead, mainly > just some memory > > > overhead). > > > > > > > > > The PR is available > > > > at https://github.com/apache/beam/pull/4594 > <https://github.com/apache/beam/pull/4594> > > <https://github.com/apache/beam/pull/4594 > <https://github.com/apache/beam/pull/4594>>. If you > > > check you will see I put it "WIP". > The main reason > > > is that it changes the encoding > format for > > > containers (lists, iterable, ...) > and therefore > > > breaks python/go/... tests and the > > > standard_coders.yml definition. > Some > help on that > > > would be very welcomed. > > > > > > Technical side note if you > > > wonder: UnownedInputStream doesn't > even allow to > > > mark the stream so there is no > real > fast way > > to read > > > the stream as fast as possible > with > standard > > > buffering strategies and to > support > this automatic > > > IterableCoder wrapping which is > implicit. In other > > > words, if beam wants to support > any > coder, > > including > > > the ones not requiring to write > the > size of the > > > output - most of the codecs - then > we need to > > change > > > the way it works to something like > that which does > > > it for the user which doesn't know > its coder got > > > wrapped. > > > > > > Hope it makes sense, if not, don't > hesitate to ask > > > questions. > > > > > > Happy end of week-end. > > > > > > Romain Manni-Bucau > > > @rmannibucau > <https://twitter.com/rmannibucau <https://twitter.com/rmannibucau> > > <https://twitter.com/rmannibucau > <https://twitter.com/rmannibucau>>> | > > > Blog > <https://rmannibucau.metawerx.net/ <https://rmannibucau.metawerx.net/> > > <https://rmannibucau.metawerx.net/ > <https://rmannibucau.metawerx.net/>>> | Old Blog > > > <http://rmannibucau.wordpress.com > <http://rmannibucau.wordpress.com> > > <http://rmannibucau.wordpress.com > <http://rmannibucau.wordpress.com>>> > | Github > > > <https://github.com/rmannibucau > <https://github.com/rmannibucau> > > <https://github.com/rmannibucau > <https://github.com/rmannibucau>>> | LinkedIn > > > > <https://www.linkedin.com/in/rmannibucau > <https://www.linkedin.com/in/rmannibucau> > > <https://www.linkedin.com/in/rmannibucau > <https://www.linkedin.com/in/rmannibucau>>> | Book > > > > > > > <https://www.packtpub.com/application-development/java-ee-8-high-performance > > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > <https://www.packtpub.com/application-development/java-ee-8-high-performance > > <https://www.packtpub.com/application-development/java-ee-8-high-performance>>> > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Jean-Baptiste Onofré > > jbono...@apache.org <mailto:jbono...@apache.org> > <mailto:jbono...@apache.org <mailto:jbono...@apache.org>> > > http://blog.nanthrax.net > > Talend - http://www.talend.com > > > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org <mailto:jbono...@apache.org> > http://blog.nanthrax.net > Talend - http://www.talend.com > > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
