Thank you, Chad! This is awesome. On Wed, Dec 11, 2019 at 2:05 PM Robert Bradshaw <rober...@google.com> wrote:
> +1. Thanks! > > On Wed, Dec 11, 2019 at 1:44 PM Pablo Estrada <pabl...@google.com> wrote: > > > > Love it. I've merged it, since it was already approved by Robert - and > yes, we don't want to hit a merge conflict. > > > > On Wed, Dec 11, 2019 at 1:36 PM Heejong Lee <heej...@google.com> wrote: > >> > >> Wow, this is a big step forward. As a long fan of strongly typed > functional languages, I'm glad to see this change :) > >> > >> On Wed, Dec 11, 2019 at 9:44 AM Chad Dombrova <chad...@gmail.com> > wrote: > >>> > >>> Hi all, > >>> Robert has diligently reviewed the first batch of changes for this PR, > and all review notes are addressed and tests are passing: > https://github.com/apache/beam/pull/9915 > >>> > >>> Due to the number of file touched there's a short window of about one > or two days before a merge conflict arrives on master, and after resolving > that it usually takes another 1-2 days of pasting "Run Python PreCommit" > until they pass again, so it would be great to get this merged while the > window is open! Despite the number of files touched, the changes are > almost entirely type comments, so the PR is designed to be quite safe. > >>> > >>> -chad > >>> > >>> > >>> On Tue, Nov 5, 2019 at 2:50 PM Chad Dombrova <chad...@gmail.com> > wrote: > >>>> > >>>> Glad to hear we have such a forward-thinking community! > >>>> > >>>> > >>>> On Tue, Nov 5, 2019 at 2:43 PM Robert Bradshaw <rober...@google.com> > wrote: > >>>>> > >>>>> Sounds like we have consensus. Let's move forward. I'll follow up > with > >>>>> the discussions on the PRs themselves. > >>>>> > >>>>> On Wed, Oct 30, 2019 at 2:38 PM Robert Bradshaw <rober...@google.com> > wrote: > >>>>> > > >>>>> > On Wed, Oct 30, 2019 at 1:26 PM Chad Dombrova <chad...@gmail.com> > wrote: > >>>>> > > > >>>>> > >> Do you believe that a future mypy plugin could replace pipeline > type checks in Beam, or are there limits to what it can do? > >>>>> > > > >>>>> > > mypy will get us quite far on its own once we completely > annotate the beam code. That said, my PR does not include my efforts to > turn PTransforms into Generics, which will be required to properly analyze > pipelines, so there's still a lot more work to do. I've experimented with > a mypy plugin to smooth over some of the rough spots in that workflow and I > will just say that the mypy API has a very steep learning curve. > >>>>> > > > >>>>> > > Another thing to note: mypy is very explicit about function > annotations. It does not do the "implicit" inference that Beam does, such > as automatically detecting function return types. I think it should be > possible to do a lot of that as a mypy plugin, and in fact, since it has > little to do with Beam it could grow into its own project with outside > contributors. > >>>>> > > >>>>> > Yeah, I don't think, as is, it can replace what we do, but with > >>>>> > plugins I think it could possibly come closer. Certainly there is > >>>>> > information that is only available at runtime (e.g. reading from a > >>>>> > database or avro/parquet file could provide the schema which can be > >>>>> > used for downstream checking) which may limit the ability to do > >>>>> > everything statically (even Beam Java is moving this direction). > Mypy > >>>>> > clearly has an implementation of the "is compatible with" operator > >>>>> > that I would love to borrow, but unfortunately it's not (easily?) > >>>>> > exposed. > >>>>> > > >>>>> > That being said, we should leverage what we can for pipeline > >>>>> > authoring, and it'll be a great development too regardless. >
