Thanks for the kind words, everyone. Note that the PR that was merged was the first half, so the mypy-lint job is not yet setup to trigger failures.
Part 2 is up now: https://github.com/apache/beam/pull/10367 My goal is to bundle up changes into smaller PRs from here on out. It might take another 3 PRs to get through the rest. -chad On Wed, Dec 11, 2019 at 2:13 PM Ahmet Altay <al...@google.com> wrote: > Thank you, Chad! This is awesome. > > On Wed, Dec 11, 2019 at 2:05 PM Robert Bradshaw <rober...@google.com> > wrote: > >> +1. Thanks! >> >> On Wed, Dec 11, 2019 at 1:44 PM Pablo Estrada <pabl...@google.com> wrote: >> > >> > Love it. I've merged it, since it was already approved by Robert - and >> yes, we don't want to hit a merge conflict. >> > >> > On Wed, Dec 11, 2019 at 1:36 PM Heejong Lee <heej...@google.com> wrote: >> >> >> >> Wow, this is a big step forward. As a long fan of strongly typed >> functional languages, I'm glad to see this change :) >> >> >> >> On Wed, Dec 11, 2019 at 9:44 AM Chad Dombrova <chad...@gmail.com> >> wrote: >> >>> >> >>> Hi all, >> >>> Robert has diligently reviewed the first batch of changes for this >> PR, and all review notes are addressed and tests are passing: >> https://github.com/apache/beam/pull/9915 >> >>> >> >>> Due to the number of file touched there's a short window of about one >> or two days before a merge conflict arrives on master, and after resolving >> that it usually takes another 1-2 days of pasting "Run Python PreCommit" >> until they pass again, so it would be great to get this merged while the >> window is open! Despite the number of files touched, the changes are >> almost entirely type comments, so the PR is designed to be quite safe. >> >>> >> >>> -chad >> >>> >> >>> >> >>> On Tue, Nov 5, 2019 at 2:50 PM Chad Dombrova <chad...@gmail.com> >> wrote: >> >>>> >> >>>> Glad to hear we have such a forward-thinking community! >> >>>> >> >>>> >> >>>> On Tue, Nov 5, 2019 at 2:43 PM Robert Bradshaw <rober...@google.com> >> wrote: >> >>>>> >> >>>>> Sounds like we have consensus. Let's move forward. I'll follow up >> with >> >>>>> the discussions on the PRs themselves. >> >>>>> >> >>>>> On Wed, Oct 30, 2019 at 2:38 PM Robert Bradshaw < >> rober...@google.com> wrote: >> >>>>> > >> >>>>> > On Wed, Oct 30, 2019 at 1:26 PM Chad Dombrova <chad...@gmail.com> >> wrote: >> >>>>> > > >> >>>>> > >> Do you believe that a future mypy plugin could replace >> pipeline type checks in Beam, or are there limits to what it can do? >> >>>>> > > >> >>>>> > > mypy will get us quite far on its own once we completely >> annotate the beam code. That said, my PR does not include my efforts to >> turn PTransforms into Generics, which will be required to properly analyze >> pipelines, so there's still a lot more work to do. I've experimented with >> a mypy plugin to smooth over some of the rough spots in that workflow and I >> will just say that the mypy API has a very steep learning curve. >> >>>>> > > >> >>>>> > > Another thing to note: mypy is very explicit about function >> annotations. It does not do the "implicit" inference that Beam does, such >> as automatically detecting function return types. I think it should be >> possible to do a lot of that as a mypy plugin, and in fact, since it has >> little to do with Beam it could grow into its own project with outside >> contributors. >> >>>>> > >> >>>>> > Yeah, I don't think, as is, it can replace what we do, but with >> >>>>> > plugins I think it could possibly come closer. Certainly there is >> >>>>> > information that is only available at runtime (e.g. reading from a >> >>>>> > database or avro/parquet file could provide the schema which can >> be >> >>>>> > used for downstream checking) which may limit the ability to do >> >>>>> > everything statically (even Beam Java is moving this direction). >> Mypy >> >>>>> > clearly has an implementation of the "is compatible with" operator >> >>>>> > that I would love to borrow, but unfortunately it's not (easily?) >> >>>>> > exposed. >> >>>>> > >> >>>>> > That being said, we should leverage what we can for pipeline >> >>>>> > authoring, and it'll be a great development too regardless. >> >
