Glad to hear we have such a forward-thinking community!
On Tue, Nov 5, 2019 at 2:43 PM Robert Bradshaw <rober...@google.com> wrote: > Sounds like we have consensus. Let's move forward. I'll follow up with > the discussions on the PRs themselves. > > On Wed, Oct 30, 2019 at 2:38 PM Robert Bradshaw <rober...@google.com> > wrote: > > > > On Wed, Oct 30, 2019 at 1:26 PM Chad Dombrova <chad...@gmail.com> wrote: > > > > > >> Do you believe that a future mypy plugin could replace pipeline type > checks in Beam, or are there limits to what it can do? > > > > > > mypy will get us quite far on its own once we completely annotate the > beam code. That said, my PR does not include my efforts to turn > PTransforms into Generics, which will be required to properly analyze > pipelines, so there's still a lot more work to do. I've experimented with > a mypy plugin to smooth over some of the rough spots in that workflow and I > will just say that the mypy API has a very steep learning curve. > > > > > > Another thing to note: mypy is very explicit about function > annotations. It does not do the "implicit" inference that Beam does, such > as automatically detecting function return types. I think it should be > possible to do a lot of that as a mypy plugin, and in fact, since it has > little to do with Beam it could grow into its own project with outside > contributors. > > > > Yeah, I don't think, as is, it can replace what we do, but with > > plugins I think it could possibly come closer. Certainly there is > > information that is only available at runtime (e.g. reading from a > > database or avro/parquet file could provide the schema which can be > > used for downstream checking) which may limit the ability to do > > everything statically (even Beam Java is moving this direction). Mypy > > clearly has an implementation of the "is compatible with" operator > > that I would love to borrow, but unfortunately it's not (easily?) > > exposed. > > > > That being said, we should leverage what we can for pipeline > > authoring, and it'll be a great development too regardless. >
