Wow, this is a big step forward. As a long fan of strongly typed functional languages, I'm glad to see this change :)
On Wed, Dec 11, 2019 at 9:44 AM Chad Dombrova <chad...@gmail.com> wrote: > Hi all, > Robert has diligently reviewed the first batch of changes for this PR, and > all review notes are addressed and tests are passing: > https://github.com/apache/beam/pull/9915 > > Due to the number of file touched there's a short window of about one or > two days before a merge conflict arrives on master, and after resolving > that it usually takes another 1-2 days of pasting "Run Python PreCommit" > until they pass again, so it would be great to get this merged while the > window is open! Despite the number of files touched, the changes are > almost entirely type comments, so the PR is designed to be quite safe. > > -chad > > > On Tue, Nov 5, 2019 at 2:50 PM Chad Dombrova <chad...@gmail.com> wrote: > >> Glad to hear we have such a forward-thinking community! >> >> >> On Tue, Nov 5, 2019 at 2:43 PM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> Sounds like we have consensus. Let's move forward. I'll follow up with >>> the discussions on the PRs themselves. >>> >>> On Wed, Oct 30, 2019 at 2:38 PM Robert Bradshaw <rober...@google.com> >>> wrote: >>> > >>> > On Wed, Oct 30, 2019 at 1:26 PM Chad Dombrova <chad...@gmail.com> >>> wrote: >>> > > >>> > >> Do you believe that a future mypy plugin could replace pipeline >>> type checks in Beam, or are there limits to what it can do? >>> > > >>> > > mypy will get us quite far on its own once we completely annotate >>> the beam code. That said, my PR does not include my efforts to turn >>> PTransforms into Generics, which will be required to properly analyze >>> pipelines, so there's still a lot more work to do. I've experimented with >>> a mypy plugin to smooth over some of the rough spots in that workflow and I >>> will just say that the mypy API has a very steep learning curve. >>> > > >>> > > Another thing to note: mypy is very explicit about function >>> annotations. It does not do the "implicit" inference that Beam does, such >>> as automatically detecting function return types. I think it should be >>> possible to do a lot of that as a mypy plugin, and in fact, since it has >>> little to do with Beam it could grow into its own project with outside >>> contributors. >>> > >>> > Yeah, I don't think, as is, it can replace what we do, but with >>> > plugins I think it could possibly come closer. Certainly there is >>> > information that is only available at runtime (e.g. reading from a >>> > database or avro/parquet file could provide the schema which can be >>> > used for downstream checking) which may limit the ability to do >>> > everything statically (even Beam Java is moving this direction). Mypy >>> > clearly has an implementation of the "is compatible with" operator >>> > that I would love to borrow, but unfortunately it's not (easily?) >>> > exposed. >>> > >>> > That being said, we should leverage what we can for pipeline >>> > authoring, and it'll be a great development too regardless. >>> >>
