Great -- congratulations to everyone who has contributed to the Python SDK!
On Mon, Jan 30, 2017 at 11:10 PM, Ahmet Altay <al...@google.com.invalid> wrote: > Hi all, > > This merge is completed. Python SDK is now officially part of the master > branch! Thank you all for the support. Please open an issue, if you notice > a reference to the now obsolete python-sdk branch in the documentation. > > There will not be any more merges to the python-sdk branch. Going forward > please use the master branch for Python SDK development. There are a few > existing open PRs to the python-sdk [1]. If you are the author of one of > those PRs, please rebase them on top of master. > > Thank you, > Ahmet > > [1] https://github.com/pulls?utf8=✓&q=is%3Aopen+is%3Apr+base% > <https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+base%25> > 3Apython-sdk+repo%3Aapache%2Fbeam+ > <https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is% > 3Apr+base%3Apython-sdk+repo%3Aapache%2Fbeam+> > > On Fri, Jan 20, 2017 at 10:06 AM, Kenneth Knowles <k...@google.com.invalid> > wrote: > > > To clarify the implied criteria of that last exchange, it is "An SDK > should > > have at least one runner that can execute the complete model (may be a > > direct runner)" > > > > I want to highlight this, because whether an _SDK_ supports unbounded > data > > is not particularly well-defined, and will evolve: > > > > - With the Runner API, an SDK will need to support building a graph with > > unbounded constructs, as today with probably minimal changes. > > > > - With the Fn API, if any part of the Fn API is specific to unbounded > > data, the SDK will need to implement it. I think right now there is no > such > > thing, and we don't want such a thing, so SDKs implementing the Fn API > > automatically support unbounded data. > > > > - There will also likely be an SDK-specific shim just as there is today, > > to leverage idiomatic deserialized representations. The richness of this > > shim will decrease so that it will need to "support" unbounded data but > > that will be a ~one liner. > > > > Getting the Python SDK on master will accelerate our progress towards the > > Fn API - partly technical, partly community - which is the best path > > towards support for unbounded data across multiple runners. I think the > > criteria are written with the completed portability framework in mind. So > > this exchange makes me actually more convinced we should merge python-sdk > > to master. > > > > On Fri, Jan 20, 2017 at 9:53 AM, Robert Bradshaw < > > rober...@google.com.invalid> wrote: > > > > > On Thu, Jan 19, 2017 at 11:56 PM, Dan Halperin > > > <dhalp...@google.com.invalid> wrote: > > > > I do not think that Python SDK yet meets the bar [1] for implementing > > the > > > > Beam model -- supporting Unbounded data is very important. That said, > > > given > > > > the committed and sustained set of contributors, it generally makes > > sense > > > > to me to make an exception in anticipation of these features being > > > fleshed > > > > out soon; including potentially new users/contributors that would > > arrive > > > > once in master. > > > > > > > > [1] https://lists.apache.org/thread.html/CAAzyFAxcmexUQnbF=Y > > > > k0plmm3f5e5bqwjz4+c5doruclnxo...@mail.gmail.com > > > > > > That is a valid point. The Python SDK supports all the unbounded parts > > > of the model except for unbounded sources, which was deferred while > > > seeing how https://s.apache.org/splittable-do-fn played out. I've been > > > working with the team and merging/reviewing most of their code, and > > > have full confidence this will be coming (and on that note can vouch > > > for a healthy community and support which are much harder to add > > > later). > > > > > > In short, I think it has the required maturity, and I'm in favor of > > > merging soonish. > > > > > > > On Wed, Jan 18, 2017 at 12:24 AM, Ahmet Altay > <al...@google.com.invalid > > > > > > > wrote: > > > > > > > >> Thank you all for the comments so far. I would follow the process as > > > >> suggested by Davor and others in this thread. > > > >> > > > >> Ahmet > > > >> > > > >> On Tue, Jan 17, 2017 at 11:47 PM, Sergio Fernández < > wik...@apache.org > > > > > > >> wrote: > > > >> > > > >> > Hi > > > >> > > > > >> > On Tue, Jan 17, 2017 at 5:22 PM, Ahmet Altay > > <al...@google.com.invalid > > > > > > > >> > wrote: > > > >> > > > > > >> > > tl;dr: I would like to start a discussion about merging > python-sdk > > > >> branch > > > >> > > to master branch. Python SDK is mature enough and merging it to > > > master > > > >> > will > > > >> > > accelerate its development and adoption. > > > >> > > > > > >> > > > > >> > Good point, Ahmet! > > > >> > > > > >> > I've following closed the development since it was imported in > June. > > > For > > > >> > the prototypes I've implemented so far it works quite well; I > guess > > > we'd > > > >> > just need to focus the next months in bringing more runners > support. > > > >> > > > > >> > With a great effort from a lot of contributors(*), Python SDK [1] > is > > > now > > > >> a > > > >> > > mostly complete, tested, performant Python implementation of the > > > Beam > > > >> > > model. Since June, when we first started with Python SDK in > Apache > > > Beam > > > >> > we > > > >> > > have been continuously improving it. > > > >> > > > > > >> > > > > >> > I wouldn't merge during the preparation of 0.5.0 release, but > after > > > that > > > >> > could be a good time to merge back into master. > > > >> > > > > >> > > > > >> > ** Python SDK currently supports: > > > >> > > > > > >> > > * Model: All main concepts are present (ParDo, GroupByKey, > > Windowing > > > >> > etc.). > > > >> > > * IO: There are extensible APIs for writing new bounded sources > > and > > > >> > sinks. > > > >> > > Implementations are provided for Text, Avro, BigQuery, and > > > Datastore. > > > >> > > * Runners: Python SDK has an extensible base runner module that > > > allows > > > >> > > building specific runners on top of it. The SDK comes with two > > > pipeline > > > >> > > runners: DirectRunner and DataflowRunner; and it is possible to > > add > > > >> more. > > > >> > > The existing runners are currently limited to bounded execution > > and > > > >> > > otherwise equivalent to their Java SDK counterparts in > > > functionality. > > > >> > > > > > >> > > > > >> > What would the effort of porting, and maintaining, parallel > versions > > > of > > > >> the > > > >> > Java runners? I guess I'd need to dig deeper in the model, but > this > > > may > > > >> > represent a major effort for the project, right? > > > >> > > > > >> > > > >> It is somewhat higher for DirectRunner because DirectRunner also > > > implements > > > >> the code for execution. It is not that high for DataflowRunner > because > > > the > > > >> base runner module has a lot of helpers with the right hooks for > > > >> implementing a generic runner. I would _expect_ the experience in > > > general > > > >> would be similar to the latter. > > > >> > > > >> > > > >> > > > > >> > > > > >> > > > > >> > > * Testing: Python SDK implements ValidatesRunner test framework > > for > > > >> > > implementing integration test for current and future runners. > > There > > > is > > > >> > unit > > > >> > > test coverage for all modules, and a number of integrations test > > for > > > >> > > validating existing runners. > > > >> > > * Documentation and examples: Documentation work has started on > > > Python > > > >> > SDK. > > > >> > > Beam Programming Guide page has been updated to include Python > > [2]. > > > The > > > >> > > code comes with many ready to use examples and we are in a good > > > place > > > >> to > > > >> > > start documenting those on the website. > > > >> > > > > > >> > > ** We are not done yet, next on the roadmap we have: > > > >> > > > > > >> > > * Streaming: Both of the existing runners lack support for > > streaming > > > >> > > execution, and currently there is work going on for adding > > streaming > > > >> > > support to DirectRunner [3]. > > > >> > > * Documentation: Filling the rest of the Beam documentations > with > > > >> Python > > > >> > > SDK specific information and examples. > > > >> > > * SDK consistency: Making Python SDK consistent with the Java > SDK. > > > We > > > >> > have > > > >> > > come a long way on this and have only a few items left [4]. > > > >> > > * Beamifying: We have been working on removing Dataflow-specific > > > >> > references > > > >> > > both from the documentation and from the code. There is some > work > > > left, > > > >> > and > > > >> > > we are currently working on those as well [5]. > > > >> > > > > > >> > > ** Steps and implications of merging to master: > > > >> > > > > > >> > > * Master branch is merged to python-sdk branch at regular > > intervals > > > and > > > >> > the > > > >> > > last merge was on 12/22. All the past merges were uneventful > > because > > > >> > there > > > >> > > is a minimal overlap in modified files between branches. > > Integrating > > > >> > > python-sdk to master will similarly touch a small number of > > existing > > > >> > files. > > > >> > > > > > >> > > * Python SDK is using the same tools for building and testing. > It > > is > > > >> > > already integrated with Maven, Jenkins and Travis. Specifically > > the > > > >> > impact > > > >> > > to the testing infrastructure would be: > > > >> > > - There will be two additional test configurations in Travis. > > Since > > > >> > Travis > > > >> > > runs all configurations in parallel there should not be a > > noticeable > > > >> > change > > > >> > > in the Travis run time. > > > >> > > - Jenkins pre-commit test will start running the Python SDK > tests. > > > It > > > >> > will > > > >> > > add an additional 5 minutes to the completion time of pre-commit > > > test. > > > >> > > Historically Python SDK tests were not flaky and did not cause > any > > > >> random > > > >> > > failures. > > > >> > > - Jenkins Python post-commit test is already separated from the > > > other > > > >> > > post-commit tests and will continue to exist. It would not > change > > > the > > > >> > > testing time for any other test. > > > >> > > > > > >> > > * The release process needs to be updated to accommodate > releasing > > > >> Python > > > >> > > artifacts. Python SDK would fit in the existing release schedule > > and > > > >> > could > > > >> > > be released along with the Java SDK. The additional steps would > > > >> include: > > > >> > > - Generating Python artifacts. This could be done with a single > > > command > > > >> > > using Maven today. > > > >> > > - Publishing the artifacts to a central repository such as PyPI. > > > >> > > > > > >> > > > > >> > I'm more than happy to help on this. We left on purpose some > things > > > open > > > >> > when we added Maven support to the Python build. > > > >> > > > > >> > > > >> That would be awesome. We can coordinate on that post-merge. > > > >> > > > >> > > > >> > > > > >> > > > > >> > > > > >> > > - Updating the release guide to reflect the changes above. > > > >> > > > > > >> > > * Users: There are existing users using the Python SDK. To give > a > > > rough > > > >> > > estimate, a distribution of the Beam Python SDK had a total of > 23K > > > >> > > downloads in the past 6 months [6]. Some of those users are > > already > > > >> > engaged > > > >> > > with the community (e.g. [7]). There might be an increased > amount > > > >> > > engagement from the rest of them after the merge. > > > >> > > > > > >> > > > > >> > Python 3 support is something we definitively need to look ahead. > > I'd > > > try > > > >> > to make the codebase compatible with both 2.7.x and 3.6.x, rather > > than > > > >> > using other solutions like 2to3. > > > >> > > > > >> > > > >> I agree with you. I think it makes more sense to make codebase > > > compatible > > > >> with both. As you mentioned Python 3 support is not a short-term > goal > > in > > > >> the roadmap, and we can discuss it more as we approach that. > > > >> > > > >> > > > >> > > > > >> > > > > >> > Looking forward to hearing your thoughts and comments on > > “graduating” > > > >> > > python-sdk to the master. > > > >> > > > > > >> > > Thank you, > > > >> > > Ahmet > > > >> > > > > > >> > > (*) Python SDK branch currently has a diverse group of > > contributors. > > > >> > > Regular contributors include Charles Chen, Chamikara Jayalath, > > María > > > >> > García > > > >> > > Herrero, Mark Liu, Pablo Estrada, Robert Bradshaw (Apache Beam > > PMC), > > > >> > > Sourabh Bajaj, and Vikas Kedigehalli. We have also had > > contributions > > > >> from > > > >> > > Abdullah Bashir, Marco Buccini, Sergio Fernández, Seunghyun Lee, > > and > > > >> > > Younghee Kwon. > > > >> > > > > > >> > > [1] https://github.com/apache/beam/tree/python-sdk/sdks/python > > > >> > > [2] https://beam.apache.org/documentation/programming-guide/ > > > >> > > [3] https://issues.apache.org/jira/browse/BEAM-1265 > > > >> > > [4] > > > >> > > https://issues.apache.org/jira/issues/?jql=status%20%3D%20Op > > > >> > > en%20AND%20labels%20%3D%20sdk-consistency > > > >> > > [5] https://issues.apache.org/jira/browse/BEAM-1218 > > > >> > > [6] https://pypi.python.org/pypi/google-cloud-dataflow/json > > > >> > > [7] https://issues.apache.org/jira/browse/BEAM-1251 > > > >> > > > > > >> > > > > >> > > > > >> > Great summary, Ahmet. Thanks. > > > >> > > > > >> > Cheers, > > > >> > > > > >> > -- > > > >> > Sergio Fernández > > > >> > Partner Technology Manager > > > >> > Redlink GmbH > > > >> > m: +43 6602747925 > > > >> > e: sergio.fernan...@redlink.co > > > >> > w: http://redlink.co > > > >> > > > > >> > > > > > >
