Thank you Prabeesh and Sergio for fixing those! On Tue, Jan 31, 2017 at 4:51 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote:
> Awesome, thanks Sergio ! Much appreciated ;) > > Regards > JB > > > On 01/31/2017 01:42 PM, Sergio Fernández wrote: > >> PR #1879 provides the basics: https://github.com/apache/beam/pull/1879 >> >> On Tue, Jan 31, 2017 at 1:33 PM, Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >> No, that's fine as soon as we clearly document the prerequisite for the >>> build. IMHO, we should provide quick BUILDING instructions in the >>> README.md. >>> >>> Regards >>> JB >>> >>> >>> On 01/31/2017 01:24 PM, Sergio Fernández wrote: >>> >>> Originally we integrate the build in Maven with the default profile. >>>> Do you feel like it'd be better to have it under a separated profile or >>>> so? >>>> >>>> On Tue, Jan 31, 2017 at 11:07 AM, Jean-Baptiste Onofré <j...@nanthrax.net >>>> > >>>> wrote: >>>> >>>> Just to be clear, the prerequisite to be able to build the Python SDK >>>> are: >>>> >>>>> >>>>> apt-get install python-setuptools >>>>> apt-get install python-pip >>>>> >>>>> It's also required by the default "regular" build. >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> >>>>> On 01/31/2017 11:02 AM, Jean-Baptiste Onofré wrote: >>>>> >>>>> Just one thing I noticed (and can be helpful for others): to build Beam >>>>> >>>>>> we now need python setuptools installed. >>>>>> >>>>>> For instance, on Ubuntu, you have to do: >>>>>> >>>>>> apt-get install python-setuptools >>>>>> >>>>>> Same for the pip distribution. >>>>>> >>>>>> I guess (if not already done), we have to update README/Building >>>>>> instructions. >>>>>> >>>>>> Correct ? >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On 01/31/2017 08:10 AM, Ahmet Altay wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>>> >>>>>>> This merge is completed. Python SDK is now officially part of the >>>>>>> master >>>>>>> branch! Thank you all for the support. Please open an issue, if you >>>>>>> notice >>>>>>> a reference to the now obsolete python-sdk branch in the >>>>>>> documentation. >>>>>>> >>>>>>> There will not be any more merges to the python-sdk branch. Going >>>>>>> forward >>>>>>> please use the master branch for Python SDK development. There are a >>>>>>> few >>>>>>> existing open PRs to the python-sdk [1]. If you are the author of one >>>>>>> of >>>>>>> those PRs, please rebase them on top of master. >>>>>>> >>>>>>> Thank you, >>>>>>> Ahmet >>>>>>> >>>>>>> [1] https://github.com/pulls?utf8=✓&q=is%3Aopen+is%3Apr+base% >>>>>>> <https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+base%25> >>>>>>> <https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+base%25 >>>>>>> > >>>>>>> <https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+base%25 >>>>>>> > >>>>>>> 3Apython-sdk+repo%3Aapache%2Fbeam+ >>>>>>> <https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr >>>>>>> +base%3Apython-sdk+repo%3Aapache%2Fbeam+> >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 20, 2017 at 10:06 AM, Kenneth Knowles >>>>>>> <k...@google.com.invalid> >>>>>>> wrote: >>>>>>> >>>>>>> To clarify the implied criteria of that last exchange, it is "An SDK >>>>>>> >>>>>>> should >>>>>>>> have at least one runner that can execute the complete model (may >>>>>>>> be a >>>>>>>> direct runner)" >>>>>>>> >>>>>>>> I want to highlight this, because whether an _SDK_ supports >>>>>>>> unbounded >>>>>>>> data >>>>>>>> is not particularly well-defined, and will evolve: >>>>>>>> >>>>>>>> - With the Runner API, an SDK will need to support building a graph >>>>>>>> with >>>>>>>> unbounded constructs, as today with probably minimal changes. >>>>>>>> >>>>>>>> - With the Fn API, if any part of the Fn API is specific to >>>>>>>> unbounded >>>>>>>> data, the SDK will need to implement it. I think right now there is >>>>>>>> no such >>>>>>>> thing, and we don't want such a thing, so SDKs implementing the Fn >>>>>>>> API >>>>>>>> automatically support unbounded data. >>>>>>>> >>>>>>>> - There will also likely be an SDK-specific shim just as there is >>>>>>>> today, >>>>>>>> to leverage idiomatic deserialized representations. The richness of >>>>>>>> this >>>>>>>> shim will decrease so that it will need to "support" unbounded data >>>>>>>> but >>>>>>>> that will be a ~one liner. >>>>>>>> >>>>>>>> Getting the Python SDK on master will accelerate our progress >>>>>>>> towards >>>>>>>> the >>>>>>>> Fn API - partly technical, partly community - which is the best path >>>>>>>> towards support for unbounded data across multiple runners. I think >>>>>>>> the >>>>>>>> criteria are written with the completed portability framework in >>>>>>>> mind. So >>>>>>>> this exchange makes me actually more convinced we should merge >>>>>>>> python-sdk >>>>>>>> to master. >>>>>>>> >>>>>>>> On Fri, Jan 20, 2017 at 9:53 AM, Robert Bradshaw < >>>>>>>> rober...@google.com.invalid> wrote: >>>>>>>> >>>>>>>> On Thu, Jan 19, 2017 at 11:56 PM, Dan Halperin >>>>>>>> >>>>>>>> <dhalp...@google.com.invalid> wrote: >>>>>>>>> >>>>>>>>> I do not think that Python SDK yet meets the bar [1] for >>>>>>>>> implementing >>>>>>>>> >>>>>>>>>> >>>>>>>>>> the >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> Beam model -- supporting Unbounded data is very important. That >>>>>>>> said, >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> given >>>>>>>>>> >>>>>>>>> >>>>>>>>> the committed and sustained set of contributors, it generally makes >>>>>>>>> >>>>>>>>>> >>>>>>>>>> sense >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> to me to make an exception in anticipation of these features being >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> fleshed >>>>>>>>>> >>>>>>>>> >>>>>>>>> out soon; including potentially new users/contributors that would >>>>>>>>> >>>>>>>>>> >>>>>>>>>> arrive >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> once in master. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> [1] https://lists.apache.org/thread.html/CAAzyFAxcmexUQnbF=Y >>>>>>>>>> k0plmm3f5e5bqwjz4+c5doruclnxo...@mail.gmail.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> That is a valid point. The Python SDK supports all the unbounded >>>>>>>>> parts >>>>>>>>> of the model except for unbounded sources, which was deferred while >>>>>>>>> seeing how https://s.apache.org/splittable-do-fn played out. I've >>>>>>>>> been >>>>>>>>> working with the team and merging/reviewing most of their code, and >>>>>>>>> have full confidence this will be coming (and on that note can >>>>>>>>> vouch >>>>>>>>> for a healthy community and support which are much harder to add >>>>>>>>> later). >>>>>>>>> >>>>>>>>> In short, I think it has the required maturity, and I'm in favor of >>>>>>>>> merging soonish. >>>>>>>>> >>>>>>>>> On Wed, Jan 18, 2017 at 12:24 AM, Ahmet Altay >>>>>>>>> >>>>>>>>> <al...@google.com.invalid >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thank you all for the comments so far. I would follow the process >>>>>>>>>> as >>>>>>>>>> >>>>>>>>>> suggested by Davor and others in this thread. >>>>>>>>>>> >>>>>>>>>>> Ahmet >>>>>>>>>>> >>>>>>>>>>> On Tue, Jan 17, 2017 at 11:47 PM, Sergio Fernández < >>>>>>>>>>> wik...@apache.org >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Tue, Jan 17, 2017 at 5:22 PM, Ahmet Altay >>>>>>>>>>>> >>>>>>>>>>>> <al...@google.com.invalid >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> tl;dr: I would like to start a discussion about merging >>>>>>>>>>>>> python-sdk >>>>>>>>>>>>> >>>>>>>>>>>>> branch >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> to master branch. Python SDK is mature enough and merging it to >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> master >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> will >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> accelerate its development and adoption. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Good point, Ahmet! >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I've following closed the development since it was imported in >>>>>>>>>>>> June. >>>>>>>>>>>> >>>>>>>>>>>> For >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> the prototypes I've implemented so far it works quite well; I >>>>>>>>> guess >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> we'd >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> just need to focus the next months in bringing more runners >>>>>>>>> support. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> With a great effort from a lot of contributors(*), Python SDK [1] >>>>>>>>>>>> is >>>>>>>>>>>> >>>>>>>>>>>> now >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> a >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> mostly complete, tested, performant Python implementation of the >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Beam >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> model. Since June, when we first started with Python SDK in >>>>>>>>> Apache >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Beam >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> we >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> have been continuously improving it. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I wouldn't merge during the preparation of 0.5.0 release, but >>>>>>>>>>>>> >>>>>>>>>>>> after >>>>>>>>>>>> >>>>>>>>>>>> that >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> could be a good time to merge back into master. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> ** Python SDK currently supports: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> * Model: All main concepts are present (ParDo, GroupByKey, >>>>>>>>>>>>> >>>>>>>>>>>>> Windowing >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> etc.). >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> * IO: There are extensible APIs for writing new bounded sources >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> and >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> sinks. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Implementations are provided for Text, Avro, BigQuery, and >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Datastore. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> * Runners: Python SDK has an extensible base runner module that >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> allows >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> building specific runners on top of it. The SDK comes with two >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> pipeline >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> runners: DirectRunner and DataflowRunner; and it is possible to >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> add >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> more. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> The existing runners are currently limited to bounded execution >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> and >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> otherwise equivalent to their Java SDK counterparts in >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> functionality. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>>> What would the effort of porting, and maintaining, parallel >>>>>>>>>>>>> >>>>>>>>>>>> versions >>>>>>>>>>>> >>>>>>>>>>>> of >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> the >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Java runners? I guess I'd need to dig deeper in the model, but >>>>>>>>>>> this >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> may >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> represent a major effort for the project, right? >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> It is somewhat higher for DirectRunner because DirectRunner also >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> implements >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> the code for execution. It is not that high for DataflowRunner >>>>>>>>> >>>>>>>>>> >>>>>>>>>> because >>>>>>>>>>> >>>>>>>>>>> the >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> base runner module has a lot of helpers with the right hooks for >>>>>>>>> >>>>>>>>>> >>>>>>>>>> implementing a generic runner. I would _expect_ the experience in >>>>>>>>>>> >>>>>>>>>>> general >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> would be similar to the latter. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> * Testing: Python SDK implements ValidatesRunner test framework >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> for >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> implementing integration test for current and future runners. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> There >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> is >>>>>>>> >>>>>>>>> >>>>>>>>> unit >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> test coverage for all modules, and a number of integrations test >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> for >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> validating existing runners. >>>>>>>> >>>>>>>>> >>>>>>>>> * Documentation and examples: Documentation work has started on >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>> Python >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> SDK. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Beam Programming Guide page has been updated to include Python >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [2]. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> The >>>>>>>> >>>>>>>>> >>>>>>>>> code comes with many ready to use examples and we are in a good >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> place >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> to >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> start documenting those on the website. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ** We are not done yet, next on the roadmap we have: >>>>>>>>>>>>> >>>>>>>>>>>>> * Streaming: Both of the existing runners lack support for >>>>>>>>>>>>> >>>>>>>>>>>>> streaming >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> execution, and currently there is work going on for adding >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> streaming >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> support to DirectRunner [3]. >>>>>>>> >>>>>>>>> >>>>>>>>> * Documentation: Filling the rest of the Beam documentations with >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>> Python >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> SDK specific information and examples. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> * SDK consistency: Making Python SDK consistent with the Java >>>>>>>>>>>>> SDK. >>>>>>>>>>>>> >>>>>>>>>>>>> We >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> have >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> come a long way on this and have only a few items left [4]. >>>>>>>>>>>> >>>>>>>>>>>>> * Beamifying: We have been working on removing >>>>>>>>>>>>> Dataflow-specific >>>>>>>>>>>>> >>>>>>>>>>>>> references >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> both from the documentation and from the code. There is some >>>>>>>>>>>> work >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> left, >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> and >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> we are currently working on those as well [5]. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ** Steps and implications of merging to master: >>>>>>>>>>>>> >>>>>>>>>>>>> * Master branch is merged to python-sdk branch at regular >>>>>>>>>>>>> >>>>>>>>>>>>> intervals >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> and >>>>>>>> >>>>>>>>> >>>>>>>>> the >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> last merge was on 12/22. All the past merges were uneventful >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> because >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> there >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> is a minimal overlap in modified files between branches. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Integrating >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> python-sdk to master will similarly touch a small number of >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> existing >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> files. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> * Python SDK is using the same tools for building and testing. >>>>>>>>>>>>> It >>>>>>>>>>>>> >>>>>>>>>>>>> is >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> already integrated with Maven, Jenkins and Travis. Specifically >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> the >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> impact >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> to the testing infrastructure would be: >>>>>>>>>>>> >>>>>>>>>>>>> - There will be two additional test configurations in Travis. >>>>>>>>>>>>> >>>>>>>>>>>>> Since >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Travis >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> runs all configurations in parallel there should not be a >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> noticeable >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> change >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> in the Travis run time. >>>>>>>>>>>> >>>>>>>>>>>>> - Jenkins pre-commit test will start running the Python SDK >>>>>>>>>>>>> tests. >>>>>>>>>>>>> >>>>>>>>>>>>> It >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> will >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> add an additional 5 minutes to the completion time of pre-commit >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> test. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Historically Python SDK tests were not flaky and did not cause >>>>>>>>> any >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> random >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> failures. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> - Jenkins Python post-commit test is already separated from the >>>>>>>>>>>>> >>>>>>>>>>>>> other >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> post-commit tests and will continue to exist. It would not change >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> the >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> testing time for any other test. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> * The release process needs to be updated to accommodate >>>>>>>>>>>>> releasing >>>>>>>>>>>>> >>>>>>>>>>>>> Python >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> artifacts. Python SDK would fit in the existing release schedule >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> and >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> could >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> be released along with the Java SDK. The additional steps would >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> include: >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> - Generating Python artifacts. This could be done with a single >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> command >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> using Maven today. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> - Publishing the artifacts to a central repository such as PyPI. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I'm more than happy to help on this. We left on purpose some >>>>>>>>>>>>> >>>>>>>>>>>> things >>>>>>>>>>>> >>>>>>>>>>>> open >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> when we added Maven support to the Python build. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> That would be awesome. We can coordinate on that post-merge. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> - Updating the release guide to reflect the changes above. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> * Users: There are existing users using the Python SDK. To >>>>>>>>>>>>> give a >>>>>>>>>>>>> >>>>>>>>>>>>> rough >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> estimate, a distribution of the Beam Python SDK had a total of >>>>>>>>> 23K >>>>>>>>> >>>>>>>>>> >>>>>>>>>> downloads in the past 6 months [6]. Some of those users are >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> already >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> engaged >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> with the community (e.g. [7]). There might be an increased amount >>>>>>>>>>>> >>>>>>>>>>>>> engagement from the rest of them after the merge. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Python 3 support is something we definitively need to look >>>>>>>>>>>>> ahead. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I'd >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> try >>>>>>>> >>>>>>>>> >>>>>>>>> to make the codebase compatible with both 2.7.x and 3.6.x, rather >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> than >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> using other solutions like 2to3. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> I agree with you. I think it makes more sense to make codebase >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> compatible >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> with both. As you mentioned Python 3 support is not a short-term >>>>>>>>> goal >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> in >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> the roadmap, and we can discuss it more as we approach that. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Looking forward to hearing your thoughts and comments on >>>>>>>>>>>> >>>>>>>>>>>> “graduating” >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> python-sdk to the master. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>>>>> Ahmet >>>>>>>>>>>>> >>>>>>>>>>>>> (*) Python SDK branch currently has a diverse group of >>>>>>>>>>>>> >>>>>>>>>>>>> contributors. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Regular contributors include Charles Chen, Chamikara Jayalath, >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> María >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> García >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Herrero, Mark Liu, Pablo Estrada, Robert Bradshaw (Apache Beam >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> PMC), >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Sourabh Bajaj, and Vikas Kedigehalli. We have also had >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> contributions >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> from >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Abdullah Bashir, Marco Buccini, Sergio Fernández, Seunghyun Lee, >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> and >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Younghee Kwon. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> [1] https://github.com/apache/beam/tree/python-sdk/sdks/python >>>>>>>>>>>>> [2] https://beam.apache.org/documentation/programming-guide/ >>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-1265 >>>>>>>>>>>>> [4] >>>>>>>>>>>>> https://issues.apache.org/jira/issues/?jql=status%20%3D%20Op >>>>>>>>>>>>> en%20AND%20labels%20%3D%20sdk-consistency >>>>>>>>>>>>> [5] https://issues.apache.org/jira/browse/BEAM-1218 >>>>>>>>>>>>> [6] https://pypi.python.org/pypi/google-cloud-dataflow/json >>>>>>>>>>>>> [7] https://issues.apache.org/jira/browse/BEAM-1251 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Great summary, Ahmet. Thanks. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Sergio Fernández >>>>>>>>>>>> Partner Technology Manager >>>>>>>>>>>> Redlink GmbH >>>>>>>>>>>> m: +43 6602747925 >>>>>>>>>>>> e: sergio.fernan...@redlink.co >>>>>>>>>>>> w: http://redlink.co >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>> >>>>> Jean-Baptiste Onofré >>>>> jbono...@apache.org >>>>> http://blog.nanthrax.net >>>>> Talend - http://www.talend.com >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>> Jean-Baptiste Onofré >>> jbono...@apache.org >>> http://blog.nanthrax.net >>> Talend - http://www.talend.com >>> >>> >> >> >> > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
