Re: Using user developped source in streamline python

Hi Sébastien, Currently there is no work in progress for including the write transforms for the locations you listed. You could develop your own version if interested. Please see WriteToBigquery transform [1] for reference. Ahmet [1] https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dc

Re: [DISCUSS] Automation for Java code formatting

+1 This is great idea. Does anyone know a similar tool for python? I believe go already has this as part of its tools with go fmt. On Tue, Jun 26, 2018 at 9:55 PM, Ankur Goenka wrote: > +1 > > Intellij can help but still formatting is an additional thing to keep in > mind. Enabling auto format

Re: [RESULT][VOTE] Apache Beam, version 2.5.0, release candidate #2

Beam site pull request is merged. JB, please send release announcement whenever it is convenient for you. Thank you again! Ahmet On Mon, Jun 25, 2018 at 10:39 PM, Ahmet Altay wrote: > Thank you JB! > > On Mon, Jun 25, 2018 at 10:34 PM, Jean-Baptiste Onofré > wrote: > >&

Re: [RESULT][VOTE] Apache Beam, version 2.5.0, release candidate #2

; a asfbot merge this morning). > > Regards > JB > > On 25/06/2018 19:49, Ahmet Altay wrote: > > JB, thank you for making this release happen. > > > > I noticed that python artifacts are not deployed to pypi yet. Would you > > like me to do that? > > > &

Re: [RESULT][VOTE] Apache Beam, version 2.5.0, release candidate #2

On Mon, Jun 25, 2018 at 10:49 AM, Ahmet Altay wrote: > JB, thank you for making this release happen. > > I noticed that python artifacts are not deployed to pypi yet. Would you > like me to do that? > > Thank you, > Ahmet > > On Sat, Jun 23, 2018 at 6:45 AM, Rafael Fer

Re: [RESULT][VOTE] Apache Beam, version 2.5.0, release candidate #2

>> Regards >> JB >> >> On 23/06/2018 05:17, Jean-Baptiste Onofré wrote: >> > Hi all, >> > >> > I'm happy to announce that we have unanimously approved this release. >> > >> > There are 12 approving votes, 5 of which are binding: >&g

Re: [PROPOSAL] Add a blog post for Beam release 2.5.0

Thank you Alexey! It is a great idea. I added my suggestions to the doc. On Fri, Jun 22, 2018 at 1:01 PM, Alexey Romanenko wrote: > Hi everyone, > > I propose to continue a tradition of publishing new blog post for Beam web > site (as it was positively accepted by community before) with announc

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

I already voted a +1 2 days ago. Again: +1 (binding) Thank you JB! On Fri, Jun 22, 2018 at 9:23 AM, Alan Myrvold wrote: > The https://dist.apache.org/repos/dist/dev/beam/2.5.0/ > apache-beam-2.5.0-source-release.zip still contains > the beam-release-2.5.0/gradle/wrapper/gradle-wrapper.jar. Do

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

nch with that > exclusion: https://github.com/pabloem/incubator-beam/ > archive/v2.5.0-TEST-ARCHIVE.zip > Best > -P. > > On Wed, Jun 20, 2018 at 11:00 AM Ahmet Altay wrote: > >> I support Kenn's proposal of removing binary artifacts and otherwise >> keeping the

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

I support Kenn's proposal of removing binary artifacts and otherwise keeping the RC2. We have done it in the pas with similar binary files that were unintended. At the same time, it will follow the suggestion from LEGAL-288. As a follow up we can build this removal into our release process until th

Re: [PROPOSAL] Merge samza-runner to master

Thank you for everyone who contributed to this runner. It is really great to see this. Xinyu, for the people like myself who were not following the development closely, could you talk about missing pieces, work in progress, future plans? On Mon, Jun 18, 2018 at 3:37 PM, Rafael Fernandez wrote:

Re: [DISCUSS] Releasing Beam in the presence of emergencies

Thank you Rafael. I think it is a good idea to include our commitment, including concrete steps on our website. This would make it easier for enterprise users to choose Beam. Even though this is already partially Apache policy and there is precedence in our project with 2.1.1 release; increasing t

Re: Proposing interactive beam runner

Thank you Sindy. I like the demo; it looks great. This would be interesting to a lot of users. What are your plans for moving this forward? What kind of an input you are looking for? Ahmet On Wed, Jun 13, 2018 at 2:32 PM, Eugene Kirpichov wrote: > This is awesome, thanks Sindy! I hope that the

Re: Proposal: keeping post-commit tests green

On Wed, Jun 13, 2018 at 3:52 PM, Mikhail Gryzykhin wrote: > Hi Ahmet, > > I've checked on tests status and most of other tests are green 98% of the > time. So I feel that we do not need any explicit actions for those tests. > Is it going to be a one time action to fix existing flaky tests? Or is

Re: Proposal: keeping post-commit tests green

On Wed, Jun 13, 2018 at 3:45 PM, Mikhail Gryzykhin wrote: > Hello everybody, > > Thanks everyone. I didn't receive any more feedback on the design proposal > document [1] and I believe we've reached consensus. I've added > implementation tasks in JIRA (BEAM-4559 [2]) and will start coding soon.

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

bugs-annotations/releases >>>>>> [3] https://github.com/apache/beam/pull/5609/commits/ >>>>>> 32c7df706e970557f154ff6bc521b2e00f9d09ab >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>

Re: Beam Dependency Check Report (2018-06-13)

Thanks Yifan, this is great! My unsolicited feedback: - Could it warn against dependencies that did not get updates for a long time? For python there were examples of a dependency being abandoned by its own developers and it took us a while to figure it out and switch to maintained one. (Currently

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

Ismaël, I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing the issue by changing the findbugs back to "com.github.stephenc.findbugs". Is this what you are referring to? Ahmet On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang wrote: > Hey JB, > > I added some instructions about ho

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

Thank you JB. For the wheel artifacts, Boyuan was trying to get the instructions from Robert and reproduce the artifacts. She can help you with this if you need. Ahmet On Mon, Jun 11, 2018 at 10:29 PM, Jean-Baptiste Onofré wrote: > Hi, > > sorry, I missed wheel artifact. Something to add on th

Re: [VOTE] Policies for managing Beam dependencies

I think this is relevant for users. It makes sense for users to know about how Beam work with its dependencies and understand how conflicts will be addressed and when dependencies will be upgraded. On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles wrote: > Do you think this has relevance for user

Re: [VOTE] Policies for managing Beam dependencies

+1 Thank you for driving these decisions. I would make a meta-point, all other recent votes and if passes this one could be converted to web site documents at some point in an easily accessible and linkable way. On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath wrote: > Hi All, > > We recently

Re: [VOTE] Code Review Process

+1 On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles wrote: > +1 > > On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh wrote: > >> As we seem to largely have consensus in "Reducing Committer Load for Code >> Reviews"[1], this is a vote to change the Beam policy on Code Reviews to >> require that >> >>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

+1 On Fri, Jun 1, 2018, 9:32 AM Jason Kuster wrote: > +1 (non-binding): automating policy ensures it is applied fairly and > evenly and lessens the load on project maintainers; hearty agreement. > > On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold wrote: > >> +1 (non-binding) I updated the pull requ

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

Congratulations to all of you! On Thu, May 31, 2018 at 7:26 PM, Chamikara Jayalath wrote: > Congrats to all three!! > > On Thu, May 31, 2018 at 7:09 PM Davor Bonaci wrote: > >> Please join me and the rest of Beam PMC in welcoming the following >> contributors as our newest committers. They have

Re: [PROPOSAL] Preparing 2.5.0 release next week

issue about ParquetIO on HDFS/S3 that I would like to > investigate with the team. > Do you know who is currently investigating the ParquetIO issue? Do you need help with that? > > I plan to start the release process asap, hopefully later today. > > Regards > JB > >

Re: Hello Beam!

Welcome Rui! On Tue, May 29, 2018 at 2:49 PM, Rui Wang wrote: > Hi there, > > I am Rui (pronounced as same as "Ray")! > > I recently joined Google Cloud. Beam is a very interesting project and I > cannot wait to contribute to it! > > > Thanks, > Rui >

Re: [PROPOSAL] Preparing 2.5.0 release next week

ieve JB is referring >> > to https://issues.apache.org/jira/browse/BEAM-4060 >> > >> > On Fri, May 18, 2018 at 10:16 AM Scott Wegner > > <mailto:sweg...@google.com>> wrote: >> > >> > J.B., can you give any

Re: [VOTE] Go SDK

+1 (binding) Congratulations to the team! On Tue, May 22, 2018 at 10:13 AM, Alan Myrvold wrote: > +1 (non-binding) > Nice work! > > On Tue, May 22, 2018 at 9:18 AM Pablo Estrada wrote: > >> +1 (binding) >> Very excited to see this! >> >> On Tue, May 22, 2018 at 9:09 AM Thomas Weise wrote: >>

Re: [PROPOSAL] Preparing 2.5.0 release next week

a release blocker, could you please add it to the list? > > Just an FYI. Since the fix is likely small fixes to build file it seems ok > to cut the branch and cherry pick. > > Kenn > > On Thu, May 17, 2018, 17:41 Ahmet Altay wrote: > >> Hi JB and all, >> >>

Re: [PROPOSAL] Preparing 2.5.0 release next week

Hi JB and all, I wanted to follow up on my previous email. The python streaming issue I mentioned is resolved and removed from the blocker list. Blocker list is empty now. You can go ahead with the release branch cut when you are ready. Thank you, Ahmet On Sun, May 13, 2018 at 8:43 AM, Jean-Bap

Re: [PROPOSAL] Preparing 2.5.0 release next week

Hi JB, We found an issue related to using side inputs in streaming mode using python SDK. Charles is currently trying to find the root cause. Would you be able to give him some additional time to investigate the issue? Charles, do you have a JIRA issue on the blocker list? Thank you everyone for

Re: [PROPOSAL] Python 3 support

to myself? > And edit rights to the Kanban board? > > Robbe > > On Tue, 17 Apr 2018 at 22:56 Ahmet Altay wrote: > >> Kanban board for python 3: https://issues.apache.org/ >> jira/secure/RapidBoard.jspa?rapidView=245 >> >> (Thank you Davor!) >> >>

Re: Merge options in Github UI are confusing

I agree with Robert. In this case one size does not fit all. There are times, another round trip with a contributor would be frustrating to the author. Especially for new contributors. Having the option to squash and merge is useful in those cases. (For reference in the past we even helped new cont

Re: [PROPOSAL] Python 3 support

Kanban board for python 3: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245 (Thank you Davor!) Ahmet On Fri, Apr 6, 2018 at 6:32 PM, Reuven Lax wrote: > I had a similar problem. > > On Fri, Apr 6, 2018, 6:23 PM Ahmet Altay wrote: > >> I tried to crea

Re: Gradle Status [April 11]

> Found another blocker in current artifacts creations: there is not pom.xml and pom.properties in META-INF. This is used by tools + libraries + integrations so it is quite important to not break it Romain, is there a JIRA for this issues? If not could you create one please? On Thu, Apr 12, 2018

Re: [PROPOSAL] Preparing 2.5.0 release next week

+1 to delaying 2 weeks. I think it will be prudent to wait in this case. There is too much in flux with Gradle migration currently and based on Scott's latest update I think we will be in a more stable state in 2 weeks. Last Beam release date was 3/20 and our plan was to make a release every 6 wee

Re: [PROPOSAL] Python 3 support

ts >>>>> unwieldy, perhaps a spreadsheet with modules/packages on one axis and the >>>>> various automated/manual conversions along the other would be helpful? >>>>> >>>>> A note on automated tools, they're sometimes overly conservat

Re: Python SDK feature set

On Mon, Apr 2, 2018 at 6:17 PM, Thomas Weise wrote: > Hi, > > I’m trying to find a summary of the feature set that is currently > supported in the Python SDK. I understand it is experimental and > currently only supports a subset of the Beam model like fixed interval > windows but not merging win

Re: [ANNOUCEMENT] New Foundation members!

Congratulations to all of you! On Fri, Mar 30, 2018, 4:29 PM Pablo Estrada wrote: > Congratulations y'all! Very cool. > Best > -P. > > On Fri, Mar 30, 2018 at 4:09 PM Davor Bonaci wrote: > >> Now that this is public... please join me in welcoming three newly >> elected members of the Apache Sof

Re: [PROPOSAL] Python 3 support

ubpackage. >> Any insights on how to best accomplish this? >> > So you can look at some of the recent changes to tox.ini in the git log to > see what we’ve done so far around this I suspect you can repeat that same > pattern. > +1 updating tox.ini and adding new checks to run_

Re: Dataflow throwing backend error

Hi Rajesh, This looks like a transient error from GCS. Beam SDK will retry tasks in the face of such errors and those typically do not make your pipeline fail. If you have additional questions please reach out to Dataflow support ( https://cloud.google.com/dataflow/support). Thank you, Ahmet On

Re: executing the pipeline from datalab

+ user, dev to bcc Eila, Is it possible that you are using an old version? I remember pending was missing in the dictionary and was added later. If that is not the reason, could you file a JIRA issue? Thank you, Ahmet On Fri, Mar 23, 2018 at 6:15 AM, Jean-Baptiste Onofré wrote: > Hi Eila, >

Re: [PROPOSAL] Python 3 support

Thank you Robbe. I reviewed the document it looks reasonable to me. I will touch on some points that were not mentioned: - Runner exercise different code paths. Doing auto conversions and focusing on DirectRunner is not enough. It is worthwhile to run things on DataflowRunner as well. This can be

Re: How to decide how much quota do I need

+ user, dev to bcc Eila, there is some information here: https://cloud.google.com/dataflow/quotas on quotas in general. Specifically for in use IP addresses, you can look at autoscaling messages and see what was autoscaling trying to upscale to. It is also possible to use large machine types (e.g

Re: Python PostCommit Broken

https://issues.apache.org/jira/browse/BEAM-3922 is the JIRA for tracking this. On Fri, Mar 23, 2018 at 10:51 AM, Pablo Estrada wrote: > Hello everyone, > I see that the Python PostCommit has been broken for a couple days. Is > there a PR / JIRA to track this? > See breakage: https://builds.apach

Re: [PROPOSAL] Scripting extension based on Java JSR-223

Thank you Ismaël, this looks really cool. On Fri, Mar 23, 2018 at 5:33 AM, Jean-Baptiste Onofré wrote: > Hi, > > it sounds like a very good extension mechanism to PTransform. > > +1 > > Regards > JB > > On 03/23/2018 12:03 PM, Ismaël Mejía wrote: > > This is a really simple proposal to add an ex

Re: Apache beam DataFlow runner throwing setup error

Hi Rajesh, Have you looked at the worker-startup logs [1]? You should be able to see the setup error there. It is possible that something in your requirements file is failing to install in the workers. If that is the case, see Managing Python Pipeline Dependencies [2] for alternative options. You

Re: Pubsub API feedback

Thank you Udi. Left some high level comments on the PR. On Mon, Mar 19, 2018 at 5:13 PM, Udi Meiri wrote: > Hi, > I wanted to get feedback about the upcoming Python Pubsub API. It is > currently experimental and only supports reading and writing UTF-8 strings. > My current proposal only concern

Re: [VOTE] Release 2.4.0, release candidate #3

I was able to run hourly_team_score. I was passing a wrong argument. No need for an alarm. :) On Mon, Mar 19, 2018 at 5:33 PM, Ahmet Altay wrote: > +1 Thank you Robert. > > Verified python mobile gaming examples using the wheel files on direct > runner. Got user_score

Re: [VOTE] Release 2.4.0, release candidate #3

+1 Thank you Robert. Verified python mobile gaming examples using the wheel files on direct runner. Got user_score working but hourly_team_score failed with ( https://issues.apache.org/jira/browse/BEAM-3824). Since this is an example, I think it is fine to continue with the release. I will work on

Re: Contributing

Hi Austin, It was great meeting with you. We mentioned a list of starter bugs, here is that list [1]. It might give you some ides on where to start. [1] https://issues.apache.org/jira/browse/BEAM-3844?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20resolution%20%3D%20U

Re: Proposal: build Python wheel distributions for Apache Beam releases

release > artifacts should be staged. (Eventually, of course, they'll be published to > PyPi). Should they be placed alongside the source artifacts in the svn > repository? > > > On Wed, Mar 7, 2018 at 3:00 PM Ahmet Altay wrote: > >> Are we planning to do this for

Re: Proposal: build Python wheel distributions for Apache Beam releases

ython is present. We can (1) make sure Cython is > installed before installing apache beam into the container, and (2) > assert as part of the build process that these modules exist. > > > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay wrote: > >> > >> +1 to wheels. The main

Re: [VOTE] Release 2.4.0, release candidate #1

-1 for the same reason as Ismaël. Python version is not updated in the release branch [1]. [1] https://github.com/apache/beam/blob/release-2.4.0/sdks/python/apache_beam/version.py#L21 On Wed, Mar 7, 2018 at 8:39 AM, Jean-Baptiste Onofré wrote: > No it's not (I'm testing the release right now),

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

(my guess is that 3.4 is the lowest we'll > want to go). > Thank you for this explanation. Filed ( https://issues.apache.org/jira/browse/BEAM-3781) as a reminder to figure this out once have some reasonable python 3 support. > > > On Fri, Mar 2, 2018 at 11:06 AM Ahmet Altay

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

on in Spark is 3.4. We could enable lint tests in Jenkins and see how > they go? > > On Fri, Mar 2, 2018 at 10:06 AM Alan Myrvold wrote: > >> I ran "python3 --version" on each worker and all showed python 3.4.3. Is >> that too old? >> >> >> On

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

n infra? (e.g. it's > important but out of my hands). > > On Mar 1, 2018 11:05 PM, "Ahmet Altay" wrote: > >> I think we should prioritize the issue of installing Python 3 on the >> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would >> a

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

I think we should prioritize the issue of installing Python 3 on the workers (https://issues.apache.org/jira/browse/BEAM-3671). I would appreciate if folks pay attention to these 2 steps but I am worried that it will be easily forgotten. On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau wrote: > I ma

Re: Instructions to install Python SDK from source

Hi Pablo, You have a great point. Getting started instructions for developers for Python SDK is not well documented. Installing Python SDK from source is a subset of this lack of documentation. There is a JIRA for this ( https://issues.apache.org/jira/browse/BEAM-3075). It would be great if you ca

Re: Python 3 flake 8: splitting up on the errors?

I think this is a great idea. I would encourage everyone who would like to help with Python 3 migration to help with this effort. Holden, if you already have a list, could you either share the list or create individual JIRAs so that we can track the work among us. On Tue, Feb 27, 2018 at 4:53 PM,

Re: Proposed improvements to our documentation

+1 I think this is a great idea, it can also serve as an inventory of where a language might be lacking in transforms and provide a good starting point for new contributors to fill in those gaps by looking at the existing Java implementations. On Wed, Feb 28, 2018 at 10:53 AM, Lukasz Cwik wrote:

Re: Python 3 reviewers

Thank you Holden for doing this work. I agree with Robert's comment. I know there are a few folks working on this now (you, @luke-zhu and @cclauss). Perhaps you could do python 3 related code reviews within that group. I would be happy to chime in and review some chunks as well. On Thu, Feb 22, 20

Re: [DISCUSS]: Beam 2.3.0 release archetypes missing mobile gaming examples

In my opinion waiting for the 2.4.0 release makes sense, since there is a plan to cut 2.4.0 release soon and this is examples and not a core function. In the meantime, we could add a notice to the website warning users about this issue and suggest them to use the previous release for trying out the

Re: Beam 2.4.0

+1 for having regular release cycles. Finalizing a release takes time in the order of a few weeks and starting a new release soon after the previous one is a reliable way for having releases every 6 weeks. On Tue, Feb 20, 2018 at 2:30 PM, Robert Bradshaw wrote: > Yep. I am starting the "Let's do

Re: [VOTE] Release 2.3.0, release candidate #3

On Fri, Feb 16, 2018 at 9:52 PM, Jean-Baptiste Onofré wrote: > Hi, > > Can someone from Python grand me permission to upload Python SDK 2.3.0 to > PyPi ? > lukecwik, kennknowles, aljoscha, robertwb, davorbonaci are the package owners, any of them can do it. > > My user is jbonofre. > > Thanks

Re: [VOTE] Release 2.3.0, release candidate #3

+1 Thank you JB and thank you everyone for doing the validations. On Wed, Feb 14, 2018 at 5:24 PM, Eugene Kirpichov wrote: > Thanks Kenn. I retract my -1, but then someone must verify it with Flink > 1.4. I might give it a shot tomorrow (installing Flink 1.4 on Dataproc). > > On Wed, Feb 14, 20

Re: Proposal: build Python wheel distributions for Apache Beam releases

+1 to wheels. The main effort for this would be updating the release guide, and adding support for other platforms in Jenkins for building and testing wheels. In light of this, maybe we can prioritize having test infrastructure for other platforms. On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía w

Re: Off for 3 weeks

Best wishes, hope you will recover quickly. On Mon, Feb 12, 2018 at 9:03 AM, Kenneth Knowles wrote: > Best wishes for a swift recovery. > > Kenn > > > On Sat, Feb 10, 2018 at 5:56 AM, Etienne Chauchot > wrote: > >> Hi guys, >> >> I've been off this week for a surgery. I will not be available un

Re: [VOTE] Release 2.3.0, release candidate #2

+1 I verified python quick start, mobile gaming examples, streaming on Direct and Dataflow runners. Thank you JB! On Thu, Feb 8, 2018 at 2:27 AM, Romain Manni-Bucau wrote: > +1 (non-binding), thanks JB for the effort! > > > Romain Manni-Bucau > @rmannibucau |

Re: Replacing Python DirectRunner apply_* hooks with PTransformOverrides

+1 to this change. Thank you Charles for improving the DirectRunner, sharing your progress and seeking feedback. This change would allow us to migrate to a faster DirectRunner for Python. A long time requested feature and an important part of the first use experience for new users trying out Beam.

Re: [VOTE] Release 2.3.0, release candidate #1

On Wed, Jan 31, 2018 at 2:09 PM, Ahmet Altay wrote: > This will require a change in the Beam code, because image names are > hardcoded in to code (python) and configuration (java). RC1 as it is will > not work correctly with Cloud Dataflow. > Please ignore this. Looking at the

Re: [VOTE] Release 2.3.0, release candidate #1

This will require a change in the Beam code, because image names are hardcoded in to code (python) and configuration (java). RC1 as it is will not work correctly with Cloud Dataflow. On Wed, Jan 31, 2018 at 2:08 PM, Reuven Lax wrote: > Hopefully we can validate soon. I believe some of the delays

Re: [HEADS UP] Preparing Beam 2.3.0

I moved BEAM-3559 out of 2.3.0, it is not targeted for this release. On Mon, Jan 29, 2018 at 5:22 AM, Jean-Baptiste Onofré wrote: > Hi, > > new update (and I hope the last one ;)): > > - BEAM-3551 has been reviewed and merged. This Jira is fixed for 2.3.0. > - I just updated the PR for BEAM-793

Re: Does Apache Beam for python support server-based shuffle with Dataflow runner yet?

Hi Nima, You can try this feature with python SDK using the same instructions from the announcement. However, it is not ready for production usage. Team is working official supporting it. We cannot share an ETA, once it is available it will be announced. For future questions related to the Datafl

Re: Pushing daily/test containers for python

ersion. Breaking changes in their interaction would cause confusion and >>>> create noise. Any local tests can also in theory just build the container >>>> images directly and not use any registry, so it might make sense to set up >>>> the tests so that pushing occurs less frequently

Pushing daily/test containers for python

Hi all, After some recent changes (e.g. [1]) we have a feasible container that we can use to test Python SDK on portability framework. Until now we were using Google provided container images for testing and for the released product. We can gradually move away from that (at least partially) for Py

Re: A question regarding BEAM-3280

n to you. (I guess you first need to be added as a contributor to the project). I added comment mentioning that you are working on this issue. Thank you again! Ahmet > > Regards, > Norio Akagi > > > On Dec 11, 2017, at 4:50 PM, Ahmet Altay wrote: > > Hi Norio, > >

Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')

that our releases do not > break > > when newer versions of dependencies are released. > > > > On Fri, Dec 15, 2017 at 2:44 PM Ahmet Altay wrote: > >> > >> On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath < > chamik...@google.com> > >>

Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')

t breaking of already released versions. > > Thanks, > Cham > > On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay wrote: > >> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw >> wrote: >> >>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay wrote: >>

Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')

On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw wrote: > On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay wrote: > > > > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw > > wrote: > >> > >> I am also in favor of pinning as an immediate fix, bumping the bo

Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')

On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw wrote: > I am also in favor of pinning as an immediate fix, bumping the bound > otherwise. > > Regarding putting an upper bound to avoid being broken, the last two > breaks have been due to just having an (unneeded) upper bound (which > held us bac

Re: A personal update

Welcome back! Looking forward to your contributions. Ahmet On Tue, Dec 12, 2017 at 10:05 PM, Jesse Anderson wrote: > Congrats! > > On Wed, Dec 13, 2017, 5:54 AM Jean-Baptiste Onofré > wrote: > >> Hi Davor, >> >> welcome back !! >> >> It's really great to see you back active in the Beam communi

Re: A question regarding BEAM-3280

Hi Norio, Thank you for your interest. If you would like to work on this I can assign the JIRA to you. I do not think this change in sufficient or correct. This reads as if SplitLinesToWordsFn returns a Tuple of things, however instead it produces three unrelated collections of different types. I

Re: Introduction + interest in helping Beam builds, tests, and releases

Welcome Alan, this sounds great! On Thu, Dec 7, 2017 at 8:00 PM, Holden Karau wrote: > Also, and I know this is maybe a bit beyond the scope of what would make > sense initially, but if you wanted to set up something to test BEAM against > the new Spark/Flink RCs we could give feedback about any

Re: Apache Beam, version 2.2.0

On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov wrote: > I've sent the poll https://lists.apache.org/thread.html/ > 5bc2e184a24de9dbc8184ffd2720d1894010497d47d956b395e037df@% > 3Cuser.beam.apache.org%3E > Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll > as well (or ask

Re: Apache Beam, version 2.2.0

Thank you Reuven! I tweeted the release announcement on Beam's account. On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax wrote: > Technically it's a backwards-incompatible change, however if we are > convinced the risk is low we could do it. > > As mentioned on the original thread, it's not clear that

Re: [DISCUSS] Thinking about Beam 3.x roadmap and release schedule

My wishlist for 2018 would be - Python 3 support - Python SDK to work with more runners. This is covered in portability in general. I would like to see an enterprise grade Python SDK that can run on a range of Beam runners. - Related to the above item, full streaming support with Python SDK. - Pyt

Re: Version 2.2.0 release date

Hi Stefania, Release candidate for 2.2.0 is currently being voted [1]. The release will happen after a successful vote. Ahmet [1] https://lists.apache.org/thread.html/da2acabdb15c9f8d11351f9167633a 4b089664fe3cce014ba619c937@%3Cdev.beam.apache.org%3E On Mon, Nov 20, 2017 at 7:04 AM, Stefania Ma

Re: Python SDK DirectRunner (new feature)

Thank you María. On Mon, Nov 20, 2017 at 5:31 PM, María García Herrero < mari...@google.com.invalid> wrote: > Hello, > > I recently worked on adding a bundle retry for the Python SDK DirectRunner > ( > https://issues.apache.org/jira/browse/BEAM-2718). > > The goal was to have a more reliable proc

Re: HDFS Support for Python SDK

Thank you Udi, this is a great comparison of available options. On Mon, Nov 20, 2017 at 5:26 PM, Udi Meiri wrote: > Hi, > > I've done some research into implementing HDFS support for Python SDK and > I'd like your input. This work is regarding BEAM-3099 >

Re: [VOTE] Release 2.2.0, release candidate #4

+1 I verified the python quick start on Windows. I could not verify the documentation changes because the staged version expired. On Mon, Nov 20, 2017 at 12:08 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Thanks Luke. I was able to validate quickstart on Dataflow and on Spark >

Re: python3 support schedule

For reference https://issues.apache.org/jira/browse/BEAM-1251 is the umbrella issue tracking python3 support in the core SDK. There needs to be additional runner specific work (e.g. DataflowRunner needs to use python3 binary on its workers) once the core work is completed. Ahmet On Thu, Nov 2, 20

Re: Hello

Welcome Ankur! On Wed, Nov 1, 2017 at 12:04 PM, Lukasz Cwik wrote: > Welcome, you have been added. > > On Tue, Oct 31, 2017 at 5:31 PM, Ankur Goenka > wrote: > > > Hi Guys, > > > > I recently joined Google and will be contributing to Apache Beam project. > > Please add me to the apache jira. My

Re: [DISCUSS] Move away from Apache Maven as build tool

Has anyone started a POC with Bazel? I would be interested in helping that effort. On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik wrote: > I have started a POC for using Gradle here: > https://github.com/lukecwik/incubator-beam/tree/gradle > > Things that work: > * compiling all Java code (src/main

Re: [Proposal] Sharing Neville's post and upcoming meetups in the Twitter handle

Done. On Thu, Oct 26, 2017 at 11:26 AM, Griselda Cuevas wrote: > Hi folks, could you help us to tweet the second part of Neville's blogpost? > > Here's a suggested tweet: > 2nd part of @sinisa_lyh's post is out! Read how @Spotify developed Scio, a > high level Scala API 4 the Beam Java SDK. http

Re: [Proposal] Sharing Neville's post and upcoming meetups in the Twitter handle

This makes sense to me. I published the first tweet, we can publish the second one perhaps closer to the event. Ahmet On Fri, Oct 20, 2017 at 1:08 PM, Griselda Cuevas wrote: > Hi everyone - What do you think about sharing Neville's blogpost[1] about > the road to Scio on the Apache Beam Twitter

Re: New contributor

Welcome Vilhem! On Wed, Oct 18, 2017 at 4:55 AM, Etienne Chauchot wrote: > Welcome! > > > > Le 17/10/2017 à 22:18, Vilhelm von Ehrenheim a écrit : > >> Hi everyone! >> My name is Vilhelm von Ehrenheim and I would like to start contributing to >> Beam. >> I work as a Data Engineer at EQT in Stock

Re: 100 open pull requests

We had a previous discussion about closing stale PRs. This might be a good time to force close some of those, especially if the authors are not active. Ahmet On Thu, Oct 12, 2017 at 1:43 PM, Jean-Baptiste Onofré wrote: > Agree > > I will start to update/review some PRs. > > Regards > JB > > On

Re: [VOTE] Migrate to gitbox

+1 On Tue, Oct 10, 2017 at 9:35 AM, Thomas Groh wrote: > +1 > > On Tue, Oct 10, 2017 at 9:12 AM, Kenneth Knowles > wrote: > > > +1 > > > > On Tue, Oct 10, 2017 at 8:33 AM, Tyler Akidau > > > wrote: > > > > > +1 > > > > > > On Tue, Oct 10, 2017 at 2:13 AM Ismaël Mejía > wrote: > > > > > > > +1

Re: Problem while upgrading lib

google-apitools dependency (which is required for GCS) does not work with oauth2client >= 4.0.0 [1]. Because of this Beam Python SDK also does not work with oauth2client >= 4.0.0 versions, and this is captured correctly in the setup.py [2]. Ahmet [1] https://github.com/google/apitools/blob/7aff8d

<    4   5   6   7   8   9   10   >