Re: Pipeline AttributeError on Python3

d_fn) > > then I would try to trim the doFn in the example to a > minimally-reproducible example. It could be another issue with dill > dependency. > > > On Mon, Oct 28, 2019 at 2:48 PM Rakesh Kumar wrote: > >> Hi All, >> >> We have noticed a weird intermittent iss

Pipeline AttributeError on Python3

Hi All, We have noticed a weird intermittent issue on Python3 but we don't run into this issue on python2. Sometimes when we are trying to submit the pipeline, we get AttributeError (Check the stack trace below). we have double-checked and we do find the attribute/methods are present in the

Re: installing Apache Beam on Pycharm with Python 3.7

Hi Priti, It would be helpful if you can provide more information about your environment and the error message. You can also ask this question in stackoverflow with 'apache-beam' tag for better visibility. On Thu, Sep 5, 2019 at 9:43 AM Priti Badami < pbadami.srdataengin...@gmail.com> wrote: >

Re: Hackathon @BeamSummit @ApacheCon

Hi Austin, I am attending Beam Summit and also presenting on a use case of Beam. I would be more than happy to provide you some help in the hackathon. On Thu, Aug 22, 2019 at 9:48 AM Austin Bennett wrote: > Less than 3 weeks till Beam Summit @ApacheCon! > > We are to be in Vegas for

Re: [ANNOUNCE] New committer: Rui Wang

Congrats Rui!! On Wed, Aug 7, 2019 at 12:54 PM Rui Wang wrote: > Thank you guys! Looking forward to contributing more to Beam community! > > > -Rui > > On Wed, Aug 7, 2019 at 11:05 AM Tanay Tummalapalli > wrote: > >> Congratulations! >> >> On Wed, Aug 7, 2019 at 11:28 PM Robin Qiu wrote: >>

Re: [ANNOUNCE] New committer: Kyle Weaver

Congrats Kyle!! On Wed, Aug 7, 2019 at 11:30 AM Heejong Lee wrote: > Congratulations! > > On Wed, Aug 7, 2019 at 11:05 AM Tanay Tummalapalli > wrote: > >> Congratulations! >> >> On Wed, Aug 7, 2019 at 11:27 PM Robin Qiu wrote: >> >>> Congratulations, Kyle! >>> >>> On Wed, Aug 7, 2019 at 5:04

Re: [ANNOUNCE] New committer: Jan Lukavský

Congratulations Jan!! On Wed, Aug 7, 2019 at 5:05 AM Valentyn Tymofieiev wrote: > Congrats, Jan! > > On Wed, Aug 7, 2019 at 8:01 AM Connell O'Callaghan > wrote: > >> Well done Jan!!! >> >> On Tue, Aug 6, 2019 at 15:44 Chamikara Jayalath >> wrote: >> >>> Congrats Jan! >>> >>> On Thu, Aug 1,

Re: [DISCUSS] Reconciling ValueState in Java and Python (was: [docs] Python State & Timers)

Hi Brian/Robert I am moving ahead with implementing ReadModifyWriteState. I will just repurpose this PR: https://github.com/apache/beam/pull/9067 . On Mon, Jul 15, 2019 at 7:47 PM Rakesh Kumar wrote: > Brian, > > I just want to follow up. Let me know if you are working on this. &g

Re: Write-through-cache in State logic

e. > > On Wed, Jul 17, 2019 at 6:00 PM Rakesh Kumar wrote: > > > > I checked the python sdk[1] and it has similar implementation as Java > SDK. > > > > I would agree with Thomas. In case of high volume event stream and > bigger cluster size, network call can pote

Re: [DISCUSS] Reconciling SetState in Java and Python

Here is the PR (https://github.com/apache/beam/pull/9090) for the SetState. Feel free to review it. On Mon, Jul 15, 2019 at 11:01 PM Rakesh Kumar wrote: > Hi, > > I noticed that SetState is implemented in Java SDK but not implemented in > Python SDK. I have filed the jira tic

Re: Live fixing of a Beam bug on July 25 at 3:30pm-4:30pm PST

+1, I really appreciate this initiative. It would be really helpful newbies like me. Is it possible to list out what are the things that you are planning to cover? On Tue, Jul 16, 2019 at 11:19 AM Yichi Zhang wrote: > Thanks for organizing this Pablo, it'll be very helpful! > > On Tue, Jul

Re: Write-through-cache in State logic

, 2019 at 6:02 AM Reuven Lax wrote: >>> > >>> > It's runner dependent. Some runners (e.g. the Dataflow runner) do have >>> such a cache, though I think it's currently has a cap for large bags. >>> > >>> > Reuven >>> > >>

Re: [ANNOUNCE] New committer: Robert Burke

Congrats Rob!!! On Tue, Jul 16, 2019 at 10:24 AM Ahmet Altay wrote: > Hi, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Robert > Burke. > > Robert has been contributing to Beam and actively involved in the > community for over a year. He has been actively

[DISCUSS] Reconciling SetState in Java and Python

Hi, I noticed that SetState is implemented in Java SDK but not implemented in Python SDK. I have filed the jira ticket and I am thinking to implement in Python SDK. Let me know if anyone has any concerns. Also, feel free to pass the link of the

Write-through-cache in State logic

Hi, I have been using python sdk for the application and also using BagState in production. I was wondering whether state logic has any write-through-cache implemented or not. If we are sending every read and write request through network then it comes with a performance cost. We can avoid

Re: [DISCUSS] Reconciling ValueState in Java and Python (was: [docs] Python State & Timers)

Brian, I just want to follow up. Let me know if you are working on this. Otherwise, I can implement ReadModifyWriteState. On Mon, May 6, 2019 at 4:52 PM Reza Rokni wrote: > When used as metadata I think the ReadModifyWrite naming is very accurate > for the majority of cases. > > The only case

Re: ParDo Execution Time stat is always 0

Installing cython in the application environment fixed the issue. Now I am able to see the operator metrics ({organization_specific_prefix} .operator.beam-metric-pardo_execution_time-process_bundle_ msecs-v1.gauge.mean) Thanks Ankur for looking into it and providing support. I am going to close

Re: python precommits failing at head

Even I am running into the same issue. Though my test passed but somehow the task didn't terminate, eventually the task was aborted. I have already tried a couple of times to retrigger tye python precommit but it failed every

Re: Beam local website development environment

Thanks Alan, I will give it a try. On Tue, Jun 11, 2019 at 3:31 PM Alan Myrvold wrote: > You can serve (or test) the website locally, using > https://cwiki.apache.org/confluence/display/BEAM/Website+Tips > > On Tue, Jun 11, 2019 at 3:27 PM Rakesh Kumar wrote: > >>

Beam local website development environment

Hi, I have been updating the documentation section of Beam website, mostly the code examples [1], & [2]. I want to check these changes locally before merging it to master just to make sure that changes will look good on the website. What is the best way of testing this in local? [1]:

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Congratulations Pablo!!! On Mon, Jun 10, 2019 at 6:03 AM Gleb Kanterov wrote: > Congratulations! > > On Fri, May 24, 2019 at 9:50 PM Joana Filipa Bernardo Carrasqueira < > joanafil...@google.com> wrote: > >> Congratulations Pablo! Well deserved :D >> >> On Fri, May 17, 2019 at 3:14 PM Hannah

Re: Need help regarding memory leak issue

using the issue, or >> from beam sdk. just my 2 cents. >> >> >> On Thu, Nov 15, 2018 at 4:54 PM Rakesh Kumar >> wrote: >> >>> Thanks for responding Ruoyun, >>> >>> We are not sure yet who is causing the leak, but once we run out of t

Re: Need help regarding memory leak issue

ill try your suggested ideas and report it here. When Beam and Model methods are run in isolation they don't show increase in memory consumption so we feel that it is the intraction which is causing the issue. On Thu, Nov 15, 2018 at 4:54 PM Rakesh Kumar wrote: > >> Thanks for responding

Re: Python 3: final step

Hi Rob, I am, Rakesh Kumar, using Beam SDK for one of my projects at Lyft. I have been working closely with Thomas Weise. I have already met a couple of Python SDK developers in person. I am interested to help migrate to Python 3. You can assign me PRs for review. I am also more than happy

Re: Beam python sdk with gevent

Hi Ahmet, I filed the jira ticket https://issues.apache.org/jira/browse/BEAM-5497. Let me know if you need anything else from us. Thank you, Rakesh On Mon, Sep 24, 2018 at 5:08 PM Ahmet Altay wrote: > On Sun, Sep 23, 2018 at 9:21 PM, Rakesh Kumar > wrote: > >> Thanks Ahme

Re: Beam python sdk with gevent

708/sdks/python/apache_beam/internal/util.py#L117 > > Ahmet > > > On Wed, Sep 19, 2018 at 10:25 PM, Rakesh Kumar > wrote: > >> >> Gevent <http://www.gevent.org/> is basically used to make parallel >> network calls. We are using gevent in one of the transf

Re: Beam python sdk with gevent

t;>> gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch >>> File "src/gevent/__greenlet_primitives.pxd", line 35, in >>> gevent.__greenlet_primitives._greenlet_switch >>> greenlet.error: cannot switch to a different thread >>> >>> and do not make any progress. >>> >>> Has anybody else successfully used the portable python sdk with gevent? >>> Or is there a recommended alternative for doing async IO in python >>> pipelines? >>> >>> [0] https://github.com/grpc/grpc/issues/4629#issuecomment-376962677 >>> >>> Micah >>> >> > -- Rakesh Kumar Software Engineer 510-761-1364 | <https://www.lyft.com/>

Re: An example of Integration test case

Robin > > On Tue, Aug 28, 2018 at 1:10 PM Rakesh Kumar wrote: > >> Hi, >> >> I am writing my streaming application using Python SDK. I also want to >> write an integration test cases. Do we have any good example of integration >> test that I can refer? >>

An example of Integration test case

Hi, I am writing my streaming application using Python SDK. I also want to write an integration test cases. Do we have any good example of integration test that I can refer? Thank you, Rakesh -- Rakesh Kumar Software Engineer 510-761-1364 | <https://www.lyft.com/>

Re: Tensor Flow support on Flink Runner

preprocess_flink.sh > Note, There might be some rough edges as the PortableFlinkRunner is still > in development. > > Thanks, > Ankur > > > On Sat, Aug 25, 2018 at 12:58 PM Rakesh Kumar > wrote: > >> Hi all, >> >> I would like to know whether Tensor Flow is suppor

Tensor Flow support on Flink Runner

Hi all, I would like to know whether Tensor Flow is supported on Flink Runner & DirectRunner? If it is not currently supported then what is the plan to provide the support. Thank you Rakesh -- Rakesh Kumar Software Engineer 510-761-1364 | <https://www.lyft.com/>