Re: Beam python sdk with gevent

Thank you Rakesh! Let us know how it goes if you get a chance to try multiprocess. On Tue, Sep 25, 2018 at 9:19 AM, Rakesh Kumar wrote: > Hi Ahmet, > > I filed the jira ticket https://issues.apache.org/jira/browse/BEAM-5497. > Let me know if you need anything else from us. > > Thank you, > Rakes

Re: Beam python sdk with gevent

Hi Ahmet, I filed the jira ticket https://issues.apache.org/jira/browse/BEAM-5497. Let me know if you need anything else from us. Thank you, Rakesh On Mon, Sep 24, 2018 at 5:08 PM Ahmet Altay wrote: > On Sun, Sep 23, 2018 at 9:21 PM, Rakesh Kumar > wrote: > >> Thanks Ahmet for providing a ref

Re: Beam python sdk with gevent

On Sun, Sep 23, 2018 at 9:21 PM, Rakesh Kumar wrote: > Thanks Ahmet for providing a reference code. I will give it a try. > > I also tried to read the code it feels like you are using the multiprocess > for parallelizing runtime jobs. We wanted to use Gevent because it is > lightweight and good f

Re: Beam python sdk with gevent

Thanks Ahmet for providing a reference code. I will give it a try. I also tried to read the code it feels like you are using the multiprocess for parallelizing runtime jobs. We wanted to use Gevent because it is lightweight and good for parallelizing IO/Network bound jobs. I would also recommend p

Re: Beam python sdk with gevent

Thank you for the example it helps. I still do not know what is wrong with gevent. Would you consider using multiprocessing package? We are already using that to accomplish something similar in file based sinks, and there is already utility function that wraps it around similar to your example [1]

Re: Beam python sdk with gevent

Gevent is basically used to make parallel network calls. We are using gevent in one of the transformation methods to call internal services. The transformation method is making multiple network call in parallel. Here is the code snippet: /__init__.py import gevent.monkey ge

Re: Beam python sdk with gevent

I am also not familiar with gevent. Could you explain what are you trying to do and how do you plan to use gevent? On Tue, Sep 18, 2018 at 9:38 AM, Lukasz Cwik wrote: > I don't think anyone has tried what your doing. The code that your working > with is very new. > > On Mon, Sep 17, 2018 at 5:02

Re: Beam python sdk with gevent

I don't think anyone has tried what your doing. The code that your working with is very new. On Mon, Sep 17, 2018 at 5:02 PM Micah Wylde wrote: > Hi all, > > We're using the Python SDK with the portable Flink runner and running into > some problems integrating gevent. We're patching the gRPC run

Beam python sdk with gevent

Hi all, We're using the Python SDK with the portable Flink runner and running into some problems integrating gevent. We're patching the gRPC runtime for gevent as described in [0] which allows pipelines to start and partially run. However the tasks produce a stream of gevent exceptions: Exception