Re: Beam [python] - docker run in ParDo?

ications such as Java's ProcessBuilder[1]. > > 1: https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html > > On Tue, Jan 4, 2022 at 8:13 PM Mahesh Vangala > wrote: > >> Hello Beam community - >> >> Is it possible to run docker image as part of a

Beam [python] - docker run in ParDo?

Hello Beam community - Is it possible to run docker image as part of a transformation step? Could you point me to any examples with using "docker run" as a step in the pipeline? Let me know. Thank you, Mahesh

Re: Beam python pipeline on spark

Super 👍 Thank you for the great news! I will file issues in this thread accordingly. Best, Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Fri, Aug 2, 2019 at 2:51 PM Kyle Weaver wrote: > Hi Manesh, > > I'm happy to say

Beam python pipeline on spark

ks, Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>*

Re: writing a single record to Kafka ...

PCollection p = Create.of(li); // But, this creates PValues. Any help would be appreciated. Thank you. *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Tue, Sep 11, 2018 at 6:26 PM Mahesh Vangala wrote: > Thanks, Lukasz. > Appreciate

Re: writing a single record to Kafka ...

Thanks, Lukasz. Appreciate your advice. *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Tue, Sep 11, 2018 at 6:19 PM Lukasz Cwik wrote: > A PCollection is a bag of elements. PCollections can be empty, have only > one element or have many.

writing a single record to Kafka ...

Hello - I'd like to write a single record to kafka topic through beam. However, I only see examples that work with PCollection. Any thoughts about how I can approach to this? Thank you. Regards, Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>*

Help in reading hadoop HDFS file into PCollection

local file in yarn is causing issues. I did look into beam api beam.sdk.io.hadoop and spark, but no luck in finding right info. If you could nudge me in the right direction, that'd be great! Thank you for your help. Regards, Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com

spark runner job/pipeline monitoring via beam

REST server but wanted to get your thoughts before experimenting further. Let me know. Thank you. Regards, Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>*

Re: Beam Summit London 2018

Hi Pascal - Just curious to know that in your message you mentioned about beam python work with respect to spark/hadoop. I was under the impression that python beam is not yet available to run on spark/hadoop. Am I missing something? Let me know. Thanks. *--* *Mahesh Vangala* *(Ph) 443-326-1957

Re: Launching a subprocess in DoFn

. Regards, Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Mon, Aug 20, 2018 at 11:18 AM Mahesh Vangala wrote: > Hi Romain - > > Got it. Apparently, I needed to run beam as sudo to launch the docker > containers. > I still n

Re: Launching a subprocess in DoFn

Hi Romain - Got it. Apparently, I needed to run beam as sudo to launch the docker containers. I still need to figure out why that's the case. Thank you. *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Mon, Aug 20, 2018 at 11:05 AM Mahesh Van

Re: Launching a subprocess in DoFn

help. *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Mon, Aug 20, 2018 at 10:58 AM Romain Manni-Bucau wrote: > Weird, this code works: > > https://gist.github.com/rmannibucau/4703f321bb1962d1303f8eccbd05df0e > > Are you sure your

Re: Launching a subprocess in DoFn

a security manager active I am using DirectRunner though. Let me know. Thank you! *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Mon, Aug 20, 2018 at 10:28 AM Romain Manni-Bucau wrote: > Hi Mahesh, > > Did you get the same error? This is

Re: Launching a subprocess in DoFn

-* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Sun, Aug 19, 2018 at 3:53 AM Romain Manni-Bucau wrote: > waitFor and not java wait primitive? > > Le dim. 19 août 2018 04:35, Mahesh Vangala a > écrit : > >> Hello Beamers - >> &

Launching a subprocess in DoFn

ct.java:502) at pipelines.variant_caller.LaunchDocker$LaunchJobs.processElement( LaunchDocker.java:19) Can you share your ideas what's the best way of achieving this? Thank you for your help! Sincerely, Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>*

Re: coder issue?

o you think, getCoder implementation is not necessary? Thanks for your help though. Much appreciate! - Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Fri, Aug 17, 2018 at 1:01 PM Robin Qiu wrote: > Hi Mahesh, > >

Re: coder issue?

cumulators(AddLines.java:35) at pipelines.variant_caller.AddLines.mergeAccumulators(AddLines.java:1) *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Thu, Aug 16, 2018 at 2:41 PM Robin Qiu wrote: > Hello Mahesh, > > You can add "implements Se

coder issue?

.sdk.values.PCollection.apply(PCollection.java:325) at pipelines.variant_caller.VariantCaller.main(VariantCaller.java:27) *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>*

Re: Running beam script using SparkRunner

Please ignore my question :) This URL did it! https://beam.apache.org/documentation/runners/spark/ Thank you. *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>* On Mon, Aug 13, 2018 at 2:31 PM Mahesh Vangala wrote: > Hello all - > > I h

Running beam script using SparkRunner

eam code run on spark. (Sorry, there seems to be not so great documentation for this use case, or perhaps, I overlooked?) Thank you for your help. *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com <http://mvangala.com>*

Re: Beam Java newbie.

Hello - Please ignore my previous question. Apparently, including beam core and direct runner in pom.xml did fix the issue I was facing. Thank you. Regards, Mahesh On Sat, Aug 11, 2018 at 10:56 PM Mahesh Vangala wrote: > Hello guys - > > Is there a doc that describes how to insta

Re: Beam Java newbie.

Hello guys - Is there a doc that describes how to install apache-beam java sdk on mac, so that I can try out direct runner pipeline? Let me know. Thank you, Mahesh On Sat, Aug 11, 2018 at 6:51 PM Mahesh Vangala wrote: > Hi Pablo - > > Thanks for the quick response. > Here&

Re: Beam Java newbie.

9 PM Pablo Estrada wrote: > Can you elaborate on your setup? Are you using a Maven archetype? Did you > hand-write a pom.xml? Are you installing from code with gradle? > Best > -P. > > On Sat, Aug 11, 2018, 3:43 PM Mahesh Vangala > wrote: > >> Hello folks - >>

Beam Java newbie.

Hello folks - I am enthusiastic about learning beam using java sdk. I set up maven using eclipse today. However, I ran into the following error message: Caused by: java.lang.ClassNotFoundException: org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ProtocolMessageEnum at java.net.URLClassLoa