If it helps, I’m using Spark 2.4.4. The Apache Beam Python library on master is 2.17.0-dev.
> On 17 Sep 2019, at 9:39 PM, Tom Barber <[email protected]> wrote: > > Cool thanks Benjamin, I’ll give it a shot. > > Tom > > >> On 17 September 2019 at 13:56:14, Benjamin Tan ([email protected]) >> wrote: >> >> >> I encountered the exact same thing today. High five! Here’s how I managed to >> make some progress: >> >> 1. Used the master branch >> 2. Built and installed the Python SDK >> >> cd into the sdk library and python ./setup install >> >> I got some other errors but they didn’t seem to be show stoppers. >> >> On 17 Sep 2019, at 8:26 PM, Tom Barber <[email protected]> wrote: >> >>> Hello folks, >>> >>> Day 3 of trying to get the basics going with Python & Spark 2.2.3. >>> >>> I’ve downgraded the spark version to 2.2.3 in the cradle build so that I >>> can run jobs against it. >>> >>> I’ve then written this: >>> >>> options = PipelineOptions(["--runner=PortableRunner", >>> "--job_endpoint=localhost:8099", "--environment_type=LOOPBACK"]) >>> >>> with beam.Pipeline(options=options) as p: >>> print("hello") >>> lines = p | 'Create' >> beam.Create(['hello', 'world']) >>> >>> Which in turn explodes with: >>> >>> Caused by: >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: >>> UNIMPLEMENTED: Method not found! >>> at >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235) >>> at >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216) >>> at >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141) >>> at >>> org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.startWorker(BeamFnExternalWorkerPoolGrpc.java:226) >>> >>> When run against python 2 and >>> >>> AttributeError: module 'apache_beam.coders.coders' has no attribute >>> ‘VarIntCoder' >>> >>> When run with python 3.5. >>> >>> Should I just give up trying the portable runner at the moment or am I >>> doing something dumb/incompatible? I’m stuck on Spark 2.2.3 so upgrading >>> isn’t currently an option. >>> >>> Tom >>> >>> Spicule Limited is registered in England & Wales. Company Number: 09954122. >>> Registered office: First Floor, Telecom House, 125-135 Preston Road, >>> Brighton, England, BN1 6AF. VAT No. 251478891. >>> >>> >>> >>> All engagements are subject to Spicule Terms and Conditions of Business. >>> This email and its contents are intended solely for the individual to whom >>> it is addressed and may contain information that is confidential, >>> privileged or otherwise protected from disclosure, distributing or copying. >>> Any views or opinions presented in this email are solely those of the >>> author and do not necessarily represent those of Spicule Limited. The >>> company accepts no liability for any damage caused by any virus transmitted >>> by this email. If you have received this message in error, please notify us >>> immediately by reply email before deleting it from your system. Service of >>> legal notice cannot be effected on Spicule Limited by email. >>> > > Spicule Limited is registered in England & Wales. Company Number: 09954122. > Registered office: First Floor, Telecom House, 125-135 Preston Road, > Brighton, England, BN1 6AF. VAT No. 251478891. > > > > All engagements are subject to Spicule Terms and Conditions of Business. This > email and its contents are intended solely for the individual to whom it is > addressed and may contain information that is confidential, privileged or > otherwise protected from disclosure, distributing or copying. Any views or > opinions presented in this email are solely those of the author and do not > necessarily represent those of Spicule Limited. The company accepts no > liability for any damage caused by any virus transmitted by this email. If > you have received this message in error, please notify us immediately by > reply email before deleting it from your system. Service of legal notice > cannot be effected on Spicule Limited by email.
