Probably the most stable is running on Dataflow still. But I’m excited to see the progress towards a Spark runner, can’t wait to try TFT on it :)
On Tue, Sep 17, 2019 at 4:37 PM Kyle Weaver <[email protected]> wrote: > The Flink runner is definitely more stable, as it's been around for longer > and has more developers and users on it. But a lot of the code is shared, > so for example some of the issues above would also happen on the Flink > runner. > > Kyle Weaver | Software Engineer | github.com/ibzib | [email protected] > > > On Tue, Sep 17, 2019 at 4:34 PM Benjamin Tan <[email protected]> > wrote: > >> Thanks for all the replies Kyle! You've been super helpful :D. >> >> Would you say that the Flink runner more stable than the Spark one? Or >> which combo is the most stable for now? >> >> On 2019/09/17 19:43:54, Tom Barber <[email protected]> wrote: >> > Thanks Kyle, >> > >> > From my pov Alpha is fine, I’m just trying to test out some of the >> > capabilities currently, but trying to dig around the website doesn’t >> > explain a great deal. Luckily Benjamin seems a step ahead of me… I hope >> it >> > stays that way! ;) >> > >> > >> > On 17 September 2019 at 19:33:40, Kyle Weaver ([email protected]) >> wrote: >> > >> > > The amount of issues I've encountered as a newbie is indeed troubling. >> > Spark portability is very much "alpha" quality software, a point we >> should >> > maybe emphasize on the website more. Anyway, I appreciate your patience, >> > and I'll do my best to address all these issues. >> > >> > > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: >> > UNIMPLEMENTED: Method not found! >> > > AttributeError: module 'apache_beam.coders.coders' has no >> > attribute ‘VarIntCoder' >> > This class of errors occurs when the SDK version does not match up with >> the >> > runner version -- unfortunately, we cannot guarantee compatibility >> between >> > the two, so manual syncing is required for now. We are looking for ways >> to >> > improve this. >> > >> > > ERROR:grpc._server:Exception calling application: u'2-1' >> > It looks like these errors are entirely spurious. Have a fix for it >> here: >> > https://github.com/apache/beam/pull/9600 >> > >> > Note that there may still be other spurious errors like: >> > >> > 19/09/17 11:25:31 ERROR ManagedChannelOrphanWrapper: *~*~*~ Channel >> > ManagedChannelImpl{logId=84, target=localhost:36129} was not shutdown >> > properly!!! ~*~*~* >> > >> > and >> > >> > 19/09/17 11:25:32 ERROR SerializingExecutor: Exception while executing >> > runnable >> > >> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@47db89b1 >> > java.lang.IllegalStateException: call already closed >> > >> > I will try to get rid of them also, but for now you can just ignore >> them. >> > They are annoying but harmless. >> > >> > Kyle Weaver | Software Engineer | github.com/ibzib | >> [email protected] >> > >> > >> > On Tue, Sep 17, 2019 at 8:05 AM Benjamin Tan < >> [email protected]> >> > wrote: >> > >> > > :D. Still, I'm curious as to the error we both are getting. Maybe >> someone >> > > can shed some light on it. >> > > >> > > On Tue, Sep 17, 2019 at 10:54 PM Tom Barber <[email protected]> >> wrote: >> > > >> > >> I do see hello written to 1 file and world to another, I guess it >> works! >> > >> Thanks for the pointers Benjamin I was about to give up. >> > >> >> > >> Tom >> > >> >> > >> >> > >> On 17 September 2019 at 15:51:13, Benjamin Tan ( >> > >> [email protected]) wrote: >> > >> >> > >> Tell me if you see any output. Anyway, here's the link to the same >> issue >> > >> you're facing: >> > >> >> > >> >> > >> >> https://lists.apache.org/thread.html/4e8e1455916debe096de32551f9ab05853524cf282bc312cd4620d68@%3Cuser.beam.apache.org%3E >> > >> >> > >> The amount of issues I've encountered as a newbie is indeed >> troubling. >> > >> >> > >> On 2019/09/17 14:43:11, Tom Barber <[email protected]> wrote: >> > >> > 🤣 okay I’ll look again, I assumed it just crashed in a ball of >> flames! >> > >> > >> > >> > >> > >> > On 17 September 2019 at 15:39:33, Benjamin Tan ( >> > >> [email protected]) >> > >> > wrote: >> > >> > >> > >> > I got this too! Did you manage to get any output? (I did) I >> reported >> > >> this >> > >> > in another thread. >> > >> > >> > >> > This looks like a key error when StopWorker is called. As far as I >> > >> know, it >> > >> > seems like the work has been processed. >> > >> > >> > >> > On Tue, Sep 17, 2019 at 10:31 PM Tom Barber <[email protected]> >> wrote: >> > >> > >> > >> > > Well my errors are different but still terminal: >> > >> > > >> > >> > > ERROR:grpc._server:Exception calling application: u'1-1' >> > >> > > Traceback (most recent call last): >> > >> > > File "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", >> line >> > >> 434, >> > >> > > in _call_behavior >> > >> > > response_or_iterator = behavior(argument, context) >> > >> > > File >> > >> > > >> > >> >> "/usr/local/lib/python2.7/dist-packages/apache_beam-2.17.0.dev0-py2.7.egg/apache_beam/runners/worker/worker_pool_main.py", >> > >> > > line 126, in StopWorker >> > >> > > worker_process = >> > >> > > self._worker_processes.pop(stop_worker_request.worker_id) >> > >> > > KeyError: u'1-1' >> > >> > > ERROR:grpc._server:Exception calling application: u'2-1' >> > >> > > Traceback (most recent call last): >> > >> > > File "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", >> line >> > >> 434, >> > >> > > in _call_behavior >> > >> > > response_or_iterator = behavior(argument, context) >> > >> > > File >> > >> > > >> > >> >> "/usr/local/lib/python2.7/dist-packages/apache_beam-2.17.0.dev0-py2.7.egg/apache_beam/runners/worker/worker_pool_main.py", >> > >> > > line 126, in StopWorker >> > >> > > worker_process = >> > >> > > self._worker_processes.pop(stop_worker_request.worker_id) >> > >> > > KeyError: u'2-1' >> > >> > > >> > >> > > >> > >> > > On 17 September 2019 at 14:46:12, Benjamin Tan ( >> > >> > > [email protected]) wrote: >> > >> > > >> > >> > > If it helps, I’m using Spark 2.4.4. The Apache Beam Python >> library on >> > >> > > master is 2.17.0-dev. >> > >> > > >> > >> > > >> > >> > > On 17 Sep 2019, at 9:39 PM, Tom Barber <[email protected]> >> wrote: >> > >> > > >> > >> > > Cool thanks Benjamin, I’ll give it a shot. >> > >> > > >> > >> > > Tom >> > >> > > >> > >> > > >> > >> > > On 17 September 2019 at 13:56:14, Benjamin Tan ( >> > >> > > [email protected]) wrote: >> > >> > > >> > >> > > >> > >> > > I encountered the exact same thing today. High five! Here’s how I >> > >> managed >> > >> > > to make some progress: >> > >> > > >> > >> > > 1. Used the master branch >> > >> > > 2. Built and installed the Python SDK >> > >> > > >> > >> > > cd into the sdk library and python ./setup install >> > >> > > >> > >> > > I got some other errors but they didn’t seem to be show stoppers. >> > >> > > >> > >> > > On 17 Sep 2019, at 8:26 PM, Tom Barber <[email protected]> >> wrote: >> > >> > > >> > >> > > Hello folks, >> > >> > > >> > >> > > Day 3 of trying to get the basics going with Python & Spark >> 2.2.3. >> > >> > > >> > >> > > I’ve downgraded the spark version to 2.2.3 in the cradle build so >> > >> that I >> > >> > > can run jobs against it. >> > >> > > >> > >> > > I’ve then written this: >> > >> > > >> > >> > > options = PipelineOptions(["--runner=PortableRunner", >> > >> > > "--job_endpoint=localhost:8099", "--environment_type=LOOPBACK"]) >> > >> > > >> > >> > > with beam.Pipeline(options=options) as p: >> > >> > > print("hello") >> > >> > > lines = p | 'Create' >> beam.Create(['hello', 'world']) >> > >> > > >> > >> > > Which in turn explodes with: >> > >> > > >> > >> > > Caused by: >> > >> > > >> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: >> > >> > > UNIMPLEMENTED: Method not found! >> > >> > > at >> > >> > > >> > >> >> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235) >> > >> > > at >> > >> > > >> > >> >> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216) >> > >> > > at >> > >> > > >> > >> >> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141) >> > >> > > at >> > >> > > >> > >> >> org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.startWorker(BeamFnExternalWorkerPoolGrpc.java:226) >> > >> > > >> > >> > > When run against python 2 and >> > >> > > >> > >> > > AttributeError: module 'apache_beam.coders.coders' has no >> > >> > > attribute ‘VarIntCoder' >> > >> > > >> > >> > > When run with python 3.5. >> > >> > > >> > >> > > Should I just give up trying the portable runner at the moment >> or am I >> > >> > > doing something dumb/incompatible? I’m stuck on Spark 2.2.3 so >> > >> upgrading >> > >> > > isn’t currently an option. >> > >> > > >> > >> > > Tom >> > >> > > >> > >> > > Spicule Limited is registered in England & Wales. Company Number: >> > >> > > 09954122. Registered office: First Floor, Telecom House, 125-135 >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> >> Preston >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> >> > > Road, Brighton, England, BN1 6AF >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >> VAT No. 251478891. >> > >> > > >> > >> > > >> > >> > > All engagements are subject to Spicule Terms and Conditions of >> > >> Business. >> > >> > > This email and its contents are intended solely for the >> individual to >> > >> whom >> > >> > > it is addressed and may contain information that is confidential, >> > >> > > privileged or otherwise protected from disclosure, distributing >> or >> > >> copying. >> > >> > > Any views or opinions presented in this email are solely those >> of the >> > >> > > author and do not necessarily represent those of Spicule >> Limited. The >> > >> > > company accepts no liability for any damage caused by any virus >> > >> transmitted >> > >> > > by this email. If you have received this message in error, please >> > >> notify us >> > >> > > immediately by reply email before deleting it from your system. >> > >> Service of >> > >> > > legal notice cannot be effected on Spicule Limited by email. >> > >> > > >> > >> > > >> > >> > > Spicule Limited is registered in England & Wales. Company Number: >> > >> > > 09954122. Registered office: First Floor, Telecom House, 125-135 >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> >> Preston >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> >> > > Road, Brighton, England, BN1 6AF >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >> VAT No. 251478891. >> > >> > > >> > >> > > >> > >> > > All engagements are subject to Spicule Terms and Conditions of >> > >> Business. >> > >> > > This email and its contents are intended solely for the >> individual to >> > >> whom >> > >> > > it is addressed and may contain information that is confidential, >> > >> > > privileged or otherwise protected from disclosure, distributing >> or >> > >> copying. >> > >> > > Any views or opinions presented in this email are solely those >> of the >> > >> > > author and do not necessarily represent those of Spicule >> Limited. The >> > >> > > company accepts no liability for any damage caused by any virus >> > >> transmitted >> > >> > > by this email. If you have received this message in error, please >> > >> notify us >> > >> > > immediately by reply email before deleting it from your system. >> > >> Service of >> > >> > > legal notice cannot be effected on Spicule Limited by email. >> > >> > > >> > >> > > >> > >> > > Spicule Limited is registered in England & Wales. Company Number: >> > >> > > 09954122. Registered office: First Floor, Telecom House, 125-135 >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> >> Preston >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> >> > > Road, Brighton, England, BN1 6AF >> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >> VAT No. 251478891. >> > >> > > >> > >> > > >> > >> > > All engagements are subject to Spicule Terms and Conditions of >> > >> Business. >> > >> > > This email and its contents are intended solely for the >> individual to >> > >> whom >> > >> > > it is addressed and may contain information that is confidential, >> > >> > > privileged or otherwise protected from disclosure, distributing >> or >> > >> copying. >> > >> > > Any views or opinions presented in this email are solely those >> of the >> > >> > > author and do not necessarily represent those of Spicule >> Limited. The >> > >> > > company accepts no liability for any damage caused by any virus >> > >> transmitted >> > >> > > by this email. If you have received this message in error, please >> > >> notify us >> > >> > > immediately by reply email before deleting it from your system. >> > >> Service of >> > >> > > legal notice cannot be effected on Spicule Limited by email. >> > >> > > >> > >> > >> > >> > -- >> > >> > >> > >> > >> > >> > Spicule Limited is registered in England & Wales. Company Number: >> > >> > 09954122. Registered office: First Floor, Telecom House, 125-135 >> Preston >> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> >> > Road, Brighton, England, BN1 6AF >> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >> VAT No. 251478891. >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > All engagements >> > >> > are subject to Spicule Terms and Conditions of Business. This >> email and >> > >> its >> > >> > contents are intended solely for the individual to whom it is >> addressed >> > >> and >> > >> > may contain information that is confidential, privileged or >> otherwise >> > >> > protected from disclosure, distributing or copying. Any views or >> > >> opinions >> > >> > presented in this email are solely those of the author and do not >> > >> > necessarily represent those of Spicule Limited. The company >> accepts no >> > >> > liability for any damage caused by any virus transmitted by this >> email. >> > >> If >> > >> > you have received this message in error, please notify us >> immediately by >> > >> > reply email before deleting it from your system. Service of legal >> notice >> > >> > cannot be effected on Spicule Limited by email. >> > >> > >> > >> >> > >> >> > >> Spicule Limited is registered in England & Wales. Company Number: >> > >> 09954122. Registered office: First Floor, Telecom House, 125-135 >> Preston >> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> >> Road, Brighton, England, BN1 6AF >> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >> VAT No. 251478891. >> > >> >> > >> >> > >> All engagements are subject to Spicule Terms and Conditions of >> Business. >> > >> This email and its contents are intended solely for the individual >> to whom >> > >> it is addressed and may contain information that is confidential, >> > >> privileged or otherwise protected from disclosure, distributing or >> copying. >> > >> Any views or opinions presented in this email are solely those of the >> > >> author and do not necessarily represent those of Spicule Limited. The >> > >> company accepts no liability for any damage caused by any virus >> transmitted >> > >> by this email. If you have received this message in error, please >> notify us >> > >> immediately by reply email before deleting it from your system. >> Service of >> > >> legal notice cannot be effected on Spicule Limited by email. >> > >> >> > > >> > >> > -- >> > >> > >> > Spicule Limited is registered in England & Wales. Company Number: >> > 09954122. Registered office: First Floor, Telecom House, 125-135 >> Preston >> <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >> >> <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >> Road, Brighton, England, BN1 6AF >> <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >> VAT No. 251478891. >> > >> > >> > >> > >> > All engagements >> > are subject to Spicule Terms and Conditions of Business. This email and >> its >> > contents are intended solely for the individual to whom it is addressed >> and >> > may contain information that is confidential, privileged or otherwise >> > protected from disclosure, distributing or copying. Any views or >> opinions >> > presented in this email are solely those of the author and do not >> > necessarily represent those of Spicule Limited. The company accepts no >> > liability for any damage caused by any virus transmitted by this email. >> If >> > you have received this message in error, please notify us immediately >> by >> > reply email before deleting it from your system. Service of legal >> notice >> > cannot be effected on Spicule Limited by email. >> > >> > -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
