Probably the most stable is running on Dataflow still. But I’m excited
to see the progress towards a Spark runner, can’t wait to try TFT on it :)
On Tue, Sep 17, 2019 at 4:37 PM Kyle Weaver <[email protected]
<mailto:[email protected]>> wrote:
The Flink runner is definitely more stable, as it's been around for
longer and has more developers and users on it. But a lot of the
code is shared, so for example some of the issues above would also
happen on the Flink runner.
Kyle Weaver | Software Engineer | github.com/ibzib
<http://github.com/ibzib> | [email protected]
<mailto:[email protected]>
On Tue, Sep 17, 2019 at 4:34 PM Benjamin Tan
<[email protected] <mailto:[email protected]>>
wrote:
Thanks for all the replies Kyle! You've been super helpful :D.
Would you say that the Flink runner more stable than the Spark
one? Or which combo is the most stable for now?
On 2019/09/17 19:43:54, Tom Barber <[email protected]
<mailto:[email protected]>> wrote:
> Thanks Kyle,
>
> From my pov Alpha is fine, I’m just trying to test out some
of the
> capabilities currently, but trying to dig around the website
doesn’t
> explain a great deal. Luckily Benjamin seems a step ahead of
me… I hope it
> stays that way! ;)
>
>
> On 17 September 2019 at 19:33:40, Kyle Weaver
([email protected] <mailto:[email protected]>) wrote:
>
> > The amount of issues I've encountered as a newbie is indeed
troubling.
> Spark portability is very much "alpha" quality software, a
point we should
> maybe emphasize on the website more. Anyway, I appreciate
your patience,
> and I'll do my best to address all these issues.
>
> >
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException:
> UNIMPLEMENTED: Method not found!
> > AttributeError: module 'apache_beam.coders.coders' has no
> attribute ‘VarIntCoder'
> This class of errors occurs when the SDK version does not
match up with the
> runner version -- unfortunately, we cannot guarantee
compatibility between
> the two, so manual syncing is required for now. We are
looking for ways to
> improve this.
>
> > ERROR:grpc._server:Exception calling application: u'2-1'
> It looks like these errors are entirely spurious. Have a fix
for it here:
> https://github.com/apache/beam/pull/9600
>
> Note that there may still be other spurious errors like:
>
> 19/09/17 11:25:31 ERROR ManagedChannelOrphanWrapper: *~*~*~
Channel
> ManagedChannelImpl{logId=84, target=localhost:36129} was not
shutdown
> properly!!! ~*~*~*
>
> and
>
> 19/09/17 11:25:32 ERROR SerializingExecutor: Exception while
executing
> runnable
>
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@47db89b1
> java.lang.IllegalStateException: call already closed
>
> I will try to get rid of them also, but for now you can just
ignore them.
> They are annoying but harmless.
>
> Kyle Weaver | Software Engineer | github.com/ibzib
<http://github.com/ibzib> | [email protected]
<mailto:[email protected]>
>
>
> On Tue, Sep 17, 2019 at 8:05 AM Benjamin Tan
<[email protected] <mailto:[email protected]>>
> wrote:
>
> > :D. Still, I'm curious as to the error we both are getting.
Maybe someone
> > can shed some light on it.
> >
> > On Tue, Sep 17, 2019 at 10:54 PM Tom Barber
<[email protected] <mailto:[email protected]>> wrote:
> >
> >> I do see hello written to 1 file and world to another, I
guess it works!
> >> Thanks for the pointers Benjamin I was about to give up.
> >>
> >> Tom
> >>
> >>
> >> On 17 September 2019 at 15:51:13, Benjamin Tan (
> >> [email protected]
<mailto:[email protected]>) wrote:
> >>
> >> Tell me if you see any output. Anyway, here's the link to
the same issue
> >> you're facing:
> >>
> >>
> >>
https://lists.apache.org/thread.html/4e8e1455916debe096de32551f9ab05853524cf282bc312cd4620d68@%3Cuser.beam.apache.org%3E
> >>
> >> The amount of issues I've encountered as a newbie is
indeed troubling.
> >>
> >> On 2019/09/17 14:43:11, Tom Barber <[email protected]
<mailto:[email protected]>> wrote:
> >> > 🤣 okay I’ll look again, I assumed it just crashed in a
ball of flames!
> >> >
> >> >
> >> > On 17 September 2019 at 15:39:33, Benjamin Tan (
> >> [email protected]
<mailto:[email protected]>)
> >> > wrote:
> >> >
> >> > I got this too! Did you manage to get any output? (I
did) I reported
> >> this
> >> > in another thread.
> >> >
> >> > This looks like a key error when StopWorker is called.
As far as I
> >> know, it
> >> > seems like the work has been processed.
> >> >
> >> > On Tue, Sep 17, 2019 at 10:31 PM Tom Barber
<[email protected] <mailto:[email protected]>> wrote:
> >> >
> >> > > Well my errors are different but still terminal:
> >> > >
> >> > > ERROR:grpc._server:Exception calling application: u'1-1'
> >> > > Traceback (most recent call last):
> >> > > File
"/usr/local/lib/python2.7/dist-packages/grpc/_server.py", line
> >> 434,
> >> > > in _call_behavior
> >> > > response_or_iterator = behavior(argument, context)
> >> > > File
> >> > >
> >>
"/usr/local/lib/python2.7/dist-packages/apache_beam-2.17.0.dev0-py2.7.egg/apache_beam/runners/worker/worker_pool_main.py",
> >> > > line 126, in StopWorker
> >> > > worker_process =
> >> > > self._worker_processes.pop(stop_worker_request.worker_id)
> >> > > KeyError: u'1-1'
> >> > > ERROR:grpc._server:Exception calling application: u'2-1'
> >> > > Traceback (most recent call last):
> >> > > File
"/usr/local/lib/python2.7/dist-packages/grpc/_server.py", line
> >> 434,
> >> > > in _call_behavior
> >> > > response_or_iterator = behavior(argument, context)
> >> > > File
> >> > >
> >>
"/usr/local/lib/python2.7/dist-packages/apache_beam-2.17.0.dev0-py2.7.egg/apache_beam/runners/worker/worker_pool_main.py",
> >> > > line 126, in StopWorker
> >> > > worker_process =
> >> > > self._worker_processes.pop(stop_worker_request.worker_id)
> >> > > KeyError: u'2-1'
> >> > >
> >> > >
> >> > > On 17 September 2019 at 14:46:12, Benjamin Tan (
> >> > > [email protected]
<mailto:[email protected]>) wrote:
> >> > >
> >> > > If it helps, I’m using Spark 2.4.4. The Apache Beam
Python library on
> >> > > master is 2.17.0-dev.
> >> > >
> >> > >
> >> > > On 17 Sep 2019, at 9:39 PM, Tom Barber
<[email protected] <mailto:[email protected]>> wrote:
> >> > >
> >> > > Cool thanks Benjamin, I’ll give it a shot.
> >> > >
> >> > > Tom
> >> > >
> >> > >
> >> > > On 17 September 2019 at 13:56:14, Benjamin Tan (
> >> > > [email protected]
<mailto:[email protected]>) wrote:
> >> > >
> >> > >
> >> > > I encountered the exact same thing today. High five!
Here’s how I
> >> managed
> >> > > to make some progress:
> >> > >
> >> > > 1. Used the master branch
> >> > > 2. Built and installed the Python SDK
> >> > >
> >> > > cd into the sdk library and python ./setup install
> >> > >
> >> > > I got some other errors but they didn’t seem to be
show stoppers.
> >> > >
> >> > > On 17 Sep 2019, at 8:26 PM, Tom Barber
<[email protected] <mailto:[email protected]>> wrote:
> >> > >
> >> > > Hello folks,
> >> > >
> >> > > Day 3 of trying to get the basics going with Python &
Spark 2.2.3.
> >> > >
> >> > > I’ve downgraded the spark version to 2.2.3 in the
cradle build so
> >> that I
> >> > > can run jobs against it.
> >> > >
> >> > > I’ve then written this:
> >> > >
> >> > > options = PipelineOptions(["--runner=PortableRunner",
> >> > > "--job_endpoint=localhost:8099",
"--environment_type=LOOPBACK"])
> >> > >
> >> > > with beam.Pipeline(options=options) as p:
> >> > > print("hello")
> >> > > lines = p | 'Create' >> beam.Create(['hello', 'world'])
> >> > >
> >> > > Which in turn explodes with:
> >> > >
> >> > > Caused by:
> >> > >
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException:
> >> > > UNIMPLEMENTED: Method not found!
> >> > > at
> >> > >
> >>
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235)
> >> > > at
> >> > >
> >>
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216)
> >> > > at
> >> > >
> >>
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141)
> >> > > at
> >> > >
> >>
org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.startWorker(BeamFnExternalWorkerPoolGrpc.java:226)
> >> > >
> >> > > When run against python 2 and
> >> > >
> >> > > AttributeError: module 'apache_beam.coders.coders' has no
> >> > > attribute ‘VarIntCoder'
> >> > >
> >> > > When run with python 3.5.
> >> > >
> >> > > Should I just give up trying the portable runner at
the moment or am I
> >> > > doing something dumb/incompatible? I’m stuck on Spark
2.2.3 so
> >> upgrading
> >> > > isn’t currently an option.
> >> > >
> >> > > Tom
> >> > >
> >> > > Spicule Limited is registered in England & Wales.
Company Number:
> >> > > 09954122. Registered office: First Floor, Telecom
House, 125-135
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
>> Preston
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
>> > > Road, Brighton, England, BN1 6AF
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>.
VAT No. 251478891.
> >> > >
> >> > >
> >> > > All engagements are subject to Spicule Terms and
Conditions of
> >> Business.
> >> > > This email and its contents are intended solely for
the individual to
> >> whom
> >> > > it is addressed and may contain information that is
confidential,
> >> > > privileged or otherwise protected from disclosure,
distributing or
> >> copying.
> >> > > Any views or opinions presented in this email are
solely those of the
> >> > > author and do not necessarily represent those of
Spicule Limited. The
> >> > > company accepts no liability for any damage caused by
any virus
> >> transmitted
> >> > > by this email. If you have received this message in
error, please
> >> notify us
> >> > > immediately by reply email before deleting it from
your system.
> >> Service of
> >> > > legal notice cannot be effected on Spicule Limited by
email.
> >> > >
> >> > >
> >> > > Spicule Limited is registered in England & Wales.
Company Number:
> >> > > 09954122. Registered office: First Floor, Telecom
House, 125-135
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
>> Preston
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
>> > > Road, Brighton, England, BN1 6AF
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>.
VAT No. 251478891.
> >> > >
> >> > >
> >> > > All engagements are subject to Spicule Terms and
Conditions of
> >> Business.
> >> > > This email and its contents are intended solely for
the individual to
> >> whom
> >> > > it is addressed and may contain information that is
confidential,
> >> > > privileged or otherwise protected from disclosure,
distributing or
> >> copying.
> >> > > Any views or opinions presented in this email are
solely those of the
> >> > > author and do not necessarily represent those of
Spicule Limited. The
> >> > > company accepts no liability for any damage caused by
any virus
> >> transmitted
> >> > > by this email. If you have received this message in
error, please
> >> notify us
> >> > > immediately by reply email before deleting it from
your system.
> >> Service of
> >> > > legal notice cannot be effected on Spicule Limited by
email.
> >> > >
> >> > >
> >> > > Spicule Limited is registered in England & Wales.
Company Number:
> >> > > 09954122. Registered office: First Floor, Telecom
House, 125-135
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
>> Preston
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
>> > > Road, Brighton, England, BN1 6AF
<https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>.
VAT No. 251478891.
> >> > >
> >> > >
> >> > > All engagements are subject to Spicule Terms and
Conditions of
> >> Business.
> >> > > This email and its contents are intended solely for
the individual to
> >> whom
> >> > > it is addressed and may contain information that is
confidential,
> >> > > privileged or otherwise protected from disclosure,
distributing or
> >> copying.
> >> > > Any views or opinions presented in this email are
solely those of the
> >> > > author and do not necessarily represent those of
Spicule Limited. The
> >> > > company accepts no liability for any damage caused by
any virus
> >> transmitted
> >> > > by this email. If you have received this message in
error, please
> >> notify us
> >> > > immediately by reply email before deleting it from
your system.
> >> Service of
> >> > > legal notice cannot be effected on Spicule Limited by
email.
> >> > >
> >> >
> >> > --
> >> >
> >> >
> >> > Spicule Limited is registered in England & Wales.
Company Number:
> >> > 09954122. Registered office: First Floor, Telecom House,
125-135 Preston
<https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
>> > Road, Brighton, England, BN1 6AF
<https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>.
VAT No. 251478891.
> >> >
> >> >
> >> >
> >> >
> >> > All engagements
> >> > are subject to Spicule Terms and Conditions of Business.
This email and
> >> its
> >> > contents are intended solely for the individual to whom
it is addressed
> >> and
> >> > may contain information that is confidential, privileged
or otherwise
> >> > protected from disclosure, distributing or copying. Any
views or
> >> opinions
> >> > presented in this email are solely those of the author
and do not
> >> > necessarily represent those of Spicule Limited. The
company accepts no
> >> > liability for any damage caused by any virus transmitted
by this email.
> >> If
> >> > you have received this message in error, please notify
us immediately by
> >> > reply email before deleting it from your system. Service
of legal notice
> >> > cannot be effected on Spicule Limited by email.
> >> >
> >>
> >>
> >> Spicule Limited is registered in England & Wales. Company
Number:
> >> 09954122. Registered office: First Floor, Telecom House,
125-135 Preston
<https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
>> Road, Brighton, England, BN1 6AF
<https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>.
VAT No. 251478891.
> >>
> >>
> >> All engagements are subject to Spicule Terms and
Conditions of Business.
> >> This email and its contents are intended solely for the
individual to whom
> >> it is addressed and may contain information that is
confidential,
> >> privileged or otherwise protected from disclosure,
distributing or copying.
> >> Any views or opinions presented in this email are solely
those of the
> >> author and do not necessarily represent those of Spicule
Limited. The
> >> company accepts no liability for any damage caused by any
virus transmitted
> >> by this email. If you have received this message in error,
please notify us
> >> immediately by reply email before deleting it from your
system. Service of
> >> legal notice cannot be effected on Spicule Limited by email.
> >>
> >
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company
Number:
> 09954122. Registered office: First Floor, Telecom House,
125-135 Preston
<https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>
<https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>>
Road, Brighton, England, BN1 6AF
<https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>.
VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This
email and its
> contents are intended solely for the individual to whom it is
addressed and
> may contain information that is confidential, privileged or
otherwise
> protected from disclosure, distributing or copying. Any views
or opinions
> presented in this email are solely those of the author and do
not
> necessarily represent those of Spicule Limited. The company
accepts no
> liability for any damage caused by any virus transmitted by
this email. If
> you have received this message in error, please notify us
immediately by
> reply email before deleting it from your system. Service of
legal notice
> cannot be effected on Spicule Limited by email.
>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
