Running Python Wordcount issues
Hello folks, Trying to get started running the python word count example against the portable runner using the beam 2.15 download. In on terminal I have: gradle :runners:spark:job-server:runShadow And another: gradle portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK But when it starts it says: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", line 135, in run() File "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", line 115, in run result.wait_until_finish() File "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py", line 446, in wait_until_finish self._job_id, self._state, self._last_error_message())) RuntimeError: Pipeline BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546 failed in state FAILED: java.lang.ClassCastException: org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to org.apache.beam.sdk.coders.KvCoder And I have no clue where to start looking to fix that as I’d assume the demos should just run? Tom -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
Re: Running Python Wordcount issues
This is a known issue with Beam 2.15. Using master or the 2.16 branch should fix the problem. Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com On Mon, Sep 16, 2019 at 4:31 PM Tom Barber wrote: > Hello folks, > > Trying to get started running the python word count example against the > portable runner using the beam 2.15 download. > > In on terminal I have: > > gradle :runners:spark:job-server:runShadow > > And another: > > gradle portableWordCount -PjobEndpoint=localhost:8099 > -PenvironmentType=LOOPBACK > > > But when it starts it says: > > Traceback (most recent call last): > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > line 135, in > run() > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > line 115, in run > result.wait_until_finish() > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py", > line 446, in wait_until_finish > self._job_id, self._state, self._last_error_message())) > RuntimeError: Pipeline > BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546 > failed in state FAILED: java.lang.ClassCastException: > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to > org.apache.beam.sdk.coders.KvCoder > > And I have no clue where to start looking to fix that as I’d assume the > demos should just run? > > Tom > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > All engagements are subject to Spicule Terms and Conditions of Business. > This email and its contents are intended solely for the individual to whom > it is addressed and may contain information that is confidential, > privileged or otherwise protected from disclosure, distributing or copying. > Any views or opinions presented in this email are solely those of the > author and do not necessarily represent those of Spicule Limited. The > company accepts no liability for any damage caused by any virus transmitted > by this email. If you have received this message in error, please notify us > immediately by reply email before deleting it from your system. Service of > legal notice cannot be effected on Spicule Limited by email. >
Re: Running Python Wordcount issues
Ah I found https://issues.apache.org/jira/browse/BEAM-7864 which I guess is related. I’ll try a Git based version. On 17 September 2019 at 00:30:54, Tom Barber (t...@spicule.co.uk) wrote: Hello folks, Trying to get started running the python word count example against the portable runner using the beam 2.15 download. In on terminal I have: gradle :runners:spark:job-server:runShadow And another: gradle portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK But when it starts it says: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", line 135, in run() File "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", line 115, in run result.wait_until_finish() File "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py", line 446, in wait_until_finish self._job_id, self._state, self._last_error_message())) RuntimeError: Pipeline BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546 failed in state FAILED: java.lang.ClassCastException: org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to org.apache.beam.sdk.coders.KvCoder And I have no clue where to start looking to fix that as I’d assume the demos should just run? Tom -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
Re: Running Python Wordcount issues
I can confirm. I used the master branch to get this to work. I tried the "release-2.16.0" branch but that didn't work either. On 2019/09/16 23:30:54, Tom Barber wrote: > Hello folks, > > Trying to get started running the python word count example against the > portable runner using the beam 2.15 download. > > In on terminal I have: > > gradle :runners:spark:job-server:runShadow > > And another: > > gradle portableWordCount -PjobEndpoint=localhost:8099 > -PenvironmentType=LOOPBACK > > > But when it starts it says: > > Traceback (most recent call last): > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > line 135, in > run() > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > line 115, in run > result.wait_until_finish() > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py", > line 446, in wait_until_finish > self._job_id, self._state, self._last_error_message())) > RuntimeError: Pipeline > BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546 > failed in state FAILED: java.lang.ClassCastException: > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to > org.apache.beam.sdk.coders.KvCoder > > And I have no clue where to start looking to fix that as I’d assume the > demos should just run? > > Tom > > -- > > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > All engagements > are subject to Spicule Terms and Conditions of Business. This email and its > contents are intended solely for the individual to whom it is addressed and > may contain information that is confidential, privileged or otherwise > protected from disclosure, distributing or copying. Any views or opinions > presented in this email are solely those of the author and do not > necessarily represent those of Spicule Limited. The company accepts no > liability for any damage caused by any virus transmitted by this email. If > you have received this message in error, please notify us immediately by > reply email before deleting it from your system. Service of legal notice > cannot be effected on Spicule Limited by email. >
Re: Running Python Wordcount issues
Alright well I’ve got 2.16 checked out and building so I’ll test it and feedback. Thanks! On 17 September 2019 at 00:43:57, Benjamin Tan (benjamintanwei...@gmail.com) wrote: I can confirm. I used the master branch to get this to work. I tried the "release-2.16.0" branch but that didn't work either. On 2019/09/16 23:30:54, Tom Barber wrote: > Hello folks, > > Trying to get started running the python word count example against the > portable runner using the beam 2.15 download. > > In on terminal I have: > > gradle :runners:spark:job-server:runShadow > > And another: > > gradle portableWordCount -PjobEndpoint=localhost:8099 > -PenvironmentType=LOOPBACK > > > But when it starts it says: > > Traceback (most recent call last): > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > line 135, in > run() > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > line 115, in run > result.wait_until_finish() > File > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py", > line 446, in wait_until_finish > self._job_id, self._state, self._last_error_message())) > RuntimeError: Pipeline > BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546 > failed in state FAILED: java.lang.ClassCastException: > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to > org.apache.beam.sdk.coders.KvCoder > > And I have no clue where to start looking to fix that as I’d assume the > demos should just run? > > Tom > > -- > > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > All engagements > are subject to Spicule Terms and Conditions of Business. This email and its > contents are intended solely for the individual to whom it is addressed and > may contain information that is confidential, privileged or otherwise > protected from disclosure, distributing or copying. Any views or opinions > presented in this email are solely those of the author and do not > necessarily represent those of Spicule Limited. The company accepts no > liability for any damage caused by any virus transmitted by this email. If > you have received this message in error, please notify us immediately by > reply email before deleting it from your system. Service of legal notice > cannot be effected on Spicule Limited by email. > -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
Re: Running Python Wordcount issues
What errors did you get with the release-2.16.0 branch? On Mon, Sep 16, 2019 at 4:43 PM Benjamin Tan wrote: > > I can confirm. I used the master branch to get this to work. I tried the > "release-2.16.0" branch but that didn't work either. > > On 2019/09/16 23:30:54, Tom Barber wrote: > > Hello folks, > > > > Trying to get started running the python word count example against the > > portable runner using the beam 2.15 download. > > > > In on terminal I have: > > > > gradle :runners:spark:job-server:runShadow > > > > And another: > > > > gradle portableWordCount -PjobEndpoint=localhost:8099 > > -PenvironmentType=LOOPBACK > > > > > > But when it starts it says: > > > > Traceback (most recent call last): > > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > > "__main__", fname, loader, pkg_name) > > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > > exec code in run_globals > > File > > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > > line 135, in > > run() > > File > > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > > line 115, in run > > result.wait_until_finish() > > File > > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py", > > line 446, in wait_until_finish > > self._job_id, self._state, self._last_error_message())) > > RuntimeError: Pipeline > > BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546 > > failed in state FAILED: java.lang.ClassCastException: > > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to > > org.apache.beam.sdk.coders.KvCoder > > > > And I have no clue where to start looking to fix that as I’d assume the > > demos should just run? > > > > Tom > > > > -- > > > > > > Spicule Limited is registered in England & Wales. Company Number: > > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > > > > > > All engagements > > are subject to Spicule Terms and Conditions of Business. This email and its > > contents are intended solely for the individual to whom it is addressed and > > may contain information that is confidential, privileged or otherwise > > protected from disclosure, distributing or copying. Any views or opinions > > presented in this email are solely those of the author and do not > > necessarily represent those of Spicule Limited. The company accepts no > > liability for any damage caused by any virus transmitted by this email. If > > you have received this message in error, please notify us immediately by > > reply email before deleting it from your system. Service of legal notice > > cannot be effected on Spicule Limited by email. > >
Re: Running Python Wordcount issues
That error went away for me on 2.16 from Git. Tom On 17 September 2019 at 00:47:27, Robert Bradshaw (rober...@google.com) wrote: What errors did you get with the release-2.16.0 branch? On Mon, Sep 16, 2019 at 4:43 PM Benjamin Tan wrote: > > I can confirm. I used the master branch to get this to work. I tried the "release-2.16.0" branch but that didn't work either. > > On 2019/09/16 23:30:54, Tom Barber wrote: > > Hello folks, > > > > Trying to get started running the python word count example against the > > portable runner using the beam 2.15 download. > > > > In on terminal I have: > > > > gradle :runners:spark:job-server:runShadow > > > > And another: > > > > gradle portableWordCount -PjobEndpoint=localhost:8099 > > -PenvironmentType=LOOPBACK > > > > > > But when it starts it says: > > > > Traceback (most recent call last): > > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > > "__main__", fname, loader, pkg_name) > > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > > exec code in run_globals > > File > > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > > line 135, in > > run() > > File > > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py", > > line 115, in run > > result.wait_until_finish() > > File > > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py", > > line 446, in wait_until_finish > > self._job_id, self._state, self._last_error_message())) > > RuntimeError: Pipeline > > BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546 > > failed in state FAILED: java.lang.ClassCastException: > > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to > > org.apache.beam.sdk.coders.KvCoder > > > > And I have no clue where to start looking to fix that as I’d assume the > > demos should just run? > > > > Tom > > > > -- > > > > > > Spicule Limited is registered in England & Wales. Company Number: > > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > > > > > > All engagements > > are subject to Spicule Terms and Conditions of Business. This email and its > > contents are intended solely for the individual to whom it is addressed and > > may contain information that is confidential, privileged or otherwise > > protected from disclosure, distributing or copying. Any views or opinions > > presented in this email are solely those of the author and do not > > necessarily represent those of Spicule Limited. The company accepts no > > liability for any damage caused by any virus transmitted by this email. If > > you have received this message in error, please notify us immediately by > > reply email before deleting it from your system. Service of legal notice > > cannot be effected on Spicule Limited by email. > > -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.