Sorry: needed to add LOOPBACK
From: Matthew Patterson <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Monday, September 23, 2019 at 2:18 PM
To: "[email protected]" <[email protected]>
Subject: Re: Word-count example
OK, now using release-2.15.0, calling word-count example with runner, after
building & calling
./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
, as
“f'--runner=PortableRunner', f'--job_endpoint=localhost:8099'
”
…and getting
File
"/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline
BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94
failed in state FAILED:
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED:
cancelled before receiving half close
Ideas? Thanks!
From: Matthew Patterson <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Friday, September 20, 2019 at 9:27 AM
To: "[email protected]" <[email protected]>
Subject: Re: Word-count example
411:
Although it does not seem correct, adding a symbolic link to `virtualenv` in `
<path-to-beam>/sdks/python` solves the issue of not finding `virtualenv`.
From: Matthew Patterson <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Thursday, September 19, 2019 at 2:56 PM
To: "[email protected]" <[email protected]>
Subject: Re: Word-count example
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs
nonetheless): currently I am trying to figure out if it is a difference between
.bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start
'virtualenv'
at
net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at
net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at
org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at
org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory
"/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at
net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py
setup.py tox.ini
OWNERS container scripts
test-suites
apache_beam findSupportedPython.groovy setup.cfg
test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in
/Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py
setup.py tox.ini
OWNERS container scripts
test-suites
apache_beam findSupportedPython.groovy setup.cfg
test_config.py
bash-3.2$
”””
From: Matthew Patterson <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "[email protected]" <[email protected]>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "[email protected]" <[email protected]>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer |
github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224889006&sdata=pwoY1IhnKr7BXlaoZ%2B63SK%2FUPLUZ1ymctkHgr81W%2FUI%3D&reserved=0>
| [email protected]<mailto:[email protected]>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson
<[email protected]<mailto:[email protected]>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
You should probably use 2.15, since 2.16 release artifacts have not been
published yet. Just follow the instructions that say --runner=PortableRunner,
not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug
that was mentioned..
Kyle Weaver | Software Engineer |
github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224899004&sdata=AfIqyAtsTESluja%2BpfB3s2VMM5ztqCrU6erWtNnPhew%3D&reserved=0>
| [email protected]<mailto:[email protected]>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson
<[email protected]<mailto:[email protected]>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0:
presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If
I understand correctly, Matthew is just trying to run an old-fashioned Beam
Java jar, nothing to do with portability/Python, and judging by the stack trace
provided,
https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224908995&sdata=7tHvk8KMsn3MPdTJuOwyobzN7wo%2FwasgzcFFP3ZbgfE%3D&reserved=0>
is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and
the Beam runner. However, I am not familiar with the Maven commands you're
using. Since you're planning on running Python eventually anyway, could you try
following the "portable"/Python instructions on
https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224908995&sdata=bxQCAZXfqB8A72xysdBzR2l3IFBG25U0%2Bb6gqLcD298%3D&reserved=0>
and let us know how that goes?
Kyle Weaver | Software Engineer |
github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224918997&sdata=MyC%2BUO3kgB2f33GHqyKinxlxQCPovgPFk0PobGpvfkQ%3D&reserved=0>
| [email protected]<mailto:[email protected]>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson
<[email protected]<mailto:[email protected]>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to
respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
We have this bug in 2.15 which is discussed here
https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224928992&sdata=RPs2zBEjqn9TgnEVWWscatmC9DeV65B6KgSk9RdFcBo%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it
work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson
<[email protected]<mailto:[email protected]>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try
using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson
<[email protected]<mailto:[email protected]>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on
project standalone-pom: The desired archetype does not exist
(org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan"
<[email protected]<mailto:[email protected]>> wrote:
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know the
content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver
<[email protected]<mailto:[email protected]>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer |
github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224928992&sdata=XCiciM3hHt5MyYuJx4w%2FA53qKy6g9nzYqfsv8VkAbqI%3D&reserved=0>
| [email protected]<mailto:[email protected]>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan
<[email protected]<mailto:[email protected]>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson
<[email protected]<mailto:[email protected]>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now
running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result.
(JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6",
GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2",
GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~
https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224938984&sdata=vb2isCuzcnbVf3w94NtQ3Sws05j15I7BwDX0aQyq9Cs%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing,
software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions
and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript"
src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> >
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224948978&sdata=2W76N2GeYj1E75KcBPfS1BAFQX5SXdlBQLDiYHLzo8E%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > >
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
