I'm guessing you need to install virtualenv: `pip install virtualenv` Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mpatter...@aligntech.com> wrote: > Kyle, > > > > Excellent, will do: unfortunately switch to 2.16 was only thing that fixed > > > > FAILURE: Build failed with an exception. > > > > * What went wrong: > > Execution failed for task ':sdks:python:setupVirtualenv'. > > > A problem occurred starting process 'command 'virtualenv'' > > > > > > …if you have suggestions there, I would be most appreciative. > > Matt > > > > > > *From: *Kyle Weaver <kcwea...@google.com> > *Reply-To: *"user@beam.apache.org" <user@beam.apache.org> > *Date: *Thursday, September 19, 2019 at 2:25 PM > *To: *"user@beam.apache.org" <user@beam.apache.org> > *Subject: *Re: Word-count example > > > > *CAUTION:* This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > You should probably use 2.15, since 2.16 release artifacts have not been > published yet. Just follow the instructions that > say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit > that other deserialization bug that was mentioned.. > > > Kyle Weaver | Software Engineer | github.com/ibzib > <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466096491&sdata=K%2F35WYwuy5EMBvpFX2uFmlJnoQSX94FuZNg%2Bj5RNelg%3D&reserved=0> > | kcwea...@google.com > > > > > > On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson < > mpatter...@aligntech.com> wrote: > > Kyle, > > > > Happily: should I be working from (source) release-2.15.0 or > release-2.16.0: presumably still specifying flink-1.8. > > Matt > > > > *From: *Kyle Weaver <kcwea...@google.com> > *Reply-To: *"user@beam.apache.org" <user@beam.apache.org> > *Date: *Thursday, September 19, 2019 at 2:16 PM > *To: *"user@beam.apache.org" <user@beam.apache.org> > *Subject: *Re: Word-count example > > > > *CAUTION:* This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies > here. If I understand correctly, Matthew is just trying to run an > old-fashioned Beam Java jar, nothing to do with portability/Python, and > judging by the stack trace provided, > https://issues.apache.org/jira/browse/BEAM-8037 > <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466106484&sdata=U0r4Nx4dJuEy2IHe6o1ozKpe6uwCo6F%2F3%2Bn4mK9pge0%3D&reserved=0> > is > not the issue here. > > > > Re Matthew: I suspect a Flink version mismatch between the Flink cluster > and the Beam runner. However, I am not familiar with the Maven commands > you're using. Since you're planning on running Python eventually anyway, > could you try following the "portable"/Python instructions on > https://beam.apache.org/documentation/runners/flink/ > <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466106484&sdata=I9WBW9BkLx3RMsBSD4NPZUJHugmyh0UpxGWzHdSeLbU%3D&reserved=0> > and > let us know how that goes? > > > Kyle Weaver | Software Engineer | github.com/ibzib > <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466116480&sdata=fDrnP5FRiHYt0ceL98BywtKPtHMAA4uKlF40Xp5gi9k%3D&reserved=0> > | kcwea...@google.com > > > > > > On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson < > mpatter...@aligntech.com> wrote: > > Thanks Ankur, > > > > As one who speaks almost no gradle, is there a preferred way to get gradle > to respect the conda configured python on `gradlew build` ? > > > > Matt > > > > *From: *Ankur Goenka <goe...@google.com> > *Reply-To: *"user@beam.apache.org" <user@beam.apache.org> > *Date: *Thursday, September 19, 2019 at 1:50 PM > *To: *"user@beam.apache.org" <user@beam.apache.org> > *Subject: *Re: Word-count example > > > > *CAUTION:* This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > We have this bug in 2.15 which is discussed here > https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E > <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466126476&sdata=aMWF9Sbh2lKEupW45eddMclF2fZ5on9kJHgtQfAJIAQ%3D&reserved=0> > > > > For now, please add "--experiments=beam_fn_api" to your pipeline to make > it work. > > > > On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson < > mpatter...@aligntech.com> wrote: > > Hi Ankur, > > > > Yes, I was using 2.15, but was getting failure to deserialize. > > > > Thanks, > > Matt > > > > *From: *Ankur Goenka <goe...@google.com> > *Reply-To: *"user@beam.apache.org" <user@beam.apache.org> > *Date: *Wednesday, September 18, 2019 at 9:34 PM > *To: *"user@beam.apache.org" <user@beam.apache.org> > *Subject: *Re: Word-count example > > > > *CAUTION:* This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > Hi Matthew, > > > > Beam 2.16.0 is not yet released hence you are getting the error. Can you > try using 2.15.0 version? > > > > Thanks, > > Ankur > > > > On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson < > mpatter...@aligntech.com> wrote: > > Tried > > " > mvn archetype:generate \ > -DarchetypeGroupId=org.apache.beam \ > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ > -DarchetypeVersion=2.16.0 \ > -DgroupId=org.example \ > -DartifactId=word-count-beam \ > -Dversion="0.1" \ > -Dpackage=org.apache.beam.examples \ > -DinteractiveMode=false > " > got > > -> [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate > (default-cli) on project standalone-pom: The desired archetype does not > exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> > [Help 1] > > > On 9/17/19, 7:51 PM, "Benjamin Tan" <benjamintanwei...@gmail.com> wrote: > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you recognize the sender and > know the content is safe. > > > Ah. So maybe try 2.16.0? > > On 2019/09/17 23:47:16, Kyle Weaver <kcwea...@google.com> wrote: > > --experiments=beam_fn_api doesn't apply here, as this is a Java > pipeline > > using the non-portable version of the Flink runner. > > > > Kyle Weaver | Software Engineer | github.com/ibzib > <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466126476&sdata=t17K7Pf1pA4GQd1gbEEEaPiO%2ByyeggXTheF6szt5e4c%3D&reserved=0> > | kcwea...@google.com > > > > > > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan < > benjamintanwei...@gmail.com> > > wrote: > > > > > Could you try adding "--experiments=beam_fn_api" to the runner > options? > > > Kyle did mention that it should be fixed in 2.16.0. > > > > > > On 2019/09/17 16:40:41, Matthew Patterson < > mpatter...@aligntech.com> > > > wrote: > > > > Folks, > > > > > > > > Am sure I am doing it wrong, but have been beating head against > wall > > > long enough I need to ask for help. > > > > > > > > My goal is to use the python SDK to run jobs on AWS flink > cluster, but > > > very little is working, so I am trying to back down to the very > simplest > > > thing that works, and build back up: not going so well. > > > > > > > > Now I am running, or trying to, on a minikube link cluster. Now > running > > > from just java/mvn, so I go through the following: > > > > > > > > 1. bring-up-cluster (see below) > > > > 2. grab-the-code (see below) > > > > 3. maven-submit (see below) > > > > > > > > Eventual failure is > > > > > > > > “[ERROR] Failed to execute goal > > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on > project > > > word-count-beam: An exception occured while executing the Java > class. > > > Pipeline execution failed: Could not retrieve the execution > result. (JobID: > > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. > [Failed to > > > deserialize JobGraph.]” > > > > > > > > and log is attached (I hope) > > > > > > > > Other details: > > > > > > > > on osx > > > > > > > > > > > > > > > > ➜ minikube version > > > > > > > > minikube version: v1.3.1 > > > > > > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631 > > > > > > > > > > > > ➜ kubectl version > > > > > > > > Client Version: version.Info{Major:"1", Minor:"14", > > > GitVersion:"v1.14.6", > GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", > > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", > > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} > > > > > > > > Server Version: version.Info{Major:"1", Minor:"15", > > > GitVersion:"v1.15.2", > GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", > > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", > > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} > > > > > > > > > > > > ➜ flink --version > > > > > > > > Version: 1.8.1, Commit ID: 7297bac > > > > > > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)" > > > > 2.15.0 > > > > > > > > ➜ curl localhost:8081 > > > > <!-- > > > > ~ Licensed to the Apache Software Foundation (ASF) under one > > > > ~ or more contributor license agreements. See the NOTICE file > > > > ~ distributed with this work for additional information > > > > ~ regarding copyright ownership. The ASF licenses this file > > > > ~ to you under the Apache License, Version 2.0 (the > > > > ~ "License"); you may not use this file except in compliance > > > > ~ with the License. You may obtain a copy of the License at > > > > ~ > > > > ~ > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0 > <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466136471&sdata=eUFzyBye%2F13md7qkKI0jHnAT1suszfti6PgOPADLDno%3D&reserved=0> > > > > ~ > > > > ~ Unless required by applicable law or agreed to in writing, > software > > > > ~ distributed under the License is distributed on an "AS IS" > BASIS, > > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express > or > > > implied. > > > > ~ See the License for the specific language governing > permissions and > > > > ~ limitations under the License. > > > > --> > > > > > > > > <!doctype html> > > > > <html lang="en"> > > > > <head> > > > > <meta charset="utf-8"> > > > > <meta http-equiv="X-UA-Compatible" content="IE=edge"> > > > > <meta name="viewport" content="width=device-width, > initial-scale=1"> > > > > <title>Apache Flink Web Dashboard</title> > > > > <link rel="apple-touch-icon" sizes="180x180" > > > href="assets/favicon/apple-touch-icon.png"> > > > > <link rel="icon" type="image/png" > > > href="assets/favicon/favicon-32x32.png" sizes="32x32"> > > > > <link rel="icon" type="image/png" > > > href="assets/favicon/favicon-16x16.png" sizes="16x16"> > > > > <link rel="manifest" href="assets/favicon/manifest.json"> > > > > <link rel="mask-icon" > href="assets/favicon/safari-pinned-tab.svg" > > > color="#aa1919"> > > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico"> > > > > <meta name="msapplication-config" > > > content="assets/favicon/browserconfig.xml"> > > > > <meta name="theme-color" content="#ffffff"> > > > > <base href="./"><link rel="stylesheet" > > > href="styles.30d0912c1ece284d8d9a.css"></head> > > > > <body> > > > > <flink-root></flink-root> > > > > <script type="text/javascript" > > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script > > > type="text/javascript" > src="es2015-polyfills.5e343224e81eefb7658e.js" > > > nomodule></script><script type="text/javascript" > > > src="polyfills.b37850e8279bc3caafc9.js"></script><script > > > type="text/javascript" > src="main.dc6e0997d092df051db0.js"></script></body> > > > > </html> > > > > > > > > Thanks for any ideas, > > > > Matt > > > > > > > > >>> bring-up-cluster > > > > > > > > # [yamls from here]( > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0 > <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466146466&sdata=UtKVyOEu6bFQd6g06WHtBv%2FZQE7Dj2umyr5aPhuAKlg%3D&reserved=0> > > > ) > > > > minikube start > > > > minikube ssh 'sudo ip link set docker0 promisc on' > > > > > > > > kubectl create -f flink-configuration-configmap.yaml > > > > kubectl create -f jobmanager-service.yaml > > > > kubectl create -f jobmanager-deployment.yaml > > > > kubectl create -f taskmanager-deployment.yaml > > > > kubectl apply -f jobmanager-rest-service.yaml > > > > > > > > >>> grab-the-code > > > > mvn archetype:generate \ > > > > -DarchetypeGroupId=org.apache.beam \ > > > > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ > > > > -DarchetypeVersion=2.15.0 \ > > > > -DgroupId=org.example \ > > > > -DartifactId=word-count-beam \ > > > > -Dversion="0.1" \ > > > > -Dpackage=org.apache.beam.examples \ > > > > -DinteractiveMode=false > > > > > > > > >>> maven-submit > > > > > > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in > separate > > > shell, as blocking > > > > mvn package exec:java -X > > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ > > > > -Dexec.args="--runner=FlinkRunner\ > > > > --flinkMaster="localhost:8081"\ > > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\ > > > > --output=<directory-that-exists>" -Pflink-runner > > > > > > > > > > > > > > > > > > >
