Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037 is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/ and let us know how that goes? Kyle Weaver | Software Engineer | github.com/ibzib | [email protected] On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <[email protected]> wrote: > Thanks Ankur, > > > > As one who speaks almost no gradle, is there a preferred way to get gradle > to respect the conda configured python on `gradlew build` ? > > > > Matt > > > > *From: *Ankur Goenka <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Thursday, September 19, 2019 at 1:50 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Word-count example > > > > *CAUTION:* This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > We have this bug in 2.15 which is discussed here > https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E > <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081208279&sdata=mUbuMEq0UHke7igqGwC0Lv4p7CLpS3tG7zDEHoS9B%2B4%3D&reserved=0> > > > > For now, please add "--experiments=beam_fn_api" to your pipeline to make > it work. > > > > On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson < > [email protected]> wrote: > > Hi Ankur, > > > > Yes, I was using 2.15, but was getting failure to deserialize. > > > > Thanks, > > Matt > > > > *From: *Ankur Goenka <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Wednesday, September 18, 2019 at 9:34 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Word-count example > > > > *CAUTION:* This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > Hi Matthew, > > > > Beam 2.16.0 is not yet released hence you are getting the error. Can you > try using 2.15.0 version? > > > > Thanks, > > Ankur > > > > On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson < > [email protected]> wrote: > > Tried > > " > mvn archetype:generate \ > -DarchetypeGroupId=org.apache.beam \ > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ > -DarchetypeVersion=2.16.0 \ > -DgroupId=org.example \ > -DartifactId=word-count-beam \ > -Dversion="0.1" \ > -Dpackage=org.apache.beam.examples \ > -DinteractiveMode=false > " > got > > -> [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate > (default-cli) on project standalone-pom: The desired archetype does not > exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> > [Help 1] > > > On 9/17/19, 7:51 PM, "Benjamin Tan" <[email protected]> wrote: > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you recognize the sender and > know the content is safe. > > > Ah. So maybe try 2.16.0? > > On 2019/09/17 23:47:16, Kyle Weaver <[email protected]> wrote: > > --experiments=beam_fn_api doesn't apply here, as this is a Java > pipeline > > using the non-portable version of the Flink runner. > > > > Kyle Weaver | Software Engineer | github.com/ibzib > <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081218275&sdata=lFcrLDfXJ86ZICr70bvdnBa4gyKiukFhWYolPrBHZO4%3D&reserved=0> > | [email protected] > > > > > > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan < > [email protected]> > > wrote: > > > > > Could you try adding "--experiments=beam_fn_api" to the runner > options? > > > Kyle did mention that it should be fixed in 2.16.0. > > > > > > On 2019/09/17 16:40:41, Matthew Patterson < > [email protected]> > > > wrote: > > > > Folks, > > > > > > > > Am sure I am doing it wrong, but have been beating head against > wall > > > long enough I need to ask for help. > > > > > > > > My goal is to use the python SDK to run jobs on AWS flink > cluster, but > > > very little is working, so I am trying to back down to the very > simplest > > > thing that works, and build back up: not going so well. > > > > > > > > Now I am running, or trying to, on a minikube link cluster. Now > running > > > from just java/mvn, so I go through the following: > > > > > > > > 1. bring-up-cluster (see below) > > > > 2. grab-the-code (see below) > > > > 3. maven-submit (see below) > > > > > > > > Eventual failure is > > > > > > > > “[ERROR] Failed to execute goal > > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on > project > > > word-count-beam: An exception occured while executing the Java > class. > > > Pipeline execution failed: Could not retrieve the execution > result. (JobID: > > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. > [Failed to > > > deserialize JobGraph.]” > > > > > > > > and log is attached (I hope) > > > > > > > > Other details: > > > > > > > > on osx > > > > > > > > > > > > > > > > ➜ minikube version > > > > > > > > minikube version: v1.3.1 > > > > > > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631 > > > > > > > > > > > > ➜ kubectl version > > > > > > > > Client Version: version.Info{Major:"1", Minor:"14", > > > GitVersion:"v1.14.6", > GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", > > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", > > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} > > > > > > > > Server Version: version.Info{Major:"1", Minor:"15", > > > GitVersion:"v1.15.2", > GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", > > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", > > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} > > > > > > > > > > > > ➜ flink --version > > > > > > > > Version: 1.8.1, Commit ID: 7297bac > > > > > > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)" > > > > 2.15.0 > > > > > > > > ➜ curl localhost:8081 > > > > <!-- > > > > ~ Licensed to the Apache Software Foundation (ASF) under one > > > > ~ or more contributor license agreements. See the NOTICE file > > > > ~ distributed with this work for additional information > > > > ~ regarding copyright ownership. The ASF licenses this file > > > > ~ to you under the Apache License, Version 2.0 (the > > > > ~ "License"); you may not use this file except in compliance > > > > ~ with the License. You may obtain a copy of the License at > > > > ~ > > > > ~ > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0 > <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081228269&sdata=Zhg43%2FegQVw%2BX9h0ht4bw%2FoQnT7Vbclgx6Wg%2BI16jOg%3D&reserved=0> > > > > ~ > > > > ~ Unless required by applicable law or agreed to in writing, > software > > > > ~ distributed under the License is distributed on an "AS IS" > BASIS, > > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express > or > > > implied. > > > > ~ See the License for the specific language governing > permissions and > > > > ~ limitations under the License. > > > > --> > > > > > > > > <!doctype html> > > > > <html lang="en"> > > > > <head> > > > > <meta charset="utf-8"> > > > > <meta http-equiv="X-UA-Compatible" content="IE=edge"> > > > > <meta name="viewport" content="width=device-width, > initial-scale=1"> > > > > <title>Apache Flink Web Dashboard</title> > > > > <link rel="apple-touch-icon" sizes="180x180" > > > href="assets/favicon/apple-touch-icon.png"> > > > > <link rel="icon" type="image/png" > > > href="assets/favicon/favicon-32x32.png" sizes="32x32"> > > > > <link rel="icon" type="image/png" > > > href="assets/favicon/favicon-16x16.png" sizes="16x16"> > > > > <link rel="manifest" href="assets/favicon/manifest.json"> > > > > <link rel="mask-icon" > href="assets/favicon/safari-pinned-tab.svg" > > > color="#aa1919"> > > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico"> > > > > <meta name="msapplication-config" > > > content="assets/favicon/browserconfig.xml"> > > > > <meta name="theme-color" content="#ffffff"> > > > > <base href="./"><link rel="stylesheet" > > > href="styles.30d0912c1ece284d8d9a.css"></head> > > > > <body> > > > > <flink-root></flink-root> > > > > <script type="text/javascript" > > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script > > > type="text/javascript" > src="es2015-polyfills.5e343224e81eefb7658e.js" > > > nomodule></script><script type="text/javascript" > > > src="polyfills.b37850e8279bc3caafc9.js"></script><script > > > type="text/javascript" > src="main.dc6e0997d092df051db0.js"></script></body> > > > > </html> > > > > > > > > Thanks for any ideas, > > > > Matt > > > > > > > > >>> bring-up-cluster > > > > > > > > # [yamls from here]( > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0 > <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081228269&sdata=eegtSWsLzx1MtI6b%2Bvnm0H58VSRhOhph1Zh86CJL0iI%3D&reserved=0> > > > ) > > > > minikube start > > > > minikube ssh 'sudo ip link set docker0 promisc on' > > > > > > > > kubectl create -f flink-configuration-configmap.yaml > > > > kubectl create -f jobmanager-service.yaml > > > > kubectl create -f jobmanager-deployment.yaml > > > > kubectl create -f taskmanager-deployment.yaml > > > > kubectl apply -f jobmanager-rest-service.yaml > > > > > > > > >>> grab-the-code > > > > mvn archetype:generate \ > > > > -DarchetypeGroupId=org.apache.beam \ > > > > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ > > > > -DarchetypeVersion=2.15.0 \ > > > > -DgroupId=org.example \ > > > > -DartifactId=word-count-beam \ > > > > -Dversion="0.1" \ > > > > -Dpackage=org.apache.beam.examples \ > > > > -DinteractiveMode=false > > > > > > > > >>> maven-submit > > > > > > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in > separate > > > shell, as blocking > > > > mvn package exec:java -X > > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ > > > > -Dexec.args="--runner=FlinkRunner\ > > > > --flinkMaster="localhost:8081"\ > > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\ > > > > --output=<directory-that-exists>" -Pflink-runner > > > > > > > > > > > > > > > > > > >
