Thanks Ankur, As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt From: Ankur Goenka <goe...@google.com> Reply-To: "user@beam.apache.org" <user@beam.apache.org> Date: Thursday, September 19, 2019 at 1:50 PM To: "user@beam.apache.org" <user@beam.apache.org> Subject: Re: Word-count example CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081208279&sdata=mUbuMEq0UHke7igqGwC0Lv4p7CLpS3tG7zDEHoS9B%2B4%3D&reserved=0> For now, please add "--experiments=beam_fn_api" to your pipeline to make it work. On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mpatter...@aligntech.com<mailto:mpatter...@aligntech.com>> wrote: Hi Ankur, Yes, I was using 2.15, but was getting failure to deserialize. Thanks, Matt From: Ankur Goenka <goe...@google.com<mailto:goe...@google.com>> Reply-To: "user@beam.apache.org<mailto:user@beam.apache.org>" <user@beam.apache.org<mailto:user@beam.apache.org>> Date: Wednesday, September 18, 2019 at 9:34 PM To: "user@beam.apache.org<mailto:user@beam.apache.org>" <user@beam.apache.org<mailto:user@beam.apache.org>> Subject: Re: Word-count example CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Matthew, Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version? Thanks, Ankur On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mpatter...@aligntech.com<mailto:mpatter...@aligntech.com>> wrote: Tried " mvn archetype:generate \ -DarchetypeGroupId=org.apache.beam \ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ -DarchetypeVersion=2.16.0 \ -DgroupId=org.example \ -DartifactId=word-count-beam \ -Dversion="0.1" \ -Dpackage=org.apache.beam.examples \ -DinteractiveMode=false " got -> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1] On 9/17/19, 7:51 PM, "Benjamin Tan" <benjamintanwei...@gmail.com<mailto:benjamintanwei...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Ah. So maybe try 2.16.0? On 2019/09/17 23:47:16, Kyle Weaver <kcwea...@google.com<mailto:kcwea...@google.com>> wrote: > --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline > using the non-portable version of the Flink runner. > > Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081218275&sdata=lFcrLDfXJ86ZICr70bvdnBa4gyKiukFhWYolPrBHZO4%3D&reserved=0> | kcwea...@google.com<mailto:kcwea...@google.com> > > > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <benjamintanwei...@gmail.com<mailto:benjamintanwei...@gmail.com>> > wrote: > > > Could you try adding "--experiments=beam_fn_api" to the runner options? > > Kyle did mention that it should be fixed in 2.16.0. > > > > On 2019/09/17 16:40:41, Matthew Patterson <mpatter...@aligntech.com<mailto:mpatter...@aligntech.com>> > > wrote: > > > Folks, > > > > > > Am sure I am doing it wrong, but have been beating head against wall > > long enough I need to ask for help. > > > > > > My goal is to use the python SDK to run jobs on AWS flink cluster, but > > very little is working, so I am trying to back down to the very simplest > > thing that works, and build back up: not going so well. > > > > > > Now I am running, or trying to, on a minikube link cluster. Now running > > from just java/mvn, so I go through the following: > > > > > > 1. bring-up-cluster (see below) > > > 2. grab-the-code (see below) > > > 3. maven-submit (see below) > > > > > > Eventual failure is > > > > > > “[ERROR] Failed to execute goal > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project > > word-count-beam: An exception occured while executing the Java class. > > Pipeline execution failed: Could not retrieve the execution result. (JobID: > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to > > deserialize JobGraph.]” > > > > > > and log is attached (I hope) > > > > > > Other details: > > > > > > on osx > > > > > > > > > > > > ➜ minikube version > > > > > > minikube version: v1.3.1 > > > > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631 > > > > > > > > > ➜ kubectl version > > > > > > Client Version: version.Info{Major:"1", Minor:"14", > > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} > > > > > > Server Version: version.Info{Major:"1", Minor:"15", > > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} > > > > > > > > > ➜ flink --version > > > > > > Version: 1.8.1, Commit ID: 7297bac > > > > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)" > > > 2.15.0 > > > > > > ➜ curl localhost:8081 > > > <!-- > > > ~ Licensed to the Apache Software Foundation (ASF) under one > > > ~ or more contributor license agreements. See the NOTICE file > > > ~ distributed with this work for additional information > > > ~ regarding copyright ownership. The ASF licenses this file > > > ~ to you under the Apache License, Version 2.0 (the > > > ~ "License"); you may not use this file except in compliance > > > ~ with the License. You may obtain a copy of the License at > > > ~ > > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081228269&sdata=Zhg43%2FegQVw%2BX9h0ht4bw%2FoQnT7Vbclgx6Wg%2BI16jOg%3D&reserved=0> > > > ~ > > > ~ Unless required by applicable law or agreed to in writing, software > > > ~ distributed under the License is distributed on an "AS IS" BASIS, > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > > implied. > > > ~ See the License for the specific language governing permissions and > > > ~ limitations under the License. > > > --> > > > > > > <!doctype html> > > > <html lang="en"> > > > <head> > > > <meta charset="utf-8"> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge"> > > > <meta name="viewport" content="width=device-width, initial-scale=1"> > > > <title>Apache Flink Web Dashboard</title> > > > <link rel="apple-touch-icon" sizes="180x180" > > href="assets/favicon/apple-touch-icon.png"> > > > <link rel="icon" type="image/png" > > href="assets/favicon/favicon-32x32.png" sizes="32x32"> > > > <link rel="icon" type="image/png" > > href="assets/favicon/favicon-16x16.png" sizes="16x16"> > > > <link rel="manifest" href="assets/favicon/manifest.json"> > > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg" > > color="#aa1919"> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico"> > > > <meta name="msapplication-config" > > content="assets/favicon/browserconfig.xml"> > > > <meta name="theme-color" content="#ffffff"> > > > <base href="./"><link rel="stylesheet" > > href="styles.30d0912c1ece284d8d9a.css"></head> > > > <body> > > > <flink-root></flink-root> > > > <script type="text/javascript" > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script > > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js" > > nomodule></script><script type="text/javascript" > > src="polyfills.b37850e8279bc3caafc9.js"></script><script > > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body> > > > </html> > > > > > > Thanks for any ideas, > > > Matt > > > > > > >>> bring-up-cluster > > > > > > # [yamls from here]( > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081228269&sdata=eegtSWsLzx1MtI6b%2Bvnm0H58VSRhOhph1Zh86CJL0iI%3D&reserved=0> > > ) > > > minikube start > > > minikube ssh 'sudo ip link set docker0 promisc on' > > > > > > kubectl create -f flink-configuration-configmap.yaml > > > kubectl create -f jobmanager-service.yaml > > > kubectl create -f jobmanager-deployment.yaml > > > kubectl create -f taskmanager-deployment.yaml > > > kubectl apply -f jobmanager-rest-service.yaml > > > > > > >>> grab-the-code > > > mvn archetype:generate \ > > > -DarchetypeGroupId=org.apache.beam \ > > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ > > > -DarchetypeVersion=2.15.0 \ > > > -DgroupId=org.example \ > > > -DartifactId=word-count-beam \ > > > -Dversion="0.1" \ > > > -Dpackage=org.apache.beam.examples \ > > > -DinteractiveMode=false > > > > > > >>> maven-submit > > > > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate > > shell, as blocking > > > mvn package exec:java -X > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ > > > -Dexec.args="--runner=FlinkRunner\ > > > --flinkMaster="localhost:8081"\ > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\ > > > --output=<directory-that-exists>" -Pflink-runner > > > > > > > > > > > >
