To get Flink 1.4.0 installed on Dataproc, I used my own init action. After starting a detached flink session (e.g, HADOOP_CONF_DIR=/etc/hadoop/conf /usr/lib/flink/bin/yarn-session.sh -n 4 -tm 4000 -d), I pulled the Flink Job Manager address and port from the Application Master web UI. After that, I invoked WordCount as specified in the quickstart (via maven exec).
For sanity, I also packaged an uberjar and made sure that it ran locally and that it worked via flink run. To get the uberjar working, I had to use an org.apache.maven.plugins.shade.resource.AppendingTransformer on resource.conf files (Akka configurations). This change should probably be added to the quickstart archetype, but it's not necessary if applications are submitted as specified in the instructions. On Thu, Feb 15, 2018 at 6:40 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Great !!! > > Thanks for the update, I will close the vote then. > > Regards > JB > > On 02/15/2018 11:45 PM, Ben Sidhom wrote: > > I just successfully ran the quickstart on Flink 1.4.0 on Dataproc. > Should be > > good to go. > > > > > > On Thu, Feb 15, 2018 at 9:21 AM Jean-Baptiste Onofré <j...@nanthrax.net > > <mailto:j...@nanthrax.net>> wrote: > > > > Luke said he tested Flink locally. So, we have to test on a Yarn > cluster. > > > > Regards > > JB > > > > On 02/15/2018 06:16 PM, Reuven Lax wrote: > > > I count enough votes :) So sounds like someone needs to verify > Flink 1.4 > > > quickstart, and then we're ready? > > > > > > On Wed, Feb 14, 2018 at 5:38 PM, Robert Bradshaw < > rober...@google.com > > <mailto:rober...@google.com> > > > <mailto:rober...@google.com <mailto:rober...@google.com>>> wrote: > > > > > > +1 (binding) pending Flink verification. I have checked the > signatures > > > and checksums of the artifacts, and that it agrees with commit > > > 67b5e1bab25d284cdac2127b47f44acc8e83499e on github *except* > for 76 > > > places where -SNAPSHOT was removed. > > > > > > FWIW, http://www.apache.org/dev/release-distribution > > > <http://www.apache.org/dev/release-distribution> now > recommends > > > sha512 (over sha1 which has been broken). > > > > > > On Wed, Feb 14, 2018 at 5:24 PM, Eugene Kirpichov > > <kirpic...@google.com <mailto:kirpic...@google.com> > > > <mailto:kirpic...@google.com <mailto:kirpic...@google.com>>> > wrote: > > > > Thanks Kenn. I retract my -1, but then someone must verify > it with Flink > > > > 1.4. I might give it a shot tomorrow (installing Flink 1.4 > on Dataproc). > > > > > > > > On Wed, Feb 14, 2018 at 4:53 PM Reuven Lax <re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>>> wrote: > > > >> > > > >> +1 (binding) > > > >> > > > >> On Wed, Feb 14, 2018 at 10:42 AM, Alan Myrvold < > amyrv...@google.com > > <mailto:amyrv...@google.com> > > > <mailto:amyrv...@google.com <mailto:amyrv...@google.com>>> > > > >> wrote: > > > >>> > > > >>> +1 Validated java quickstarts for direct, dataflow, apex, > flink, and > > > >>> spark. > > > >>> > > > >>> On Wed, Feb 14, 2018 at 9:21 AM, Lukasz Cwik < > lc...@google.com > > <mailto:lc...@google.com> > > > <mailto:lc...@google.com <mailto:lc...@google.com>>> wrote: > > > >>>> > > > >>>> +1 (binding) > > > >>>> Validated several quickstarts including the regression > that I > > originally > > > >>>> reported with Spark. > > > >>>> > > > >>>> On Wed, Feb 14, 2018 at 5:34 AM, Ismaël Mejía < > ieme...@gmail.com > > <mailto:ieme...@gmail.com> > > > <mailto:ieme...@gmail.com <mailto:ieme...@gmail.com>>> wrote: > > > >>>>> > > > >>>>> +1 (binding) > > > >>>>> > > > >>>>> Validated SHAs + tag vs source.zip file. > > > >>>>> Run mvn clean install -Prelease OK > > > >>>>> Validated that the 3 regressions reported for RC1 were > fixed. > > > >>>>> Run Nexmark on Direct/Flink runner on local mode, no > regressions > > now. > > > >>>>> Installed python version on virtualenv and run local > wordcount with > > > >>>>> success. > > > >>>>> Checked that the hadoop-input-format artifact is in the > extended > > > >>>>> staging area. > > > >>>>> > > > >>>>> On Tue, Feb 13, 2018 at 5:41 PM, Jean-Baptiste Onofré > > <j...@nanthrax.net <mailto:j...@nanthrax.net> > > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> > > > >>>>> wrote: > > > >>>>> > +1 (binding) > > > >>>>> > > > > >>>>> > Tested the Spark runner (with wordcount example and > beam samples) > > > >>>>> > Tested the performance of the direct runner > > > >>>>> > > > > >>>>> > I just updated the spreadsheet. > > > >>>>> > > > > >>>>> > Regards > > > >>>>> > JB > > > >>>>> > > > > >>>>> > On 02/11/2018 06:33 AM, Jean-Baptiste Onofré wrote: > > > >>>>> >> Hi everyone, > > > >>>>> >> > > > >>>>> >> Please review and vote on the release candidate #3 > for the > > version > > > >>>>> >> 2.3.0, as > > > >>>>> >> follows: > > > >>>>> >> > > > >>>>> >> [ ] +1, Approve the release > > > >>>>> >> [ ] -1, Do not approve the release (please provide > specific > > > >>>>> >> comments) > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> The complete staging area is available for your > review, which > > > >>>>> >> includes: > > > >>>>> >> * JIRA release notes [1], > > > >>>>> >> * the official Apache source release to be deployed to > > > >>>>> >> dist.apache.org <http://dist.apache.org> > > <http://dist.apache.org> [2], > > > >>>>> >> which is signed with the key with fingerprint > C8282E76 [3], > > > >>>>> >> * all artifacts to be deployed to the Maven Central > > Repository [4], > > > >>>>> >> * source code tag "v2.3.0-RC3" [5], > > > >>>>> >> * website pull request listing the release and > publishing the API > > > >>>>> >> reference > > > >>>>> >> manual [6]. > > > >>>>> >> * Java artifacts were built with Maven 3.3.9 and > Oracle JDK > > > >>>>> >> 1.8.0_111. > > > >>>>> >> * Python artifacts are deployed along with the source > release > > to the > > > >>>>> >> dist.apache.org <http://dist.apache.org> > > <http://dist.apache.org> [2]. > > > >>>>> >> > > > >>>>> >> The vote will be open for at least 72 hours. It is > adopted by > > > >>>>> >> majority approval, > > > >>>>> >> with at least 3 PMC affirmative votes. > > > >>>>> >> > > > >>>>> >> Thanks, > > > >>>>> >> JB > > > >>>>> >> > > > >>>>> >> [1] > > > >>>>> >> > > > >>>>> >> > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608 > > > > > < > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608 > > > > > >>>>> >> [2] > https://dist.apache.org/repos/dist/dev/beam/2.3.0/ > > > <https://dist.apache.org/repos/dist/dev/beam/2.3.0/> > > > >>>>> >> [3] > https://dist.apache.org/repos/dist/release/beam/KEYS > > > <https://dist.apache.org/repos/dist/release/beam/KEYS> > > > >>>>> >> [4] > > > >>>>> >> > > > > https://repository.apache.org/content/repositories/orgapachebeam-1028/ > > > < > https://repository.apache.org/content/repositories/orgapachebeam-1028/> > > > >>>>> >> [5] https://github.com/apache/beam/tree/v2.3.0-RC3 > > > <https://github.com/apache/beam/tree/v2.3.0-RC3> > > > >>>>> >> [6] https://github.com/apache/beam-site/pull/381 > > > <https://github.com/apache/beam-site/pull/381> > > > >>>>> >> > > > >>>>> > > > > >>>>> > -- > > > >>>>> > Jean-Baptiste Onofré > > > >>>>> > jbono...@apache.org <mailto:jbono...@apache.org> > > <mailto:jbono...@apache.org <mailto:jbono...@apache.org>> > > > >>>>> > http://blog.nanthrax.net > > > >>>>> > Talend - http://www.talend.com > > > >>>> > > > >>>> > > > >>> > > > > > > > > > > > > > > -- > > Jean-Baptiste Onofré > > jbono...@apache.org <mailto:jbono...@apache.org> > > http://blog.nanthrax.net > > Talend - http://www.talend.com > > > > > > > > -- > > -Ben > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com > -- -Ben
