Hi, Typo in previous mail. I meant Flink runner.
Thanks, Nishu On Wed, 22 Nov 2017 at 19.17, > Hi, > > I build a pipeline using RC 2.2 today and ran with runner on yarn. > It worked seamlessly for unbounded sources. Couldn’t see any issues with > my pipeline so far :) > > > Thanks,Nishu > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax <re...@google.com.invalid> wrote: > >> Who is validating Flink and Yarn? >> >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles <k...@google.com.invalid> >> wrote: >> >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov < >> > kirpic...@google.com.invalid> wrote: >> > >> > > In the verification spreadsheet, I'm not sure I understand the >> difference >> > > between the "YARN" and "Standalone cluster/service". Which is >> Dataproc? >> > It >> > > definitely uses YARN, but it is also a standalone cluster/service. >> Does >> > it >> > > count for both? >> > > >> > >> > No, it doesn't. A number of runners have their own non-YARN cluster >> mode. I >> > would expect that the launching experience might be different and the >> > portable container management to differ. If they are identical, experts >> in >> > those systems should feel free to coalesce the rows. Conversely, as >> other >> > platforms become supported, they could be added or not based on whether >> > they are substantively different from a user experience or QA point of >> > view. >> > >> > Kenn >> > >> > >> > > Seems now we're missing just Apex and Flink cluster verifications. >> > > >> > > *though Spark runner took 6x longer to run UserScore, partially I >> guess >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5 >> workers >> > > whereas Spark runner used 2 workers). For some reason Spark runner >> chose >> > > not to split the 10GB input files into chunks. >> > > >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax <re...@google.com.invalid> >> > > wrote: >> > > >> > > > Done >> > > > >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw < >> > > > rober...@google.com.invalid> wrote: >> > > > >> > > > > Thanks. You need to re-sign as well. >> > > > > >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax >> > <re...@google.com.invalid >> > > > >> > > > > wrote: >> > > > > > FYI these generated files have been removed from the source >> > > > distribution. >> > > > > > >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax <re...@google.com> >> > > wrote: >> > > > > > >> > > > > >> hmmm, I thought I removed those generated files from the zip >> file >> > > > before >> > > > > >> sending this email. Let me check again. >> > > > > >> >> > > > > >> Reuven >> > > > > >> >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw < >> > > > > >> rober...@google.com.invalid> wrote: >> > > > > >> >> > > > > >>> The source distribution contains a couple of files not on >> github >> > > > (e.g. >> > > > > >>> folders that were added on master, Python generated files). >> The >> > pom >> > > > > >>> files differed only by missing -SNAPSHOT, other than that >> > > presumably >> > > > > >>> the source release should just be "wget >> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip";? >> > > > > >>> >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml >> > > > > >>> >> > > > > >>> # OK? >> > > > > >>> >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES >> > > > > >>> >> > > > > >>> # Expected. >> > > > > >>> >> > > > > >>> Only in beam/: .git >> > > > > >>> Only in beam/: .gitattributes >> > > > > >>> Only in beam/: .gitignore >> > > > > >>> >> > > > > >>> # These folders are probably from switching around between >> master >> > > and >> > > > > >>> git branches. >> > > > > >>> >> > > > > >>> Only in apache-beam-2.2.0: model >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner >> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ >> > maven-archetypes/examples/src/ >> > > ma >> > > > > >>> in/resources/archetype-resources: >> > > > > >>> src >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ >> maven-archetypes/examples- >> > > java8/ >> > > > > >>> src/main/resources/archetype-resources: >> > > > > >>> src >> > > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks >> > > > > >>> >> > > > > >>> # Here's the generated protos. >> > > > > >>> >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_artifact_api_pb2_grpc.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_artifact_api_pb2.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_fn_api_pb2_grpc.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_fn_api_pb2.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_job_api_pb2_grpc.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_job_api_pb2.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_provision_api_pb2_grpc.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_provision_api_pb2.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_runner_api_pb2_grpc.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> beam_runner_api_pb2.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> endpoints_pb2_grpc.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> endpoints_pb2.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> standard_window_fns_pb2_grpc.py >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ >> > apache_beam/portability/api: >> > > > > >>> standard_window_fns_pb2.py >> > > > > >>> >> > > > > >>> And some other sdist generated Python files. >> > > > > >>> >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: .eggs >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: LICENSE >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: NOTICE >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: README.md >> > > > > >>> >> > > > > >>> Presumably we should just purge these files from the rc? >> > > > > >>> >> > > > > >>> >> > > > > >>> FWIW, the Python tarball looks fine. >> > > > > >>> >> > > > > >>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov >> > > > > >>> <kirpic...@google.com.invalid> wrote: >> > > > > >>> > How can I specify a dependency on the staged RC? E.g. I'm >> > trying >> > > to >> > > > > >>> > validate the quickstart per >> > > > > >>> > https://beam.apache.org/get-started/quickstart-java/ and >> > > > specifying >> > > > > >>> version >> > > > > >>> > 2.2.0 doesn't work I suppose because it's not released yet. >> > > Should >> > > > I >> > > > > >>> pass >> > > > > >>> > some command-line flag to mvn to make it fetch the version >> from >> > > the >> > > > > >>> staging >> > > > > >>> > area? >> > > > > >>> > >> > > > > >>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik >> > > > <lc...@google.com.invalid >> > > > > > >> > > > > >>> > wrote: >> > > > > >>> > >> > > > > >>> >> Its open to all, its just that there are binding votes and >> > > > > non-binding >> > > > > >>> >> votes. >> > > > > >>> >> >> > > > > >>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev < >> > > > > >>> >> valen...@google.com.invalid> wrote: >> > > > > >>> >> >> > > > > >>> >> > I have a process question: is the vote open for >> committers >> > > only >> > > > or >> > > > > >>> for >> > > > > >>> >> all >> > > > > >>> >> > contributors? >> > > > > >>> >> > >> > > > > >>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik >> > > > > >>> <lc...@google.com.invalid> >> > > > > >>> >> > wrote: >> > > > > >>> >> > >> > > > > >>> >> > > +1, Approve the release >> > > > > >>> >> > > >> > > > > >>> >> > > I have verified the wordcount quickstart on the Apache >> > Beam >> > > > > website >> > > > > >>> >> using >> > > > > >>> >> > > Apex, DirectRunner, Flink & Spark on Linux. >> > > > > >>> >> > > >> > > > > >>> >> > > The Gearpump runner is yet to have a quickstart listed >> on >> > > our >> > > > > >>> website. >> > > > > >>> >> > > Adding the quickstart is already represented by this >> > > existing >> > > > > >>> issue: >> > > > > >>> >> > > https://issues.apache.org/jira/browse/BEAM-2692 >> > > > > >>> >> > > >> > > > > >>> >> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev < >> > > > > >>> >> > > valen...@google.com.invalid> wrote: >> > > > > >>> >> > > >> > > > > >>> >> > > > I have verified: SHA & MD5 signatures of Python >> > artifacts >> > > in >> > > > > >>> [2], and >> > > > > >>> >> > > > checked Python side of the validation checklist on >> > Linux. >> > > > > >>> >> > > > >> > > > > >>> >> > > > There is one known issue in UserScore example for >> > Dataflow >> > > > > >>> runner. >> > > > > >>> >> The >> > > > > >>> >> > > > issue has been fixed on master branch and does not >> > > require a >> > > > > >>> >> > cherry-pick >> > > > > >>> >> > > at >> > > > > >>> >> > > > this point. A workaround is to pass >> --save_main_session >> > > > > pipeline >> > > > > >>> >> > option. >> > > > > >>> >> > > > >> > > > > >>> >> > > > RC4 looks good to me so far. >> > > > > >>> >> > > > >> > > > > >>> >> > > > >> > > > > >>> >> > > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles >> > > > > >>> >> > <k...@google.com.invalid >> > > > > >>> >> > > > >> > > > > >>> >> > > > wrote: >> > > > > >>> >> > > > >> > > > > >>> >> > > > > Hi all, >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > Following up on past discussions and >> > > > > >>> >> > > > > https://issues.apache.org/jira/browse/BEAM-1189 I >> > have >> > > > > >>> prepared a >> > > > > >>> >> > > > > spreadsheet so we can sign up for validation steps >> > that >> > > > > must be >> > > > > >>> >> done >> > > > > >>> >> > > by a >> > > > > >>> >> > > > > human. >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > The spreadsheet for 2.2.0 is at >> > > > > >>> >> > > > > https://s.apache.org/beam-2.2.0-release-validation >> . >> > > > > Everyone >> > > > > >>> can >> > > > > >>> >> > edit, >> > > > > >>> >> > > > so >> > > > > >>> >> > > > > it is easy to sign up or add new rows. >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > FYI the template is at >> > > > > >>> >> https://s.apache.org/beam-release-validation. >> > > > > >>> >> > > > After >> > > > > >>> >> > > > > this release, I will update the rows in the >> template >> > to >> > > > > match >> > > > > >>> >> > whatever >> > > > > >>> >> > > we >> > > > > >>> >> > > > > have added to the validation for 2.2.0. >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > Kenn >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax >> > > > > >>> >> > <re...@google.com.invalid >> > > > > >>> >> > > > >> > > > > >>> >> > > > > wrote: >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > > Hi everyone, >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > Please review and vote on the release candidate >> #4 >> > for >> > > > the >> > > > > >>> >> version >> > > > > >>> >> > > > 2.2.0, >> > > > > >>> >> > > > > > as follows: >> > > > > >>> >> > > > > > [ ] +1, Approve the release >> > > > > >>> >> > > > > > [ ] -1, Do not approve the release (please >> provide >> > > > > specific >> > > > > >>> >> > > comments) >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > The complete staging area is available for your >> > > review, >> > > > > which >> > > > > >>> >> > > includes: >> > > > > >>> >> > > > > > * JIRA release notes [1], >> > > > > >>> >> > > > > > * the official Apache source release to be >> > deployed >> > > to >> > > > > >>> >> > > > dist.apache.org >> > > > > >>> >> > > > > > [2], >> > > > > >>> >> > > > > > which is signed with the key with fingerprint >> > B98B7708 >> > > > > [3], >> > > > > >>> >> > > > > > * all artifacts to be deployed to the Maven >> > Central >> > > > > >>> Repository >> > > > > >>> >> > [4], >> > > > > >>> >> > > > > > * source code tag "v2.2.0-RC4" [5], >> > > > > >>> >> > > > > > * website pull request listing the release and >> > > > > publishing >> > > > > >>> the >> > > > > >>> >> API >> > > > > >>> >> > > > > > reference manual [6]. >> > > > > >>> >> > > > > > * Java artifacts were built with Maven 3.5.0 >> and >> > > > > >>> OpenJDK/Oracle >> > > > > >>> >> > JDK >> > > > > >>> >> > > > > > 1.8.0_144. >> > > > > >>> >> > > > > > * Python artifacts are deployed along with the >> > > source >> > > > > >>> release >> > > > > >>> >> to >> > > > > >>> >> > > the >> > > > > >>> >> > > > > > dist.apache.org [2]. >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > The vote will be open for at least 72 hours. It >> is >> > > > > adopted by >> > > > > >>> >> > > majority >> > > > > >>> >> > > > > > approval, with at least 3 PMC affirmative votes. >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > Thanks, >> > > > > >>> >> > > > > > Reuven >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > [1] https://issues.apache.org/ >> > > > > jira/secure/ReleaseNote.jspa?p >> > > > > >>> >> > > > > > rojectId=12319527&version=12341044 >> > > > > >>> >> > > > > > [2] https://dist.apache.org/repos/ >> > > dist/dev/beam/2.2.0/ >> > > > > >>> >> > > > > > [3] >> > > > https://dist.apache.org/repos/dist/release/beam/KEYS >> > > > > >>> >> > > > > > [4] https://repository.apache.org/ >> > > content/repositories/ >> > > > > >>> >> > > > > orgapachebeam-1025/ >> > > > > >>> >> > > > > > [5] https://github.com/apache/ >> beam/tree/v2.2.0-RC4 >> > > > > >>> >> > > > > > <https://github.com/apache/beam/tree/v2.2.0-RC4> >> > > > > >>> >> > > > > > [6] https://github.com/apache/beam-site/pull/337 >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > >> > > > > >>> >> > > >> > > > > >>> >> > >> > > > > >>> >> >> > > > > >>> >> > > > > >> >> > > > > >> >> > > > > >> > > > >> > > >> > >> > -- > Thanks & Regards, > Nishu Tayal > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
