Thanks Nishu! Can you clarify which pipeline you were running? The validation spreadsheet includes 1) the quickstart and 2) mobile game walkthroughs. Was it one of these, or your custom pipeline?
On Wed, Nov 22, 2017 at 10:20 AM Nishu <nishuta...@gmail.com> wrote: > Hi, > > Typo in previous mail. I meant Flink runner. > > Thanks, > Nishu > On Wed, 22 Nov 2017 at 19.17, > > > Hi, > > > > I build a pipeline using RC 2.2 today and ran with runner on yarn. > > It worked seamlessly for unbounded sources. Couldn’t see any issues with > > my pipeline so far :) > > > > > > Thanks,Nishu > > > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax <re...@google.com.invalid> > wrote: > > > >> Who is validating Flink and Yarn? > >> > >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles <k...@google.com.invalid > > > >> wrote: > >> > >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov < > >> > kirpic...@google.com.invalid> wrote: > >> > > >> > > In the verification spreadsheet, I'm not sure I understand the > >> difference > >> > > between the "YARN" and "Standalone cluster/service". Which is > >> Dataproc? > >> > It > >> > > definitely uses YARN, but it is also a standalone cluster/service. > >> Does > >> > it > >> > > count for both? > >> > > > >> > > >> > No, it doesn't. A number of runners have their own non-YARN cluster > >> mode. I > >> > would expect that the launching experience might be different and the > >> > portable container management to differ. If they are identical, > experts > >> in > >> > those systems should feel free to coalesce the rows. Conversely, as > >> other > >> > platforms become supported, they could be added or not based on > whether > >> > they are substantively different from a user experience or QA point of > >> > view. > >> > > >> > Kenn > >> > > >> > > >> > > Seems now we're missing just Apex and Flink cluster verifications. > >> > > > >> > > *though Spark runner took 6x longer to run UserScore, partially I > >> guess > >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5 > >> workers > >> > > whereas Spark runner used 2 workers). For some reason Spark runner > >> chose > >> > > not to split the 10GB input files into chunks. > >> > > > >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax <re...@google.com.invalid > > > >> > > wrote: > >> > > > >> > > > Done > >> > > > > >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw < > >> > > > rober...@google.com.invalid> wrote: > >> > > > > >> > > > > Thanks. You need to re-sign as well. > >> > > > > > >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax > >> > <re...@google.com.invalid > >> > > > > >> > > > > wrote: > >> > > > > > FYI these generated files have been removed from the source > >> > > > distribution. > >> > > > > > > >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax <re...@google.com > > > >> > > wrote: > >> > > > > > > >> > > > > >> hmmm, I thought I removed those generated files from the zip > >> file > >> > > > before > >> > > > > >> sending this email. Let me check again. > >> > > > > >> > >> > > > > >> Reuven > >> > > > > >> > >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw < > >> > > > > >> rober...@google.com.invalid> wrote: > >> > > > > >> > >> > > > > >>> The source distribution contains a couple of files not on > >> github > >> > > > (e.g. > >> > > > > >>> folders that were added on master, Python generated files). > >> The > >> > pom > >> > > > > >>> files differed only by missing -SNAPSHOT, other than that > >> > > presumably > >> > > > > >>> the source release should just be "wget > >> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip";? > >> > > > > >>> > >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml > >> > > > > >>> > >> > > > > >>> # OK? > >> > > > > >>> > >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES > >> > > > > >>> > >> > > > > >>> # Expected. > >> > > > > >>> > >> > > > > >>> Only in beam/: .git > >> > > > > >>> Only in beam/: .gitattributes > >> > > > > >>> Only in beam/: .gitignore > >> > > > > >>> > >> > > > > >>> # These folders are probably from switching around between > >> master > >> > > and > >> > > > > >>> git branches. > >> > > > > >>> > >> > > > > >>> Only in apache-beam-2.2.0: model > >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples > >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner > >> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > >> > maven-archetypes/examples/src/ > >> > > ma > >> > > > > >>> in/resources/archetype-resources: > >> > > > > >>> src > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > >> maven-archetypes/examples- > >> > > java8/ > >> > > > > >>> src/main/resources/archetype-resources: > >> > > > > >>> src > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks > >> > > > > >>> > >> > > > > >>> # Here's the generated protos. > >> > > > > >>> > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_artifact_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_artifact_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_fn_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_fn_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_job_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_job_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_provision_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_provision_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_runner_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_runner_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> endpoints_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> endpoints_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> standard_window_fns_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> standard_window_fns_pb2.py > >> > > > > >>> > >> > > > > >>> And some other sdist generated Python files. > >> > > > > >>> > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: .eggs > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: LICENSE > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: NOTICE > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: README.md > >> > > > > >>> > >> > > > > >>> Presumably we should just purge these files from the rc? > >> > > > > >>> > >> > > > > >>> > >> > > > > >>> FWIW, the Python tarball looks fine. > >> > > > > >>> > >> > > > > >>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov > >> > > > > >>> <kirpic...@google.com.invalid> wrote: > >> > > > > >>> > How can I specify a dependency on the staged RC? E.g. I'm > >> > trying > >> > > to > >> > > > > >>> > validate the quickstart per > >> > > > > >>> > https://beam.apache.org/get-started/quickstart-java/ and > >> > > > specifying > >> > > > > >>> version > >> > > > > >>> > 2.2.0 doesn't work I suppose because it's not released > yet. > >> > > Should > >> > > > I > >> > > > > >>> pass > >> > > > > >>> > some command-line flag to mvn to make it fetch the version > >> from > >> > > the > >> > > > > >>> staging > >> > > > > >>> > area? > >> > > > > >>> > > >> > > > > >>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik > >> > > > <lc...@google.com.invalid > >> > > > > > > >> > > > > >>> > wrote: > >> > > > > >>> > > >> > > > > >>> >> Its open to all, its just that there are binding votes > and > >> > > > > non-binding > >> > > > > >>> >> votes. > >> > > > > >>> >> > >> > > > > >>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev < > >> > > > > >>> >> valen...@google.com.invalid> wrote: > >> > > > > >>> >> > >> > > > > >>> >> > I have a process question: is the vote open for > >> committers > >> > > only > >> > > > or > >> > > > > >>> for > >> > > > > >>> >> all > >> > > > > >>> >> > contributors? > >> > > > > >>> >> > > >> > > > > >>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik > >> > > > > >>> <lc...@google.com.invalid> > >> > > > > >>> >> > wrote: > >> > > > > >>> >> > > >> > > > > >>> >> > > +1, Approve the release > >> > > > > >>> >> > > > >> > > > > >>> >> > > I have verified the wordcount quickstart on the > Apache > >> > Beam > >> > > > > website > >> > > > > >>> >> using > >> > > > > >>> >> > > Apex, DirectRunner, Flink & Spark on Linux. > >> > > > > >>> >> > > > >> > > > > >>> >> > > The Gearpump runner is yet to have a quickstart > listed > >> on > >> > > our > >> > > > > >>> website. > >> > > > > >>> >> > > Adding the quickstart is already represented by this > >> > > existing > >> > > > > >>> issue: > >> > > > > >>> >> > > https://issues.apache.org/jira/browse/BEAM-2692 > >> > > > > >>> >> > > > >> > > > > >>> >> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn > Tymofieiev < > >> > > > > >>> >> > > valen...@google.com.invalid> wrote: > >> > > > > >>> >> > > > >> > > > > >>> >> > > > I have verified: SHA & MD5 signatures of Python > >> > artifacts > >> > > in > >> > > > > >>> [2], and > >> > > > > >>> >> > > > checked Python side of the validation checklist on > >> > Linux. > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > There is one known issue in UserScore example for > >> > Dataflow > >> > > > > >>> runner. > >> > > > > >>> >> The > >> > > > > >>> >> > > > issue has been fixed on master branch and does not > >> > > require a > >> > > > > >>> >> > cherry-pick > >> > > > > >>> >> > > at > >> > > > > >>> >> > > > this point. A workaround is to pass > >> --save_main_session > >> > > > > pipeline > >> > > > > >>> >> > option. > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > RC4 looks good to me so far. > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles > >> > > > > >>> >> > <k...@google.com.invalid > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > wrote: > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > Hi all, > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > Following up on past discussions and > >> > > > > >>> >> > > > > https://issues.apache.org/jira/browse/BEAM-1189 > I > >> > have > >> > > > > >>> prepared a > >> > > > > >>> >> > > > > spreadsheet so we can sign up for validation > steps > >> > that > >> > > > > must be > >> > > > > >>> >> done > >> > > > > >>> >> > > by a > >> > > > > >>> >> > > > > human. > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > The spreadsheet for 2.2.0 is at > >> > > > > >>> >> > > > > > https://s.apache.org/beam-2.2.0-release-validation > >> . > >> > > > > Everyone > >> > > > > >>> can > >> > > > > >>> >> > edit, > >> > > > > >>> >> > > > so > >> > > > > >>> >> > > > > it is easy to sign up or add new rows. > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > FYI the template is at > >> > > > > >>> >> https://s.apache.org/beam-release-validation. > >> > > > > >>> >> > > > After > >> > > > > >>> >> > > > > this release, I will update the rows in the > >> template > >> > to > >> > > > > match > >> > > > > >>> >> > whatever > >> > > > > >>> >> > > we > >> > > > > >>> >> > > > > have added to the validation for 2.2.0. > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > Kenn > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax > >> > > > > >>> >> > <re...@google.com.invalid > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > wrote: > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > Hi everyone, > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > Please review and vote on the release candidate > >> #4 > >> > for > >> > > > the > >> > > > > >>> >> version > >> > > > > >>> >> > > > 2.2.0, > >> > > > > >>> >> > > > > > as follows: > >> > > > > >>> >> > > > > > [ ] +1, Approve the release > >> > > > > >>> >> > > > > > [ ] -1, Do not approve the release (please > >> provide > >> > > > > specific > >> > > > > >>> >> > > comments) > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > The complete staging area is available for your > >> > > review, > >> > > > > which > >> > > > > >>> >> > > includes: > >> > > > > >>> >> > > > > > * JIRA release notes [1], > >> > > > > >>> >> > > > > > * the official Apache source release to be > >> > deployed > >> > > to > >> > > > > >>> >> > > > dist.apache.org > >> > > > > >>> >> > > > > > [2], > >> > > > > >>> >> > > > > > which is signed with the key with fingerprint > >> > B98B7708 > >> > > > > [3], > >> > > > > >>> >> > > > > > * all artifacts to be deployed to the Maven > >> > Central > >> > > > > >>> Repository > >> > > > > >>> >> > [4], > >> > > > > >>> >> > > > > > * source code tag "v2.2.0-RC4" [5], > >> > > > > >>> >> > > > > > * website pull request listing the release > and > >> > > > > publishing > >> > > > > >>> the > >> > > > > >>> >> API > >> > > > > >>> >> > > > > > reference manual [6]. > >> > > > > >>> >> > > > > > * Java artifacts were built with Maven 3.5.0 > >> and > >> > > > > >>> OpenJDK/Oracle > >> > > > > >>> >> > JDK > >> > > > > >>> >> > > > > > 1.8.0_144. > >> > > > > >>> >> > > > > > * Python artifacts are deployed along with > the > >> > > source > >> > > > > >>> release > >> > > > > >>> >> to > >> > > > > >>> >> > > the > >> > > > > >>> >> > > > > > dist.apache.org [2]. > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > The vote will be open for at least 72 hours. It > >> is > >> > > > > adopted by > >> > > > > >>> >> > > majority > >> > > > > >>> >> > > > > > approval, with at least 3 PMC affirmative > votes. > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > Thanks, > >> > > > > >>> >> > > > > > Reuven > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > [1] https://issues.apache.org/ > >> > > > > jira/secure/ReleaseNote.jspa?p > >> > > > > >>> >> > > > > > rojectId=12319527&version=12341044 > >> > > > > >>> >> > > > > > [2] https://dist.apache.org/repos/ > >> > > dist/dev/beam/2.2.0/ > >> > > > > >>> >> > > > > > [3] > >> > > > https://dist.apache.org/repos/dist/release/beam/KEYS > >> > > > > >>> >> > > > > > [4] https://repository.apache.org/ > >> > > content/repositories/ > >> > > > > >>> >> > > > > orgapachebeam-1025/ > >> > > > > >>> >> > > > > > [5] https://github.com/apache/ > >> beam/tree/v2.2.0-RC4 > >> > > > > >>> >> > > > > > < > https://github.com/apache/beam/tree/v2.2.0-RC4> > >> > > > > >>> >> > > > > > [6] > https://github.com/apache/beam-site/pull/337 > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > >> > > > > >>> >> > > >> > > > > >>> >> > >> > > > > >>> > >> > > > > >> > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > > -- > > Thanks & Regards, > > Nishu Tayal > > > > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon > > > Virus-free. > www.avast.com > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >
