Hi Eugene, No, I didn't try with those instead I have my custom pipeline where Kafka topic is the source. I have defined a Global Window and processing time trigger to read the data. Further it runs some transformation i.e. GroupByKey and CoGroupByKey. on the windowed collections. I was running the same pipeline on direct runner and spark runner earlier.. Today gave it a try with Flink on Yarn.
Best Regards, Nishu. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Thanks Nishu! Can you clarify which pipeline you were running? > The validation spreadsheet includes 1) the quickstart and 2) mobile game > walkthroughs. Was it one of these, or your custom pipeline? > > On Wed, Nov 22, 2017 at 10:20 AM Nishu <nishuta...@gmail.com> wrote: > > > Hi, > > > > Typo in previous mail. I meant Flink runner. > > > > Thanks, > > Nishu > > On Wed, 22 Nov 2017 at 19.17, > > > > > Hi, > > > > > > I build a pipeline using RC 2.2 today and ran with runner on yarn. > > > It worked seamlessly for unbounded sources. Couldn’t see any issues > with > > > my pipeline so far :) > > > > > > > > > Thanks,Nishu > > > > > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax <re...@google.com.invalid> > > wrote: > > > > > >> Who is validating Flink and Yarn? > > >> > > >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles > <k...@google.com.invalid > > > > > >> wrote: > > >> > > >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov < > > >> > kirpic...@google.com.invalid> wrote: > > >> > > > >> > > In the verification spreadsheet, I'm not sure I understand the > > >> difference > > >> > > between the "YARN" and "Standalone cluster/service". Which is > > >> Dataproc? > > >> > It > > >> > > definitely uses YARN, but it is also a standalone cluster/service. > > >> Does > > >> > it > > >> > > count for both? > > >> > > > > >> > > > >> > No, it doesn't. A number of runners have their own non-YARN cluster > > >> mode. I > > >> > would expect that the launching experience might be different and > the > > >> > portable container management to differ. If they are identical, > > experts > > >> in > > >> > those systems should feel free to coalesce the rows. Conversely, as > > >> other > > >> > platforms become supported, they could be added or not based on > > whether > > >> > they are substantively different from a user experience or QA point > of > > >> > view. > > >> > > > >> > Kenn > > >> > > > >> > > > >> > > Seems now we're missing just Apex and Flink cluster verifications. > > >> > > > > >> > > *though Spark runner took 6x longer to run UserScore, partially I > > >> guess > > >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5 > > >> workers > > >> > > whereas Spark runner used 2 workers). For some reason Spark runner > > >> chose > > >> > > not to split the 10GB input files into chunks. > > >> > > > > >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax > <re...@google.com.invalid > > > > > >> > > wrote: > > >> > > > > >> > > > Done > > >> > > > > > >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw < > > >> > > > rober...@google.com.invalid> wrote: > > >> > > > > > >> > > > > Thanks. You need to re-sign as well. > > >> > > > > > > >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax > > >> > <re...@google.com.invalid > > >> > > > > > >> > > > > wrote: > > >> > > > > > FYI these generated files have been removed from the source > > >> > > > distribution. > > >> > > > > > > > >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax < > re...@google.com > > > > > >> > > wrote: > > >> > > > > > > > >> > > > > >> hmmm, I thought I removed those generated files from the > zip > > >> file > > >> > > > before > > >> > > > > >> sending this email. Let me check again. > > >> > > > > >> > > >> > > > > >> Reuven > > >> > > > > >> > > >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw < > > >> > > > > >> rober...@google.com.invalid> wrote: > > >> > > > > >> > > >> > > > > >>> The source distribution contains a couple of files not on > > >> github > > >> > > > (e.g. > > >> > > > > >>> folders that were added on master, Python generated > files). > > >> The > > >> > pom > > >> > > > > >>> files differed only by missing -SNAPSHOT, other than that > > >> > > presumably > > >> > > > > >>> the source release should just be "wget > > >> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip > "? > > >> > > > > >>> > > >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml > > >> > > > > >>> > > >> > > > > >>> # OK? > > >> > > > > >>> > > >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES > > >> > > > > >>> > > >> > > > > >>> # Expected. > > >> > > > > >>> > > >> > > > > >>> Only in beam/: .git > > >> > > > > >>> Only in beam/: .gitattributes > > >> > > > > >>> Only in beam/: .gitignore > > >> > > > > >>> > > >> > > > > >>> # These folders are probably from switching around between > > >> master > > >> > > and > > >> > > > > >>> git branches. > > >> > > > > >>> > > >> > > > > >>> Only in apache-beam-2.2.0: model > > >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples > > >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner > > >> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: > file-based-io-tests > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > > >> > maven-archetypes/examples/src/ > > >> > > ma > > >> > > > > >>> in/resources/archetype-resources: > > >> > > > > >>> src > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > > >> maven-archetypes/examples- > > >> > > java8/ > > >> > > > > >>> src/main/resources/archetype-resources: > > >> > > > > >>> src > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks > > >> > > > > >>> > > >> > > > > >>> # Here's the generated protos. > > >> > > > > >>> > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_artifact_api_pb2_grpc.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_artifact_api_pb2.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_fn_api_pb2_grpc.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_fn_api_pb2.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_job_api_pb2_grpc.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_job_api_pb2.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_provision_api_pb2_grpc.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_provision_api_pb2.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_runner_api_pb2_grpc.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> beam_runner_api_pb2.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> endpoints_pb2_grpc.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> endpoints_pb2.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> standard_window_fns_pb2_grpc.py > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > >> > apache_beam/portability/api: > > >> > > > > >>> standard_window_fns_pb2.py > > >> > > > > >>> > > >> > > > > >>> And some other sdist generated Python files. > > >> > > > > >>> > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: .eggs > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: LICENSE > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: NOTICE > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: README.md > > >> > > > > >>> > > >> > > > > >>> Presumably we should just purge these files from the rc? > > >> > > > > >>> > > >> > > > > >>> > > >> > > > > >>> FWIW, the Python tarball looks fine. > > >> > > > > >>> > > >> > > > > >>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov > > >> > > > > >>> <kirpic...@google.com.invalid> wrote: > > >> > > > > >>> > How can I specify a dependency on the staged RC? E.g. > I'm > > >> > trying > > >> > > to > > >> > > > > >>> > validate the quickstart per > > >> > > > > >>> > https://beam.apache.org/get-started/quickstart-java/ > and > > >> > > > specifying > > >> > > > > >>> version > > >> > > > > >>> > 2.2.0 doesn't work I suppose because it's not released > > yet. > > >> > > Should > > >> > > > I > > >> > > > > >>> pass > > >> > > > > >>> > some command-line flag to mvn to make it fetch the > version > > >> from > > >> > > the > > >> > > > > >>> staging > > >> > > > > >>> > area? > > >> > > > > >>> > > > >> > > > > >>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik > > >> > > > <lc...@google.com.invalid > > >> > > > > > > > >> > > > > >>> > wrote: > > >> > > > > >>> > > > >> > > > > >>> >> Its open to all, its just that there are binding votes > > and > > >> > > > > non-binding > > >> > > > > >>> >> votes. > > >> > > > > >>> >> > > >> > > > > >>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev < > > >> > > > > >>> >> valen...@google.com.invalid> wrote: > > >> > > > > >>> >> > > >> > > > > >>> >> > I have a process question: is the vote open for > > >> committers > > >> > > only > > >> > > > or > > >> > > > > >>> for > > >> > > > > >>> >> all > > >> > > > > >>> >> > contributors? > > >> > > > > >>> >> > > > >> > > > > >>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik > > >> > > > > >>> <lc...@google.com.invalid> > > >> > > > > >>> >> > wrote: > > >> > > > > >>> >> > > > >> > > > > >>> >> > > +1, Approve the release > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > I have verified the wordcount quickstart on the > > Apache > > >> > Beam > > >> > > > > website > > >> > > > > >>> >> using > > >> > > > > >>> >> > > Apex, DirectRunner, Flink & Spark on Linux. > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > The Gearpump runner is yet to have a quickstart > > listed > > >> on > > >> > > our > > >> > > > > >>> website. > > >> > > > > >>> >> > > Adding the quickstart is already represented by > this > > >> > > existing > > >> > > > > >>> issue: > > >> > > > > >>> >> > > https://issues.apache.org/jira/browse/BEAM-2692 > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn > > Tymofieiev < > > >> > > > > >>> >> > > valen...@google.com.invalid> wrote: > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > I have verified: SHA & MD5 signatures of Python > > >> > artifacts > > >> > > in > > >> > > > > >>> [2], and > > >> > > > > >>> >> > > > checked Python side of the validation checklist > on > > >> > Linux. > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > There is one known issue in UserScore example for > > >> > Dataflow > > >> > > > > >>> runner. > > >> > > > > >>> >> The > > >> > > > > >>> >> > > > issue has been fixed on master branch and does > not > > >> > > require a > > >> > > > > >>> >> > cherry-pick > > >> > > > > >>> >> > > at > > >> > > > > >>> >> > > > this point. A workaround is to pass > > >> --save_main_session > > >> > > > > pipeline > > >> > > > > >>> >> > option. > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > RC4 looks good to me so far. > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles > > >> > > > > >>> >> > <k...@google.com.invalid > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > wrote: > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > Hi all, > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > Following up on past discussions and > > >> > > > > >>> >> > > > > https://issues.apache.org/ > jira/browse/BEAM-1189 > > I > > >> > have > > >> > > > > >>> prepared a > > >> > > > > >>> >> > > > > spreadsheet so we can sign up for validation > > steps > > >> > that > > >> > > > > must be > > >> > > > > >>> >> done > > >> > > > > >>> >> > > by a > > >> > > > > >>> >> > > > > human. > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > The spreadsheet for 2.2.0 is at > > >> > > > > >>> >> > > > > > > https://s.apache.org/beam-2.2.0-release-validation > > >> . > > >> > > > > Everyone > > >> > > > > >>> can > > >> > > > > >>> >> > edit, > > >> > > > > >>> >> > > > so > > >> > > > > >>> >> > > > > it is easy to sign up or add new rows. > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > FYI the template is at > > >> > > > > >>> >> https://s.apache.org/beam-release-validation. > > >> > > > > >>> >> > > > After > > >> > > > > >>> >> > > > > this release, I will update the rows in the > > >> template > > >> > to > > >> > > > > match > > >> > > > > >>> >> > whatever > > >> > > > > >>> >> > > we > > >> > > > > >>> >> > > > > have added to the validation for 2.2.0. > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > Kenn > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax > > >> > > > > >>> >> > <re...@google.com.invalid > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > wrote: > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > Hi everyone, > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > Please review and vote on the release > candidate > > >> #4 > > >> > for > > >> > > > the > > >> > > > > >>> >> version > > >> > > > > >>> >> > > > 2.2.0, > > >> > > > > >>> >> > > > > > as follows: > > >> > > > > >>> >> > > > > > [ ] +1, Approve the release > > >> > > > > >>> >> > > > > > [ ] -1, Do not approve the release (please > > >> provide > > >> > > > > specific > > >> > > > > >>> >> > > comments) > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > The complete staging area is available for > your > > >> > > review, > > >> > > > > which > > >> > > > > >>> >> > > includes: > > >> > > > > >>> >> > > > > > * JIRA release notes [1], > > >> > > > > >>> >> > > > > > * the official Apache source release to be > > >> > deployed > > >> > > to > > >> > > > > >>> >> > > > dist.apache.org > > >> > > > > >>> >> > > > > > [2], > > >> > > > > >>> >> > > > > > which is signed with the key with fingerprint > > >> > B98B7708 > > >> > > > > [3], > > >> > > > > >>> >> > > > > > * all artifacts to be deployed to the Maven > > >> > Central > > >> > > > > >>> Repository > > >> > > > > >>> >> > [4], > > >> > > > > >>> >> > > > > > * source code tag "v2.2.0-RC4" [5], > > >> > > > > >>> >> > > > > > * website pull request listing the release > > and > > >> > > > > publishing > > >> > > > > >>> the > > >> > > > > >>> >> API > > >> > > > > >>> >> > > > > > reference manual [6]. > > >> > > > > >>> >> > > > > > * Java artifacts were built with Maven > 3.5.0 > > >> and > > >> > > > > >>> OpenJDK/Oracle > > >> > > > > >>> >> > JDK > > >> > > > > >>> >> > > > > > 1.8.0_144. > > >> > > > > >>> >> > > > > > * Python artifacts are deployed along with > > the > > >> > > source > > >> > > > > >>> release > > >> > > > > >>> >> to > > >> > > > > >>> >> > > the > > >> > > > > >>> >> > > > > > dist.apache.org [2]. > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > The vote will be open for at least 72 hours. > It > > >> is > > >> > > > > adopted by > > >> > > > > >>> >> > > majority > > >> > > > > >>> >> > > > > > approval, with at least 3 PMC affirmative > > votes. > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > Thanks, > > >> > > > > >>> >> > > > > > Reuven > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > [1] https://issues.apache.org/ > > >> > > > > jira/secure/ReleaseNote.jspa?p > > >> > > > > >>> >> > > > > > rojectId=12319527&version=12341044 > > >> > > > > >>> >> > > > > > [2] https://dist.apache.org/repos/ > > >> > > dist/dev/beam/2.2.0/ > > >> > > > > >>> >> > > > > > [3] > > >> > > > https://dist.apache.org/repos/dist/release/beam/KEYS > > >> > > > > >>> >> > > > > > [4] https://repository.apache.org/ > > >> > > content/repositories/ > > >> > > > > >>> >> > > > > orgapachebeam-1025/ > > >> > > > > >>> >> > > > > > [5] https://github.com/apache/ > > >> beam/tree/v2.2.0-RC4 > > >> > > > > >>> >> > > > > > < > > https://github.com/apache/beam/tree/v2.2.0-RC4> > > >> > > > > >>> >> > > > > > [6] > > https://github.com/apache/beam-site/pull/337 > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > >> > > > > >>> >> > > >> > > > > >>> > > >> > > > > >> > > >> > > > > >> > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > -- > > > Thanks & Regards, > > > Nishu Tayal > > > > > > > < > > https://www.avast.com/sig-email?utm_medium=email&utm_ > source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon > > > > > Virus-free. > > www.avast.com > > < > > https://www.avast.com/sig-email?utm_medium=email&utm_ > source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link > > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > > -- Thanks & Regards, Nishu Tayal
