+1 (non-binding) Since Beam 2.2 is blocking release of tensorflow.transform <https://github.com/tensorflow/transform> 0.4.
On Wed, Nov 22, 2017 at 10:19 AM, Nishu <nishuta...@gmail.com> wrote: > Hi, > > Typo in previous mail. I meant Flink runner. > > Thanks, > Nishu > On Wed, 22 Nov 2017 at 19.17, > > > Hi, > > > > I build a pipeline using RC 2.2 today and ran with runner on yarn. > > It worked seamlessly for unbounded sources. Couldn’t see any issues with > > my pipeline so far :) > > > > > > Thanks,Nishu > > > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax <re...@google.com.invalid> > wrote: > > > >> Who is validating Flink and Yarn? > >> > >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles <k...@google.com.invalid > > > >> wrote: > >> > >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov < > >> > kirpic...@google.com.invalid> wrote: > >> > > >> > > In the verification spreadsheet, I'm not sure I understand the > >> difference > >> > > between the "YARN" and "Standalone cluster/service". Which is > >> Dataproc? > >> > It > >> > > definitely uses YARN, but it is also a standalone cluster/service. > >> Does > >> > it > >> > > count for both? > >> > > > >> > > >> > No, it doesn't. A number of runners have their own non-YARN cluster > >> mode. I > >> > would expect that the launching experience might be different and the > >> > portable container management to differ. If they are identical, > experts > >> in > >> > those systems should feel free to coalesce the rows. Conversely, as > >> other > >> > platforms become supported, they could be added or not based on > whether > >> > they are substantively different from a user experience or QA point of > >> > view. > >> > > >> > Kenn > >> > > >> > > >> > > Seems now we're missing just Apex and Flink cluster verifications. > >> > > > >> > > *though Spark runner took 6x longer to run UserScore, partially I > >> guess > >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5 > >> workers > >> > > whereas Spark runner used 2 workers). For some reason Spark runner > >> chose > >> > > not to split the 10GB input files into chunks. > >> > > > >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax <re...@google.com.invalid > > > >> > > wrote: > >> > > > >> > > > Done > >> > > > > >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw < > >> > > > rober...@google.com.invalid> wrote: > >> > > > > >> > > > > Thanks. You need to re-sign as well. > >> > > > > > >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax > >> > <re...@google.com.invalid > >> > > > > >> > > > > wrote: > >> > > > > > FYI these generated files have been removed from the source > >> > > > distribution. > >> > > > > > > >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax <re...@google.com > > > >> > > wrote: > >> > > > > > > >> > > > > >> hmmm, I thought I removed those generated files from the zip > >> file > >> > > > before > >> > > > > >> sending this email. Let me check again. > >> > > > > >> > >> > > > > >> Reuven > >> > > > > >> > >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw < > >> > > > > >> rober...@google.com.invalid> wrote: > >> > > > > >> > >> > > > > >>> The source distribution contains a couple of files not on > >> github > >> > > > (e.g. > >> > > > > >>> folders that were added on master, Python generated files). > >> The > >> > pom > >> > > > > >>> files differed only by missing -SNAPSHOT, other than that > >> > > presumably > >> > > > > >>> the source release should just be "wget > >> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip";? > >> > > > > >>> > >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml > >> > > > > >>> > >> > > > > >>> # OK? > >> > > > > >>> > >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES > >> > > > > >>> > >> > > > > >>> # Expected. > >> > > > > >>> > >> > > > > >>> Only in beam/: .git > >> > > > > >>> Only in beam/: .gitattributes > >> > > > > >>> Only in beam/: .gitignore > >> > > > > >>> > >> > > > > >>> # These folders are probably from switching around between > >> master > >> > > and > >> > > > > >>> git branches. > >> > > > > >>> > >> > > > > >>> Only in apache-beam-2.2.0: model > >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples > >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner > >> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > >> > maven-archetypes/examples/src/ > >> > > ma > >> > > > > >>> in/resources/archetype-resources: > >> > > > > >>> src > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > >> maven-archetypes/examples- > >> > > java8/ > >> > > > > >>> src/main/resources/archetype-resources: > >> > > > > >>> src > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks > >> > > > > >>> > >> > > > > >>> # Here's the generated protos. > >> > > > > >>> > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_artifact_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_artifact_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_fn_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_fn_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_job_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_job_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_provision_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_provision_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_runner_api_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> beam_runner_api_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> endpoints_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> endpoints_pb2.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> standard_window_fns_pb2_grpc.py > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > >> > apache_beam/portability/api: > >> > > > > >>> standard_window_fns_pb2.py > >> > > > > >>> > >> > > > > >>> And some other sdist generated Python files. > >> > > > > >>> > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: .eggs > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: LICENSE > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: NOTICE > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: README.md > >> > > > > >>> > >> > > > > >>> Presumably we should just purge these files from the rc? > >> > > > > >>> > >> > > > > >>> > >> > > > > >>> FWIW, the Python tarball looks fine. > >> > > > > >>> > >> > > > > >>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov > >> > > > > >>> <kirpic...@google.com.invalid> wrote: > >> > > > > >>> > How can I specify a dependency on the staged RC? E.g. I'm > >> > trying > >> > > to > >> > > > > >>> > validate the quickstart per > >> > > > > >>> > https://beam.apache.org/get-started/quickstart-java/ and > >> > > > specifying > >> > > > > >>> version > >> > > > > >>> > 2.2.0 doesn't work I suppose because it's not released > yet. > >> > > Should > >> > > > I > >> > > > > >>> pass > >> > > > > >>> > some command-line flag to mvn to make it fetch the version > >> from > >> > > the > >> > > > > >>> staging > >> > > > > >>> > area? > >> > > > > >>> > > >> > > > > >>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik > >> > > > <lc...@google.com.invalid > >> > > > > > > >> > > > > >>> > wrote: > >> > > > > >>> > > >> > > > > >>> >> Its open to all, its just that there are binding votes > and > >> > > > > non-binding > >> > > > > >>> >> votes. > >> > > > > >>> >> > >> > > > > >>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev < > >> > > > > >>> >> valen...@google.com.invalid> wrote: > >> > > > > >>> >> > >> > > > > >>> >> > I have a process question: is the vote open for > >> committers > >> > > only > >> > > > or > >> > > > > >>> for > >> > > > > >>> >> all > >> > > > > >>> >> > contributors? > >> > > > > >>> >> > > >> > > > > >>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik > >> > > > > >>> <lc...@google.com.invalid> > >> > > > > >>> >> > wrote: > >> > > > > >>> >> > > >> > > > > >>> >> > > +1, Approve the release > >> > > > > >>> >> > > > >> > > > > >>> >> > > I have verified the wordcount quickstart on the > Apache > >> > Beam > >> > > > > website > >> > > > > >>> >> using > >> > > > > >>> >> > > Apex, DirectRunner, Flink & Spark on Linux. > >> > > > > >>> >> > > > >> > > > > >>> >> > > The Gearpump runner is yet to have a quickstart > listed > >> on > >> > > our > >> > > > > >>> website. > >> > > > > >>> >> > > Adding the quickstart is already represented by this > >> > > existing > >> > > > > >>> issue: > >> > > > > >>> >> > > https://issues.apache.org/jira/browse/BEAM-2692 > >> > > > > >>> >> > > > >> > > > > >>> >> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn > Tymofieiev < > >> > > > > >>> >> > > valen...@google.com.invalid> wrote: > >> > > > > >>> >> > > > >> > > > > >>> >> > > > I have verified: SHA & MD5 signatures of Python > >> > artifacts > >> > > in > >> > > > > >>> [2], and > >> > > > > >>> >> > > > checked Python side of the validation checklist on > >> > Linux. > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > There is one known issue in UserScore example for > >> > Dataflow > >> > > > > >>> runner. > >> > > > > >>> >> The > >> > > > > >>> >> > > > issue has been fixed on master branch and does not > >> > > require a > >> > > > > >>> >> > cherry-pick > >> > > > > >>> >> > > at > >> > > > > >>> >> > > > this point. A workaround is to pass > >> --save_main_session > >> > > > > pipeline > >> > > > > >>> >> > option. > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > RC4 looks good to me so far. > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles > >> > > > > >>> >> > <k...@google.com.invalid > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > wrote: > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > Hi all, > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > Following up on past discussions and > >> > > > > >>> >> > > > > https://issues.apache.org/jira/browse/BEAM-1189 > I > >> > have > >> > > > > >>> prepared a > >> > > > > >>> >> > > > > spreadsheet so we can sign up for validation > steps > >> > that > >> > > > > must be > >> > > > > >>> >> done > >> > > > > >>> >> > > by a > >> > > > > >>> >> > > > > human. > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > The spreadsheet for 2.2.0 is at > >> > > > > >>> >> > > > > https://s.apache.org/beam-2.2. > 0-release-validation > >> . > >> > > > > Everyone > >> > > > > >>> can > >> > > > > >>> >> > edit, > >> > > > > >>> >> > > > so > >> > > > > >>> >> > > > > it is easy to sign up or add new rows. > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > FYI the template is at > >> > > > > >>> >> https://s.apache.org/beam-release-validation. > >> > > > > >>> >> > > > After > >> > > > > >>> >> > > > > this release, I will update the rows in the > >> template > >> > to > >> > > > > match > >> > > > > >>> >> > whatever > >> > > > > >>> >> > > we > >> > > > > >>> >> > > > > have added to the validation for 2.2.0. > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > Kenn > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax > >> > > > > >>> >> > <re...@google.com.invalid > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > wrote: > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > > Hi everyone, > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > Please review and vote on the release candidate > >> #4 > >> > for > >> > > > the > >> > > > > >>> >> version > >> > > > > >>> >> > > > 2.2.0, > >> > > > > >>> >> > > > > > as follows: > >> > > > > >>> >> > > > > > [ ] +1, Approve the release > >> > > > > >>> >> > > > > > [ ] -1, Do not approve the release (please > >> provide > >> > > > > specific > >> > > > > >>> >> > > comments) > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > The complete staging area is available for your > >> > > review, > >> > > > > which > >> > > > > >>> >> > > includes: > >> > > > > >>> >> > > > > > * JIRA release notes [1], > >> > > > > >>> >> > > > > > * the official Apache source release to be > >> > deployed > >> > > to > >> > > > > >>> >> > > > dist.apache.org > >> > > > > >>> >> > > > > > [2], > >> > > > > >>> >> > > > > > which is signed with the key with fingerprint > >> > B98B7708 > >> > > > > [3], > >> > > > > >>> >> > > > > > * all artifacts to be deployed to the Maven > >> > Central > >> > > > > >>> Repository > >> > > > > >>> >> > [4], > >> > > > > >>> >> > > > > > * source code tag "v2.2.0-RC4" [5], > >> > > > > >>> >> > > > > > * website pull request listing the release > and > >> > > > > publishing > >> > > > > >>> the > >> > > > > >>> >> API > >> > > > > >>> >> > > > > > reference manual [6]. > >> > > > > >>> >> > > > > > * Java artifacts were built with Maven 3.5.0 > >> and > >> > > > > >>> OpenJDK/Oracle > >> > > > > >>> >> > JDK > >> > > > > >>> >> > > > > > 1.8.0_144. > >> > > > > >>> >> > > > > > * Python artifacts are deployed along with > the > >> > > source > >> > > > > >>> release > >> > > > > >>> >> to > >> > > > > >>> >> > > the > >> > > > > >>> >> > > > > > dist.apache.org [2]. > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > The vote will be open for at least 72 hours. It > >> is > >> > > > > adopted by > >> > > > > >>> >> > > majority > >> > > > > >>> >> > > > > > approval, with at least 3 PMC affirmative > votes. > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > Thanks, > >> > > > > >>> >> > > > > > Reuven > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > [1] https://issues.apache.org/ > >> > > > > jira/secure/ReleaseNote.jspa?p > >> > > > > >>> >> > > > > > rojectId=12319527&version=12341044 > >> > > > > >>> >> > > > > > [2] https://dist.apache.org/repos/ > >> > > dist/dev/beam/2.2.0/ > >> > > > > >>> >> > > > > > [3] > >> > > > https://dist.apache.org/repos/dist/release/beam/KEYS > >> > > > > >>> >> > > > > > [4] https://repository.apache.org/ > >> > > content/repositories/ > >> > > > > >>> >> > > > > orgapachebeam-1025/ > >> > > > > >>> >> > > > > > [5] https://github.com/apache/ > >> beam/tree/v2.2.0-RC4 > >> > > > > >>> >> > > > > > <https://github.com/apache/ > beam/tree/v2.2.0-RC4> > >> > > > > >>> >> > > > > > [6] https://github.com/apache/ > beam-site/pull/337 > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > >> > > > > >>> >> > > >> > > > > >>> >> > >> > > > > >>> > >> > > > > >> > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > > -- > > Thanks & Regards, > > Nishu Tayal > > > > <https://www.avast.com/sig-email?utm_medium=email&utm_ > source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> > Virus-free. > www.avast.com > <https://www.avast.com/sig-email?utm_medium=email&utm_ > source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > -- Gus Katsiapis | Software Engineer | katsia...@google.com | 650-918-7487
