Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

FYI I'm still working on finalizing the release. The issue is that the Beam
documentation underwent a large refactor while the release was pending, so
the website PR (pr/337) conflicts. I think it's safer to give up on
resolving these conflicts, so I'm simply going to regenerate this PR from
scratch.

Reuven

On Sat, Nov 25, 2017 at 11:13 PM, Reuven Lax  wrote:

> Sure, I was mostly surprised it was taking this long. However Robert says
> it sometimes takes three days.
>
> Reuven
>
> On Sat, Nov 25, 2017 at 9:11 PM, Jean-Baptiste Onofré 
> wrote:
>
>> mvnrepository doesn't matter (it will sync later), the actual Central URL
>> is:
>>
>> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdks-java-core/
>>
>> And 2.2.0 is there.
>>
>> Regards
>> JB
>>
>> On 11/25/2017 08:01 PM, Reuven Lax wrote:
>>
>>> BTW,
>>>
>>> It's been over a day, and I still don't see 2.2.0 listed at
>>> https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core
>>> How
>>> long does it usually take to promote the artifacts here?
>>>
>>> On Fri, Nov 24, 2017 at 3:43 PM, Reuven Lax  wrote:
>>>
>>> Appears to be a problem :)

 I tried publishing the latest artifact from Apache Nexus to Maven
 Central.
 After clicking publish, Nexus claimed that the operation has completed.
 However a look at the Maven Central page (https://mvnrepository.com/
 artifact/org.apache.beam/beam-sdks-java-core) does not show 2.2.0
 artifacts, and the staging repository has now vanished from the Nexus
 site!
 Does anyone know what happened here?

 Reuven

 On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:

 +1
>
> Run quickstart with Apex runner in embedded mode and on YARN.
>
> It needed couple tweaks to get there though.
>
> 1) Change quickstart pom.xml apex-runner profile:
>
>  
>  
>org.apache.hadoop
>hadoop-yarn-client
>${hadoop.version}
>runtime
>  
>  
>org.apache.hadoop
>hadoop-common
>${hadoop.version}
>runtime
>  
>
> 2) After copying the fat jar to the cluster:
>
> java -cp word-count-beam-bundled-0.1.jar
> org.apache.beam.examples.WordC
> ount
> \
>   --inputFile=file:///tmp/input.txt --output=/tmp/counts
> --embeddedExecution=false --configFile=beam-runners-apex.properties
> --runner=ApexRunner
>
> (this was on a single node cluster, hence the local file path)
>
> The quickstart instructions suggest to use *mvn exec:java* instead of
> *java*
> - it generally isn't valid to assume that mvn and a build environment
> exists on the edge node of a YARN cluster.
>
>
>
> On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:
>
> Hi Eugene,
>>
>> I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
>> Cluster(Yarn). Both ran successfully. :)
>>
>> Regards,
>> Nishu
>>
>> > source=link_campaign=sig-email_content=webmail_term=icon>
>> Virus-free.
>> www.avast.com
>> > source=link_campaign=sig-email_content=webmail_term=link>
>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>> On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
>> kirpic...@google.com.invalid> wrote:
>>
>> Thanks Nishu. So, if I understand correctly, your pipelines were
>>>
>> running
>
>> on
>>
>>> non-YARN, but you're planning to run with YARN?
>>>
>>> I meanwhile was able to get Flink running on Dataproc (YARN), and
>>>
>> validated
>>
>>> quickstart and game examples.
>>> At this point we need validation for Spark and Flink non-YARN [I
>>>
>> think if
>
>> Nishu's runs were non-YARN, they'd give us enough confidence, combined
>>>
>> with
>>
>>> the success of other validations of Spark and Flink runners?], and
>>>
>> Apex
>
>> on
>>
>>> YARN. However, it seems that in previous RCs we were not validating
>>>
>> Apex
>
>> on
>>
>>> YARN, only local cluster. Is it needed this time?
>>>
>>> On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:
>>>
>>> Hi Eugene,

 No, I didn't try with those instead I have my custom pipeline where

>>> Kafka
>>
>>> topic is the source. I have defined a Global Window and processing

>>> time
>
>> trigger to read the data. Further it runs some transformation i.e.
 GroupByKey and CoGroupByKey. on the windowed collections.
 I was running the same pipeline on direct 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Sure, I was mostly surprised it was taking this long. However Robert says
it sometimes takes three days.

Reuven

On Sat, Nov 25, 2017 at 9:11 PM, Jean-Baptiste Onofré 
wrote:

> mvnrepository doesn't matter (it will sync later), the actual Central URL
> is:
>
> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdks-java-core/
>
> And 2.2.0 is there.
>
> Regards
> JB
>
> On 11/25/2017 08:01 PM, Reuven Lax wrote:
>
>> BTW,
>>
>> It's been over a day, and I still don't see 2.2.0 listed at
>> https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core
>> How
>> long does it usually take to promote the artifacts here?
>>
>> On Fri, Nov 24, 2017 at 3:43 PM, Reuven Lax  wrote:
>>
>> Appears to be a problem :)
>>>
>>> I tried publishing the latest artifact from Apache Nexus to Maven
>>> Central.
>>> After clicking publish, Nexus claimed that the operation has completed.
>>> However a look at the Maven Central page (https://mvnrepository.com/
>>> artifact/org.apache.beam/beam-sdks-java-core) does not show 2.2.0
>>> artifacts, and the staging repository has now vanished from the Nexus
>>> site!
>>> Does anyone know what happened here?
>>>
>>> Reuven
>>>
>>> On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:
>>>
>>> +1

 Run quickstart with Apex runner in embedded mode and on YARN.

 It needed couple tweaks to get there though.

 1) Change quickstart pom.xml apex-runner profile:

  
  
org.apache.hadoop
hadoop-yarn-client
${hadoop.version}
runtime
  
  
org.apache.hadoop
hadoop-common
${hadoop.version}
runtime
  

 2) After copying the fat jar to the cluster:

 java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordC
 ount
 \
   --inputFile=file:///tmp/input.txt --output=/tmp/counts
 --embeddedExecution=false --configFile=beam-runners-apex.properties
 --runner=ApexRunner

 (this was on a single node cluster, hence the local file path)

 The quickstart instructions suggest to use *mvn exec:java* instead of
 *java*
 - it generally isn't valid to assume that mvn and a build environment
 exists on the edge node of a YARN cluster.



 On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:

 Hi Eugene,
>
> I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
> Cluster(Yarn). Both ran successfully. :)
>
> Regards,
> Nishu
>
>  source=link_campaign=sig-email_content=webmail_term=icon>
> Virus-free.
> www.avast.com
>  source=link_campaign=sig-email_content=webmail_term=link>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
> Thanks Nishu. So, if I understand correctly, your pipelines were
>>
> running

> on
>
>> non-YARN, but you're planning to run with YARN?
>>
>> I meanwhile was able to get Flink running on Dataproc (YARN), and
>>
> validated
>
>> quickstart and game examples.
>> At this point we need validation for Spark and Flink non-YARN [I
>>
> think if

> Nishu's runs were non-YARN, they'd give us enough confidence, combined
>>
> with
>
>> the success of other validations of Spark and Flink runners?], and
>>
> Apex

> on
>
>> YARN. However, it seems that in previous RCs we were not validating
>>
> Apex

> on
>
>> YARN, only local cluster. Is it needed this time?
>>
>> On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:
>>
>> Hi Eugene,
>>>
>>> No, I didn't try with those instead I have my custom pipeline where
>>>
>> Kafka
>
>> topic is the source. I have defined a Global Window and processing
>>>
>> time

> trigger to read the data. Further it runs some transformation i.e.
>>> GroupByKey and CoGroupByKey. on the windowed collections.
>>> I was running the same pipeline on direct runner and spark runner
>>>
>> earlier..
>>
>>> Today gave it a try with Flink on Yarn.
>>>
>>> Best Regards,
>>> Nishu.
>>>
>>> <
>>> https://www.avast.com/sig-email?utm_medium=email_
>>>
>> source=link_campaign=sig-email_content=webmail_term=icon
>>
>>>
 Virus-free.
>>> www.avast.com
>>> <
>>> https://www.avast.com/sig-email?utm_medium=email_
>>>
>> source=link_campaign=sig-email_content=webmail_term=link
>>
>>>
 

Re: [VOTE] Release 2.2.0, release candidate #4

[Robert Bradshaw]

Historically mvn has taken 3+ days to finish syncing. Maybe some of
the links are using RFC 1149 :).

On Sat, Nov 25, 2017 at 9:11 PM, Jean-Baptiste Onofré  wrote:
> mvnrepository doesn't matter (it will sync later), the actual Central URL
> is:
>
> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdks-java-core/
>
> And 2.2.0 is there.
>
> Regards
> JB
>
> On 11/25/2017 08:01 PM, Reuven Lax wrote:
>>
>> BTW,
>>
>> It's been over a day, and I still don't see 2.2.0 listed at
>> https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core How
>> long does it usually take to promote the artifacts here?
>>
>> On Fri, Nov 24, 2017 at 3:43 PM, Reuven Lax  wrote:
>>
>>> Appears to be a problem :)
>>>
>>> I tried publishing the latest artifact from Apache Nexus to Maven
>>> Central.
>>> After clicking publish, Nexus claimed that the operation has completed.
>>> However a look at the Maven Central page (https://mvnrepository.com/
>>> artifact/org.apache.beam/beam-sdks-java-core) does not show 2.2.0
>>> artifacts, and the staging repository has now vanished from the Nexus
>>> site!
>>> Does anyone know what happened here?
>>>
>>> Reuven
>>>
>>> On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:
>>>
 +1

 Run quickstart with Apex runner in embedded mode and on YARN.

 It needed couple tweaks to get there though.

 1) Change quickstart pom.xml apex-runner profile:

  
  
org.apache.hadoop
hadoop-yarn-client
${hadoop.version}
runtime
  
  
org.apache.hadoop
hadoop-common
${hadoop.version}
runtime
  

 2) After copying the fat jar to the cluster:

 java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordC
 ount
 \
   --inputFile=file:///tmp/input.txt --output=/tmp/counts
 --embeddedExecution=false --configFile=beam-runners-apex.properties
 --runner=ApexRunner

 (this was on a single node cluster, hence the local file path)

 The quickstart instructions suggest to use *mvn exec:java* instead of
 *java*
 - it generally isn't valid to assume that mvn and a build environment
 exists on the edge node of a YARN cluster.



 On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:

> Hi Eugene,
>
> I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
> Cluster(Yarn). Both ran successfully. :)
>
> Regards,
> Nishu
>
>  source=link_campaign=sig-email_content=webmail_term=icon>
> Virus-free.
> www.avast.com
>  source=link_campaign=sig-email_content=webmail_term=link>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
>> Thanks Nishu. So, if I understand correctly, your pipelines were

 running
>
> on
>>
>> non-YARN, but you're planning to run with YARN?
>>
>> I meanwhile was able to get Flink running on Dataproc (YARN), and
>
> validated
>>
>> quickstart and game examples.
>> At this point we need validation for Spark and Flink non-YARN [I

 think if
>>
>> Nishu's runs were non-YARN, they'd give us enough confidence, combined
>
> with
>>
>> the success of other validations of Spark and Flink runners?], and

 Apex
>
> on
>>
>> YARN. However, it seems that in previous RCs we were not validating

 Apex
>
> on
>>
>> YARN, only local cluster. Is it needed this time?
>>
>> On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:
>>
>>> Hi Eugene,
>>>
>>> No, I didn't try with those instead I have my custom pipeline where
>
> Kafka
>>>
>>> topic is the source. I have defined a Global Window and processing

 time
>>>
>>> trigger to read the data. Further it runs some transformation i.e.
>>> GroupByKey and CoGroupByKey. on the windowed collections.
>>> I was running the same pipeline on direct runner and spark runner
>>
>> earlier..
>>>
>>> Today gave it a try with Flink on Yarn.
>>>
>>> Best Regards,
>>> Nishu.
>>>
>>> <
>>> https://www.avast.com/sig-email?utm_medium=email_
>>
>> source=link_campaign=sig-email_content=webmail_term=icon


>>> Virus-free.
>>> www.avast.com
>>> <
>>> https://www.avast.com/sig-email?utm_medium=email_
>>
>> source=link_campaign=sig-email_content=webmail_term=link


>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: [VOTE] Release 2.2.0, release candidate #4

[Jean-Baptiste Onofré]

mvnrepository doesn't matter (it will sync later), the actual Central URL is:

http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdks-java-core/

And 2.2.0 is there.

Regards
JB

On 11/25/2017 08:01 PM, Reuven Lax wrote:

BTW,

It's been over a day, and I still don't see 2.2.0 listed at
https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core How
long does it usually take to promote the artifacts here?

On Fri, Nov 24, 2017 at 3:43 PM, Reuven Lax  wrote:


Appears to be a problem :)

I tried publishing the latest artifact from Apache Nexus to Maven Central.
After clicking publish, Nexus claimed that the operation has completed.
However a look at the Maven Central page (https://mvnrepository.com/
artifact/org.apache.beam/beam-sdks-java-core) does not show 2.2.0
artifacts, and the staging repository has now vanished from the Nexus site!
Does anyone know what happened here?

Reuven

On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:


+1

Run quickstart with Apex runner in embedded mode and on YARN.

It needed couple tweaks to get there though.

1) Change quickstart pom.xml apex-runner profile:

 
 
   org.apache.hadoop
   hadoop-yarn-client
   ${hadoop.version}
   runtime
 
 
   org.apache.hadoop
   hadoop-common
   ${hadoop.version}
   runtime
 

2) After copying the fat jar to the cluster:

java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordC
ount
\
  --inputFile=file:///tmp/input.txt --output=/tmp/counts
--embeddedExecution=false --configFile=beam-runners-apex.properties
--runner=ApexRunner

(this was on a single node cluster, hence the local file path)

The quickstart instructions suggest to use *mvn exec:java* instead of
*java*
- it generally isn't valid to assume that mvn and a build environment
exists on the edge node of a YARN cluster.



On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:


Hi Eugene,

I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
Cluster(Yarn). Both ran successfully. :)

Regards,
Nishu


Virus-free.
www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:


Thanks Nishu. So, if I understand correctly, your pipelines were

running

on

non-YARN, but you're planning to run with YARN?

I meanwhile was able to get Flink running on Dataproc (YARN), and

validated

quickstart and game examples.
At this point we need validation for Spark and Flink non-YARN [I

think if

Nishu's runs were non-YARN, they'd give us enough confidence, combined

with

the success of other validations of Spark and Flink runners?], and

Apex

on

YARN. However, it seems that in previous RCs we were not validating

Apex

on

YARN, only local cluster. Is it needed this time?

On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:


Hi Eugene,

No, I didn't try with those instead I have my custom pipeline where

Kafka

topic is the source. I have defined a Global Window and processing

time

trigger to read the data. Further it runs some transformation i.e.
GroupByKey and CoGroupByKey. on the windowed collections.
I was running the same pipeline on direct runner and spark runner

earlier..

Today gave it a try with Flink on Yarn.

Best Regards,
Nishu.

<
https://www.avast.com/sig-email?utm_medium=email_

source=link_campaign=sig-email_content=webmail_term=icon



Virus-free.
www.avast.com
<
https://www.avast.com/sig-email?utm_medium=email_

source=link_campaign=sig-email_content=webmail_term=link



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:


Thanks Nishu! Can you clarify which pipeline you were running?
The validation spreadsheet includes 1) the quickstart and 2)

mobile

game

walkthroughs. Was it one of these, or your custom pipeline?

On Wed, Nov 22, 2017 at 10:20 AM Nishu 

wrote:



Hi,

Typo in previous mail.  I meant Flink runner.

Thanks,
Nishu
On Wed, 22 Nov 2017 at 19.17,


Hi,

I build a pipeline using RC 2.2 today and ran with runner on

yarn.

It worked seamlessly for unbounded sources. Couldn’t see any

issues

with

my pipeline so far :)


Thanks,Nishu

On Wed, 22 Nov 2017 at 18.57, Reuven Lax

 wrote:


In the verification spreadsheet, I'm not sure I

understand

the

difference

between the "YARN" and "Standalone 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Release instructions just show the "git tag" command for generating the new
tag. However I'm assuming that I also need to push the new tag to the
remote, right?

Reuven

On Sat, Nov 25, 2017 at 11:01 AM, Reuven Lax  wrote:

> Getting close. A few more steps left (source release, new git tag, merging
> the website PR, and marking it as released in JIRA).
>
> On Sat, Nov 25, 2017 at 10:12 AM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
>> So, is the release done now? Have both Java and Python artifacts been
>> uploaded?
>>
>> On Sat, Nov 25, 2017, 9:05 AM Robert Bradshaw > >
>> wrote:
>>
>> > Upload to PyPi done. https://pypi.python.org/pypi/apache-beam
>> >
>> > On Sat, Nov 25, 2017 at 8:56 AM, Robert Bradshaw 
>> > wrote:
>> > > I can do that.
>> > >
>> > > On Fri, Nov 24, 2017, 11:13 PM Reuven Lax 
>> > wrote:
>> > >>
>> > >> I am not an owner or maintainer of the PyPi package, so I'm not sure
>> if
>> > >> I'll be able to release Python artifacts.
>> > >>
>> > >> On Fri, Nov 24, 2017 at 10:08 PM, Jean-Baptiste Onofré <
>> j...@nanthrax.net
>> > >
>> > >> wrote:
>> > >>
>> > >> > Awesome, Thanks Reuven !
>> > >> >
>> > >> > Regards
>> > >> > JB
>> > >> >
>> > >> > On 11/25/2017 07:01 AM, Reuven Lax wrote:
>> > >> >
>> > >> >> I'll go ahead and send the RESULT email right now.
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> On Fri, Nov 24, 2017 at 9:56 PM, Jean-Baptiste Onofré <
>> > j...@nanthrax.net>
>> > >> >> wrote:
>> > >> >>
>> > >> >> It's not sync: promoting the artifacts takes some time (at least
>> 30
>> > >> >>> minutes). So, the artifacts will be on central after a certain
>> time.
>> > >> >>>
>> > >> >>> I confirm that it's OK because now the artifacts are on Central:
>> > >> >>>
>> > >> >>> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdk
>> > >> >>> s-java-core/2.2.0/
>> > >> >>>
>> > >> >>> By the way, you are promoting the artifacts to Central but I
>> didn't
>> > >> >>> see
>> > >> >>> any [RESULT] e-mail on the vote thread. You have first to close
>> the
>> > >> >>> vote,
>> > >> >>> then promote the artifacts, announce the release, etc.
>> > >> >>>
>> > >> >>> Regards
>> > >> >>> JB
>> > >> >>>
>> > >> >>> On 11/25/2017 12:43 AM, Reuven Lax wrote:
>> > >> >>>
>> > >> >>> Appears to be a problem :)
>> > >> 
>> > >>  I tried publishing the latest artifact from Apache Nexus to
>> Maven
>> > >>  Central.
>> > >>  After clicking publish, Nexus claimed that the operation has
>> > >>  completed.
>> > >>  However a look at the Maven Central page (
>> > >> 
>> > >> 
>> > https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
>> > >>  does not show 2.2.0 artifacts, and the staging repository has
>> now
>> > >>  vanished
>> > >>  from the Nexus site! Does anyone know what happened here?
>> > >> 
>> > >>  Reuven
>> > >> 
>> > >>  On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise 
>> > >>  wrote:
>> > >> 
>> > >>  +1
>> > >> 
>> > >> >
>> > >> > Run quickstart with Apex runner in embedded mode and on YARN.
>> > >> >
>> > >> > It needed couple tweaks to get there though.
>> > >> >
>> > >> > 1) Change quickstart pom.xml apex-runner profile:
>> > >> >
>> > >> >   
>> > >> >   
>> > >> > org.apache.hadoop
>> > >> > hadoop-yarn-client
>> > >> > ${hadoop.version}
>> > >> > runtime
>> > >> >   
>> > >> >   
>> > >> > org.apache.hadoop
>> > >> > hadoop-common
>> > >> > ${hadoop.version}
>> > >> > runtime
>> > >> >   
>> > >> >
>> > >> > 2) After copying the fat jar to the cluster:
>> > >> >
>> > >> > java -cp word-count-beam-bundled-0.1.jar
>> org.apache.beam.examples.
>> > >> > WordCount
>> > >> > \
>> > >> >--inputFile=file:///tmp/input.txt --output=/tmp/counts
>> > >> > --embeddedExecution=false
>> > --configFile=beam-runners-apex.properties
>> > >> > --runner=ApexRunner
>> > >> >
>> > >> > (this was on a single node cluster, hence the local file path)
>> > >> >
>> > >> > The quickstart instructions suggest to use *mvn exec:java*
>> instead
>> > >> > of
>> > >> > *java*
>> > >> > - it generally isn't valid to assume that mvn and a build
>> > >> > environment
>> > >> > exists on the edge node of a YARN cluster.
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Wed, Nov 22, 2017 at 2:12 PM, Nishu 
>> > wrote:
>> > >> >
>> > >> > Hi Eugene,
>> > >> >
>> > >> >>
>> > >> >> I ran it on both  standalone flink(non Yarn) and  Flink on
>> > >> >> HDInsight
>> > >> >> Cluster(Yarn). Both ran successfully. :)
>> > >> >>
>> > >> >> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

BTW,

It's been over a day, and I still don't see 2.2.0 listed at
https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core How
long does it usually take to promote the artifacts here?

On Fri, Nov 24, 2017 at 3:43 PM, Reuven Lax  wrote:

> Appears to be a problem :)
>
> I tried publishing the latest artifact from Apache Nexus to Maven Central.
> After clicking publish, Nexus claimed that the operation has completed.
> However a look at the Maven Central page (https://mvnrepository.com/
> artifact/org.apache.beam/beam-sdks-java-core) does not show 2.2.0
> artifacts, and the staging repository has now vanished from the Nexus site!
> Does anyone know what happened here?
>
> Reuven
>
> On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:
>
>> +1
>>
>> Run quickstart with Apex runner in embedded mode and on YARN.
>>
>> It needed couple tweaks to get there though.
>>
>> 1) Change quickstart pom.xml apex-runner profile:
>>
>> 
>> 
>>   org.apache.hadoop
>>   hadoop-yarn-client
>>   ${hadoop.version}
>>   runtime
>> 
>> 
>>   org.apache.hadoop
>>   hadoop-common
>>   ${hadoop.version}
>>   runtime
>> 
>>
>> 2) After copying the fat jar to the cluster:
>>
>> java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordC
>> ount
>> \
>>  --inputFile=file:///tmp/input.txt --output=/tmp/counts
>> --embeddedExecution=false --configFile=beam-runners-apex.properties
>> --runner=ApexRunner
>>
>> (this was on a single node cluster, hence the local file path)
>>
>> The quickstart instructions suggest to use *mvn exec:java* instead of
>> *java*
>> - it generally isn't valid to assume that mvn and a build environment
>> exists on the edge node of a YARN cluster.
>>
>>
>>
>> On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:
>>
>> > Hi Eugene,
>> >
>> > I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
>> > Cluster(Yarn). Both ran successfully. :)
>> >
>> > Regards,
>> > Nishu
>> >
>> > > > source=link_campaign=sig-email_content=webmail_term=icon>
>> > Virus-free.
>> > www.avast.com
>> > > > source=link_campaign=sig-email_content=webmail_term=link>
>> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>> >
>> > On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
>> > kirpic...@google.com.invalid> wrote:
>> >
>> > > Thanks Nishu. So, if I understand correctly, your pipelines were
>> running
>> > on
>> > > non-YARN, but you're planning to run with YARN?
>> > >
>> > > I meanwhile was able to get Flink running on Dataproc (YARN), and
>> > validated
>> > > quickstart and game examples.
>> > > At this point we need validation for Spark and Flink non-YARN [I
>> think if
>> > > Nishu's runs were non-YARN, they'd give us enough confidence, combined
>> > with
>> > > the success of other validations of Spark and Flink runners?], and
>> Apex
>> > on
>> > > YARN. However, it seems that in previous RCs we were not validating
>> Apex
>> > on
>> > > YARN, only local cluster. Is it needed this time?
>> > >
>> > > On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:
>> > >
>> > > > Hi Eugene,
>> > > >
>> > > > No, I didn't try with those instead I have my custom pipeline where
>> > Kafka
>> > > > topic is the source. I have defined a Global Window and processing
>> time
>> > > > trigger to read the data. Further it runs some transformation i.e.
>> > > > GroupByKey and CoGroupByKey. on the windowed collections.
>> > > > I was running the same pipeline on direct runner and spark runner
>> > > earlier..
>> > > > Today gave it a try with Flink on Yarn.
>> > > >
>> > > > Best Regards,
>> > > > Nishu.
>> > > >
>> > > > <
>> > > > https://www.avast.com/sig-email?utm_medium=email_
>> > > source=link_campaign=sig-email_content=webmail_term=icon
>> > > > >
>> > > > Virus-free.
>> > > > www.avast.com
>> > > > <
>> > > > https://www.avast.com/sig-email?utm_medium=email_
>> > > source=link_campaign=sig-email_content=webmail_term=link
>> > > > >
>> > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>> > > >
>> > > > On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov <
>> > > > kirpic...@google.com.invalid> wrote:
>> > > >
>> > > > > Thanks Nishu! Can you clarify which pipeline you were running?
>> > > > > The validation spreadsheet includes 1) the quickstart and 2)
>> mobile
>> > > game
>> > > > > walkthroughs. Was it one of these, or your custom pipeline?
>> > > > >
>> > > > > On Wed, Nov 22, 2017 at 10:20 AM Nishu 
>> wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Typo in previous mail.  I meant Flink runner.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Nishu
>> > > > > > On Wed, 22 Nov 2017 at 19.17,
>> > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > I build a pipeline using RC 2.2 today and ran with 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Getting close. A few more steps left (source release, new git tag, merging
the website PR, and marking it as released in JIRA).

On Sat, Nov 25, 2017 at 10:12 AM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> So, is the release done now? Have both Java and Python artifacts been
> uploaded?
>
> On Sat, Nov 25, 2017, 9:05 AM Robert Bradshaw  >
> wrote:
>
> > Upload to PyPi done. https://pypi.python.org/pypi/apache-beam
> >
> > On Sat, Nov 25, 2017 at 8:56 AM, Robert Bradshaw 
> > wrote:
> > > I can do that.
> > >
> > > On Fri, Nov 24, 2017, 11:13 PM Reuven Lax 
> > wrote:
> > >>
> > >> I am not an owner or maintainer of the PyPi package, so I'm not sure
> if
> > >> I'll be able to release Python artifacts.
> > >>
> > >> On Fri, Nov 24, 2017 at 10:08 PM, Jean-Baptiste Onofré <
> j...@nanthrax.net
> > >
> > >> wrote:
> > >>
> > >> > Awesome, Thanks Reuven !
> > >> >
> > >> > Regards
> > >> > JB
> > >> >
> > >> > On 11/25/2017 07:01 AM, Reuven Lax wrote:
> > >> >
> > >> >> I'll go ahead and send the RESULT email right now.
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, Nov 24, 2017 at 9:56 PM, Jean-Baptiste Onofré <
> > j...@nanthrax.net>
> > >> >> wrote:
> > >> >>
> > >> >> It's not sync: promoting the artifacts takes some time (at least 30
> > >> >>> minutes). So, the artifacts will be on central after a certain
> time.
> > >> >>>
> > >> >>> I confirm that it's OK because now the artifacts are on Central:
> > >> >>>
> > >> >>> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdk
> > >> >>> s-java-core/2.2.0/
> > >> >>>
> > >> >>> By the way, you are promoting the artifacts to Central but I
> didn't
> > >> >>> see
> > >> >>> any [RESULT] e-mail on the vote thread. You have first to close
> the
> > >> >>> vote,
> > >> >>> then promote the artifacts, announce the release, etc.
> > >> >>>
> > >> >>> Regards
> > >> >>> JB
> > >> >>>
> > >> >>> On 11/25/2017 12:43 AM, Reuven Lax wrote:
> > >> >>>
> > >> >>> Appears to be a problem :)
> > >> 
> > >>  I tried publishing the latest artifact from Apache Nexus to Maven
> > >>  Central.
> > >>  After clicking publish, Nexus claimed that the operation has
> > >>  completed.
> > >>  However a look at the Maven Central page (
> > >> 
> > >> 
> > https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
> > >>  does not show 2.2.0 artifacts, and the staging repository has now
> > >>  vanished
> > >>  from the Nexus site! Does anyone know what happened here?
> > >> 
> > >>  Reuven
> > >> 
> > >>  On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise 
> > >>  wrote:
> > >> 
> > >>  +1
> > >> 
> > >> >
> > >> > Run quickstart with Apex runner in embedded mode and on YARN.
> > >> >
> > >> > It needed couple tweaks to get there though.
> > >> >
> > >> > 1) Change quickstart pom.xml apex-runner profile:
> > >> >
> > >> >   
> > >> >   
> > >> > org.apache.hadoop
> > >> > hadoop-yarn-client
> > >> > ${hadoop.version}
> > >> > runtime
> > >> >   
> > >> >   
> > >> > org.apache.hadoop
> > >> > hadoop-common
> > >> > ${hadoop.version}
> > >> > runtime
> > >> >   
> > >> >
> > >> > 2) After copying the fat jar to the cluster:
> > >> >
> > >> > java -cp word-count-beam-bundled-0.1.jar
> org.apache.beam.examples.
> > >> > WordCount
> > >> > \
> > >> >--inputFile=file:///tmp/input.txt --output=/tmp/counts
> > >> > --embeddedExecution=false
> > --configFile=beam-runners-apex.properties
> > >> > --runner=ApexRunner
> > >> >
> > >> > (this was on a single node cluster, hence the local file path)
> > >> >
> > >> > The quickstart instructions suggest to use *mvn exec:java*
> instead
> > >> > of
> > >> > *java*
> > >> > - it generally isn't valid to assume that mvn and a build
> > >> > environment
> > >> > exists on the edge node of a YARN cluster.
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Nov 22, 2017 at 2:12 PM, Nishu 
> > wrote:
> > >> >
> > >> > Hi Eugene,
> > >> >
> > >> >>
> > >> >> I ran it on both  standalone flink(non Yarn) and  Flink on
> > >> >> HDInsight
> > >> >> Cluster(Yarn). Both ran successfully. :)
> > >> >>
> > >> >> Regards,
> > >> >> Nishu
> > >> >>
> > >> >>  > >> >>
> > >> >>
> > source=link_campaign=sig-email_content=webmail_term=icon>
> > >> >> Virus-free.
> > >> >> www.avast.com
> > >> >>  > >> >>
> > >> >>
> > 

Re: [VOTE] Release 2.2.0, release candidate #4

[Eugene Kirpichov]

So, is the release done now? Have both Java and Python artifacts been
uploaded?

On Sat, Nov 25, 2017, 9:05 AM Robert Bradshaw 
wrote:

> Upload to PyPi done. https://pypi.python.org/pypi/apache-beam
>
> On Sat, Nov 25, 2017 at 8:56 AM, Robert Bradshaw 
> wrote:
> > I can do that.
> >
> > On Fri, Nov 24, 2017, 11:13 PM Reuven Lax 
> wrote:
> >>
> >> I am not an owner or maintainer of the PyPi package, so I'm not sure if
> >> I'll be able to release Python artifacts.
> >>
> >> On Fri, Nov 24, 2017 at 10:08 PM, Jean-Baptiste Onofré  >
> >> wrote:
> >>
> >> > Awesome, Thanks Reuven !
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 11/25/2017 07:01 AM, Reuven Lax wrote:
> >> >
> >> >> I'll go ahead and send the RESULT email right now.
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Nov 24, 2017 at 9:56 PM, Jean-Baptiste Onofré <
> j...@nanthrax.net>
> >> >> wrote:
> >> >>
> >> >> It's not sync: promoting the artifacts takes some time (at least 30
> >> >>> minutes). So, the artifacts will be on central after a certain time.
> >> >>>
> >> >>> I confirm that it's OK because now the artifacts are on Central:
> >> >>>
> >> >>> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdk
> >> >>> s-java-core/2.2.0/
> >> >>>
> >> >>> By the way, you are promoting the artifacts to Central but I didn't
> >> >>> see
> >> >>> any [RESULT] e-mail on the vote thread. You have first to close the
> >> >>> vote,
> >> >>> then promote the artifacts, announce the release, etc.
> >> >>>
> >> >>> Regards
> >> >>> JB
> >> >>>
> >> >>> On 11/25/2017 12:43 AM, Reuven Lax wrote:
> >> >>>
> >> >>> Appears to be a problem :)
> >> 
> >>  I tried publishing the latest artifact from Apache Nexus to Maven
> >>  Central.
> >>  After clicking publish, Nexus claimed that the operation has
> >>  completed.
> >>  However a look at the Maven Central page (
> >> 
> >> 
> https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
> >>  does not show 2.2.0 artifacts, and the staging repository has now
> >>  vanished
> >>  from the Nexus site! Does anyone know what happened here?
> >> 
> >>  Reuven
> >> 
> >>  On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise 
> >>  wrote:
> >> 
> >>  +1
> >> 
> >> >
> >> > Run quickstart with Apex runner in embedded mode and on YARN.
> >> >
> >> > It needed couple tweaks to get there though.
> >> >
> >> > 1) Change quickstart pom.xml apex-runner profile:
> >> >
> >> >   
> >> >   
> >> > org.apache.hadoop
> >> > hadoop-yarn-client
> >> > ${hadoop.version}
> >> > runtime
> >> >   
> >> >   
> >> > org.apache.hadoop
> >> > hadoop-common
> >> > ${hadoop.version}
> >> > runtime
> >> >   
> >> >
> >> > 2) After copying the fat jar to the cluster:
> >> >
> >> > java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.
> >> > WordCount
> >> > \
> >> >--inputFile=file:///tmp/input.txt --output=/tmp/counts
> >> > --embeddedExecution=false
> --configFile=beam-runners-apex.properties
> >> > --runner=ApexRunner
> >> >
> >> > (this was on a single node cluster, hence the local file path)
> >> >
> >> > The quickstart instructions suggest to use *mvn exec:java* instead
> >> > of
> >> > *java*
> >> > - it generally isn't valid to assume that mvn and a build
> >> > environment
> >> > exists on the edge node of a YARN cluster.
> >> >
> >> >
> >> >
> >> > On Wed, Nov 22, 2017 at 2:12 PM, Nishu 
> wrote:
> >> >
> >> > Hi Eugene,
> >> >
> >> >>
> >> >> I ran it on both  standalone flink(non Yarn) and  Flink on
> >> >> HDInsight
> >> >> Cluster(Yarn). Both ran successfully. :)
> >> >>
> >> >> Regards,
> >> >> Nishu
> >> >>
> >> >>  >> >>
> >> >>
> source=link_campaign=sig-email_content=webmail_term=icon>
> >> >> Virus-free.
> >> >> www.avast.com
> >> >>  >> >>
> >> >>
> source=link_campaign=sig-email_content=webmail_term=link>
> >> >> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >> >>
> >> >> On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
> >> >> kirpic...@google.com.invalid> wrote:
> >> >>
> >> >> Thanks Nishu. So, if I understand correctly, your pipelines were
> >> >>
> >> >>>
> >> >>> running
> >> >>
> >> >
> >> > on
> >> >>
> >> >> non-YARN, but you're planning to run with YARN?
> >> >>>
> >> >>> I meanwhile was able to get Flink running on Dataproc (YARN),
> and
> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Robert Bradshaw]

Upload to PyPi done. https://pypi.python.org/pypi/apache-beam

On Sat, Nov 25, 2017 at 8:56 AM, Robert Bradshaw  wrote:
> I can do that.
>
> On Fri, Nov 24, 2017, 11:13 PM Reuven Lax  wrote:
>>
>> I am not an owner or maintainer of the PyPi package, so I'm not sure if
>> I'll be able to release Python artifacts.
>>
>> On Fri, Nov 24, 2017 at 10:08 PM, Jean-Baptiste Onofré 
>> wrote:
>>
>> > Awesome, Thanks Reuven !
>> >
>> > Regards
>> > JB
>> >
>> > On 11/25/2017 07:01 AM, Reuven Lax wrote:
>> >
>> >> I'll go ahead and send the RESULT email right now.
>> >>
>> >>
>> >>
>> >> On Fri, Nov 24, 2017 at 9:56 PM, Jean-Baptiste Onofré 
>> >> wrote:
>> >>
>> >> It's not sync: promoting the artifacts takes some time (at least 30
>> >>> minutes). So, the artifacts will be on central after a certain time.
>> >>>
>> >>> I confirm that it's OK because now the artifacts are on Central:
>> >>>
>> >>> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdk
>> >>> s-java-core/2.2.0/
>> >>>
>> >>> By the way, you are promoting the artifacts to Central but I didn't
>> >>> see
>> >>> any [RESULT] e-mail on the vote thread. You have first to close the
>> >>> vote,
>> >>> then promote the artifacts, announce the release, etc.
>> >>>
>> >>> Regards
>> >>> JB
>> >>>
>> >>> On 11/25/2017 12:43 AM, Reuven Lax wrote:
>> >>>
>> >>> Appears to be a problem :)
>> 
>>  I tried publishing the latest artifact from Apache Nexus to Maven
>>  Central.
>>  After clicking publish, Nexus claimed that the operation has
>>  completed.
>>  However a look at the Maven Central page (
>> 
>>  https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
>>  does not show 2.2.0 artifacts, and the staging repository has now
>>  vanished
>>  from the Nexus site! Does anyone know what happened here?
>> 
>>  Reuven
>> 
>>  On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise 
>>  wrote:
>> 
>>  +1
>> 
>> >
>> > Run quickstart with Apex runner in embedded mode and on YARN.
>> >
>> > It needed couple tweaks to get there though.
>> >
>> > 1) Change quickstart pom.xml apex-runner profile:
>> >
>> >   
>> >   
>> > org.apache.hadoop
>> > hadoop-yarn-client
>> > ${hadoop.version}
>> > runtime
>> >   
>> >   
>> > org.apache.hadoop
>> > hadoop-common
>> > ${hadoop.version}
>> > runtime
>> >   
>> >
>> > 2) After copying the fat jar to the cluster:
>> >
>> > java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.
>> > WordCount
>> > \
>> >--inputFile=file:///tmp/input.txt --output=/tmp/counts
>> > --embeddedExecution=false --configFile=beam-runners-apex.properties
>> > --runner=ApexRunner
>> >
>> > (this was on a single node cluster, hence the local file path)
>> >
>> > The quickstart instructions suggest to use *mvn exec:java* instead
>> > of
>> > *java*
>> > - it generally isn't valid to assume that mvn and a build
>> > environment
>> > exists on the edge node of a YARN cluster.
>> >
>> >
>> >
>> > On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:
>> >
>> > Hi Eugene,
>> >
>> >>
>> >> I ran it on both  standalone flink(non Yarn) and  Flink on
>> >> HDInsight
>> >> Cluster(Yarn). Both ran successfully. :)
>> >>
>> >> Regards,
>> >> Nishu
>> >>
>> >> > >>
>> >> source=link_campaign=sig-email_content=webmail_term=icon>
>> >> Virus-free.
>> >> www.avast.com
>> >> > >>
>> >> source=link_campaign=sig-email_content=webmail_term=link>
>> >> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>> >>
>> >> On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
>> >> kirpic...@google.com.invalid> wrote:
>> >>
>> >> Thanks Nishu. So, if I understand correctly, your pipelines were
>> >>
>> >>>
>> >>> running
>> >>
>> >
>> > on
>> >>
>> >> non-YARN, but you're planning to run with YARN?
>> >>>
>> >>> I meanwhile was able to get Flink running on Dataproc (YARN), and
>> >>>
>> >>> validated
>> >>
>> >> quickstart and game examples.
>> >>> At this point we need validation for Spark and Flink non-YARN [I
>> >>> think
>> >>>
>> >>> if
>> >>
>> >
>> > Nishu's runs were non-YARN, they'd give us enough confidence,
>> > combined
>> >>
>> >>>
>> >>> with
>> >>
>> >> the success of other validations of Spark and Flink runners?], and
>> >>> Apex
>> >>>
>> >>> on

Re: [VOTE] Release 2.2.0, release candidate #4

[Robert Bradshaw]

I can do that.

On Fri, Nov 24, 2017, 11:13 PM Reuven Lax  wrote:

> I am not an owner or maintainer of the PyPi package, so I'm not sure if
> I'll be able to release Python artifacts.
>
> On Fri, Nov 24, 2017 at 10:08 PM, Jean-Baptiste Onofré 
> wrote:
>
> > Awesome, Thanks Reuven !
> >
> > Regards
> > JB
> >
> > On 11/25/2017 07:01 AM, Reuven Lax wrote:
> >
> >> I'll go ahead and send the RESULT email right now.
> >>
> >>
> >>
> >> On Fri, Nov 24, 2017 at 9:56 PM, Jean-Baptiste Onofré 
> >> wrote:
> >>
> >> It's not sync: promoting the artifacts takes some time (at least 30
> >>> minutes). So, the artifacts will be on central after a certain time.
> >>>
> >>> I confirm that it's OK because now the artifacts are on Central:
> >>>
> >>> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdk
> >>> s-java-core/2.2.0/
> >>>
> >>> By the way, you are promoting the artifacts to Central but I didn't see
> >>> any [RESULT] e-mail on the vote thread. You have first to close the
> vote,
> >>> then promote the artifacts, announce the release, etc.
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 11/25/2017 12:43 AM, Reuven Lax wrote:
> >>>
> >>> Appears to be a problem :)
> 
>  I tried publishing the latest artifact from Apache Nexus to Maven
>  Central.
>  After clicking publish, Nexus claimed that the operation has
> completed.
>  However a look at the Maven Central page (
> 
> https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
>  does not show 2.2.0 artifacts, and the staging repository has now
>  vanished
>  from the Nexus site! Does anyone know what happened here?
> 
>  Reuven
> 
>  On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise 
> wrote:
> 
>  +1
> 
> >
> > Run quickstart with Apex runner in embedded mode and on YARN.
> >
> > It needed couple tweaks to get there though.
> >
> > 1) Change quickstart pom.xml apex-runner profile:
> >
> >   
> >   
> > org.apache.hadoop
> > hadoop-yarn-client
> > ${hadoop.version}
> > runtime
> >   
> >   
> > org.apache.hadoop
> > hadoop-common
> > ${hadoop.version}
> > runtime
> >   
> >
> > 2) After copying the fat jar to the cluster:
> >
> > java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.
> > WordCount
> > \
> >--inputFile=file:///tmp/input.txt --output=/tmp/counts
> > --embeddedExecution=false --configFile=beam-runners-apex.properties
> > --runner=ApexRunner
> >
> > (this was on a single node cluster, hence the local file path)
> >
> > The quickstart instructions suggest to use *mvn exec:java* instead of
> > *java*
> > - it generally isn't valid to assume that mvn and a build environment
> > exists on the edge node of a YARN cluster.
> >
> >
> >
> > On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:
> >
> > Hi Eugene,
> >
> >>
> >> I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
> >> Cluster(Yarn). Both ran successfully. :)
> >>
> >> Regards,
> >> Nishu
> >>
> >>  >>
> source=link_campaign=sig-email_content=webmail_term=icon>
> >> Virus-free.
> >> www.avast.com
> >>  >>
> source=link_campaign=sig-email_content=webmail_term=link>
> >> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >>
> >> On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
> >> kirpic...@google.com.invalid> wrote:
> >>
> >> Thanks Nishu. So, if I understand correctly, your pipelines were
> >>
> >>>
> >>> running
> >>
> >
> > on
> >>
> >> non-YARN, but you're planning to run with YARN?
> >>>
> >>> I meanwhile was able to get Flink running on Dataproc (YARN), and
> >>>
> >>> validated
> >>
> >> quickstart and game examples.
> >>> At this point we need validation for Spark and Flink non-YARN [I
> >>> think
> >>>
> >>> if
> >>
> >
> > Nishu's runs were non-YARN, they'd give us enough confidence,
> combined
> >>
> >>>
> >>> with
> >>
> >> the success of other validations of Spark and Flink runners?], and
> >>> Apex
> >>>
> >>> on
> >>
> >> YARN. However, it seems that in previous RCs we were not validating
> >>>
> >>> Apex
> >>
> >
> > on
> >>
> >> YARN, only local cluster. Is it needed this time?
> >>>
> >>> On Wed, Nov 22, 2017 at 12:28 PM Nishu 
> wrote:
> >>>
> >>> Hi Eugene,
> >>>
> 
>  No, I didn't 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

I am not an owner or maintainer of the PyPi package, so I'm not sure if
I'll be able to release Python artifacts.

On Fri, Nov 24, 2017 at 10:08 PM, Jean-Baptiste Onofré 
wrote:

> Awesome, Thanks Reuven !
>
> Regards
> JB
>
> On 11/25/2017 07:01 AM, Reuven Lax wrote:
>
>> I'll go ahead and send the RESULT email right now.
>>
>>
>>
>> On Fri, Nov 24, 2017 at 9:56 PM, Jean-Baptiste Onofré 
>> wrote:
>>
>> It's not sync: promoting the artifacts takes some time (at least 30
>>> minutes). So, the artifacts will be on central after a certain time.
>>>
>>> I confirm that it's OK because now the artifacts are on Central:
>>>
>>> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdk
>>> s-java-core/2.2.0/
>>>
>>> By the way, you are promoting the artifacts to Central but I didn't see
>>> any [RESULT] e-mail on the vote thread. You have first to close the vote,
>>> then promote the artifacts, announce the release, etc.
>>>
>>> Regards
>>> JB
>>>
>>> On 11/25/2017 12:43 AM, Reuven Lax wrote:
>>>
>>> Appears to be a problem :)

 I tried publishing the latest artifact from Apache Nexus to Maven
 Central.
 After clicking publish, Nexus claimed that the operation has completed.
 However a look at the Maven Central page (
 https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
 does not show 2.2.0 artifacts, and the staging repository has now
 vanished
 from the Nexus site! Does anyone know what happened here?

 Reuven

 On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:

 +1

>
> Run quickstart with Apex runner in embedded mode and on YARN.
>
> It needed couple tweaks to get there though.
>
> 1) Change quickstart pom.xml apex-runner profile:
>
>   
>   
> org.apache.hadoop
> hadoop-yarn-client
> ${hadoop.version}
> runtime
>   
>   
> org.apache.hadoop
> hadoop-common
> ${hadoop.version}
> runtime
>   
>
> 2) After copying the fat jar to the cluster:
>
> java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.
> WordCount
> \
>--inputFile=file:///tmp/input.txt --output=/tmp/counts
> --embeddedExecution=false --configFile=beam-runners-apex.properties
> --runner=ApexRunner
>
> (this was on a single node cluster, hence the local file path)
>
> The quickstart instructions suggest to use *mvn exec:java* instead of
> *java*
> - it generally isn't valid to assume that mvn and a build environment
> exists on the edge node of a YARN cluster.
>
>
>
> On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:
>
> Hi Eugene,
>
>>
>> I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
>> Cluster(Yarn). Both ran successfully. :)
>>
>> Regards,
>> Nishu
>>
>> > source=link_campaign=sig-email_content=webmail_term=icon>
>> Virus-free.
>> www.avast.com
>> > source=link_campaign=sig-email_content=webmail_term=link>
>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>> On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
>> kirpic...@google.com.invalid> wrote:
>>
>> Thanks Nishu. So, if I understand correctly, your pipelines were
>>
>>>
>>> running
>>
>
> on
>>
>> non-YARN, but you're planning to run with YARN?
>>>
>>> I meanwhile was able to get Flink running on Dataproc (YARN), and
>>>
>>> validated
>>
>> quickstart and game examples.
>>> At this point we need validation for Spark and Flink non-YARN [I
>>> think
>>>
>>> if
>>
>
> Nishu's runs were non-YARN, they'd give us enough confidence, combined
>>
>>>
>>> with
>>
>> the success of other validations of Spark and Flink runners?], and
>>> Apex
>>>
>>> on
>>
>> YARN. However, it seems that in previous RCs we were not validating
>>>
>>> Apex
>>
>
> on
>>
>> YARN, only local cluster. Is it needed this time?
>>>
>>> On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:
>>>
>>> Hi Eugene,
>>>

 No, I didn't try with those instead I have my custom pipeline where

 Kafka
>>>
>>
>> topic is the source. I have defined a Global Window and processing
>>>

 time
>>>
>>
> trigger to read the data. Further it runs some transformation i.e.
>>
>>> GroupByKey and CoGroupByKey. on the windowed collections.
 I was running the same pipeline on direct runner and spark 

Re: [VOTE] Release 2.2.0, release candidate #4

[Jean-Baptiste Onofré]

Awesome, Thanks Reuven !

Regards
JB

On 11/25/2017 07:01 AM, Reuven Lax wrote:

I'll go ahead and send the RESULT email right now.



On Fri, Nov 24, 2017 at 9:56 PM, Jean-Baptiste Onofré 
wrote:


It's not sync: promoting the artifacts takes some time (at least 30
minutes). So, the artifacts will be on central after a certain time.

I confirm that it's OK because now the artifacts are on Central:

http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdk
s-java-core/2.2.0/

By the way, you are promoting the artifacts to Central but I didn't see
any [RESULT] e-mail on the vote thread. You have first to close the vote,
then promote the artifacts, announce the release, etc.

Regards
JB

On 11/25/2017 12:43 AM, Reuven Lax wrote:


Appears to be a problem :)

I tried publishing the latest artifact from Apache Nexus to Maven Central.
After clicking publish, Nexus claimed that the operation has completed.
However a look at the Maven Central page (
https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
does not show 2.2.0 artifacts, and the staging repository has now vanished
from the Nexus site! Does anyone know what happened here?

Reuven

On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:

+1


Run quickstart with Apex runner in embedded mode and on YARN.

It needed couple tweaks to get there though.

1) Change quickstart pom.xml apex-runner profile:

  
  
org.apache.hadoop
hadoop-yarn-client
${hadoop.version}
runtime
  
  
org.apache.hadoop
hadoop-common
${hadoop.version}
runtime
  

2) After copying the fat jar to the cluster:

java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.
WordCount
\
   --inputFile=file:///tmp/input.txt --output=/tmp/counts
--embeddedExecution=false --configFile=beam-runners-apex.properties
--runner=ApexRunner

(this was on a single node cluster, hence the local file path)

The quickstart instructions suggest to use *mvn exec:java* instead of
*java*
- it generally isn't valid to assume that mvn and a build environment
exists on the edge node of a YARN cluster.



On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:

Hi Eugene,


I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
Cluster(Yarn). Both ran successfully. :)

Regards,
Nishu


Virus-free.
www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

Thanks Nishu. So, if I understand correctly, your pipelines were



running



on


non-YARN, but you're planning to run with YARN?

I meanwhile was able to get Flink running on Dataproc (YARN), and


validated


quickstart and game examples.
At this point we need validation for Spark and Flink non-YARN [I think


if



Nishu's runs were non-YARN, they'd give us enough confidence, combined



with


the success of other validations of Spark and Flink runners?], and Apex


on


YARN. However, it seems that in previous RCs we were not validating


Apex



on


YARN, only local cluster. Is it needed this time?

On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:

Hi Eugene,


No, I didn't try with those instead I have my custom pipeline where


Kafka



topic is the source. I have defined a Global Window and processing



time



trigger to read the data. Further it runs some transformation i.e.

GroupByKey and CoGroupByKey. on the windowed collections.
I was running the same pipeline on direct runner and spark runner


earlier..


Today gave it a try with Flink on Yarn.

Best Regards,
Nishu.

<
https://www.avast.com/sig-email?utm_medium=email_


source=link_campaign=sig-email_content=webmail_term=icon




Virus-free.

www.avast.com
<
https://www.avast.com/sig-email?utm_medium=email_


source=link_campaign=sig-email_content=webmail_term=link




<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>


On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

Thanks Nishu! Can you clarify which pipeline you were running?

The validation spreadsheet includes 1) the quickstart and 2) mobile


game



walkthroughs. Was it one of these, or your custom pipeline?


On Wed, Nov 22, 2017 at 10:20 AM Nishu 


wrote:





Hi,


Typo in previous mail.  I meant Flink runner.

Thanks,
Nishu
On Wed, 22 Nov 2017 at 19.17,

Hi,


I build a pipeline using RC 2.2 today and ran with runner on


yarn.



It worked seamlessly for unbounded sources. Couldn’t see any



issues



with



my pipeline so far :)



Thanks,Nishu

On Wed, 22 Nov 2017 at 18.57, Reuven Lax

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

I'll go ahead and send the RESULT email right now.



On Fri, Nov 24, 2017 at 9:56 PM, Jean-Baptiste Onofré 
wrote:

> It's not sync: promoting the artifacts takes some time (at least 30
> minutes). So, the artifacts will be on central after a certain time.
>
> I confirm that it's OK because now the artifacts are on Central:
>
> http://repo.maven.apache.org/maven2/org/apache/beam/beam-sdk
> s-java-core/2.2.0/
>
> By the way, you are promoting the artifacts to Central but I didn't see
> any [RESULT] e-mail on the vote thread. You have first to close the vote,
> then promote the artifacts, announce the release, etc.
>
> Regards
> JB
>
> On 11/25/2017 12:43 AM, Reuven Lax wrote:
>
>> Appears to be a problem :)
>>
>> I tried publishing the latest artifact from Apache Nexus to Maven Central.
>> After clicking publish, Nexus claimed that the operation has completed.
>> However a look at the Maven Central page (
>> https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
>> does not show 2.2.0 artifacts, and the staging repository has now vanished
>> from the Nexus site! Does anyone know what happened here?
>>
>> Reuven
>>
>> On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:
>>
>> +1
>>>
>>> Run quickstart with Apex runner in embedded mode and on YARN.
>>>
>>> It needed couple tweaks to get there though.
>>>
>>> 1) Change quickstart pom.xml apex-runner profile:
>>>
>>>  
>>>  
>>>org.apache.hadoop
>>>hadoop-yarn-client
>>>${hadoop.version}
>>>runtime
>>>  
>>>  
>>>org.apache.hadoop
>>>hadoop-common
>>>${hadoop.version}
>>>runtime
>>>  
>>>
>>> 2) After copying the fat jar to the cluster:
>>>
>>> java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.
>>> WordCount
>>> \
>>>   --inputFile=file:///tmp/input.txt --output=/tmp/counts
>>> --embeddedExecution=false --configFile=beam-runners-apex.properties
>>> --runner=ApexRunner
>>>
>>> (this was on a single node cluster, hence the local file path)
>>>
>>> The quickstart instructions suggest to use *mvn exec:java* instead of
>>> *java*
>>> - it generally isn't valid to assume that mvn and a build environment
>>> exists on the edge node of a YARN cluster.
>>>
>>>
>>>
>>> On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:
>>>
>>> Hi Eugene,

 I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
 Cluster(Yarn). Both ran successfully. :)

 Regards,
 Nishu

 
 Virus-free.
 www.avast.com
 
 <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

 On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
 kirpic...@google.com.invalid> wrote:

 Thanks Nishu. So, if I understand correctly, your pipelines were
>
 running
>>>
 on

> non-YARN, but you're planning to run with YARN?
>
> I meanwhile was able to get Flink running on Dataproc (YARN), and
>
 validated

> quickstart and game examples.
> At this point we need validation for Spark and Flink non-YARN [I think
>
 if
>>>
 Nishu's runs were non-YARN, they'd give us enough confidence, combined
>
 with

> the success of other validations of Spark and Flink runners?], and Apex
>
 on

> YARN. However, it seems that in previous RCs we were not validating
>
 Apex
>>>
 on

> YARN, only local cluster. Is it needed this time?
>
> On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:
>
> Hi Eugene,
>>
>> No, I didn't try with those instead I have my custom pipeline where
>>
> Kafka

> topic is the source. I have defined a Global Window and processing
>>
> time
>>>
 trigger to read the data. Further it runs some transformation i.e.
>> GroupByKey and CoGroupByKey. on the windowed collections.
>> I was running the same pipeline on direct runner and spark runner
>>
> earlier..
>
>> Today gave it a try with Flink on Yarn.
>>
>> Best Regards,
>> Nishu.
>>
>> <
>> https://www.avast.com/sig-email?utm_medium=email_
>>
> source=link_campaign=sig-email_content=webmail_term=icon
>
>>
>>> Virus-free.
>> www.avast.com
>> <
>> https://www.avast.com/sig-email?utm_medium=email_
>>
> source=link_campaign=sig-email_content=webmail_term=link
>
>>
>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>> On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov <
>> kirpic...@google.com.invalid> wrote:
>>
>> Thanks Nishu! Can you clarify which pipeline you were running?

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Appears to be a problem :)

I tried publishing the latest artifact from Apache Nexus to Maven Central.
After clicking publish, Nexus claimed that the operation has completed.
However a look at the Maven Central page (
https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core)
does not show 2.2.0 artifacts, and the staging repository has now vanished
from the Nexus site! Does anyone know what happened here?

Reuven

On Wed, Nov 22, 2017 at 11:04 PM, Thomas Weise  wrote:

> +1
>
> Run quickstart with Apex runner in embedded mode and on YARN.
>
> It needed couple tweaks to get there though.
>
> 1) Change quickstart pom.xml apex-runner profile:
>
> 
> 
>   org.apache.hadoop
>   hadoop-yarn-client
>   ${hadoop.version}
>   runtime
> 
> 
>   org.apache.hadoop
>   hadoop-common
>   ${hadoop.version}
>   runtime
> 
>
> 2) After copying the fat jar to the cluster:
>
> java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.
> WordCount
> \
>  --inputFile=file:///tmp/input.txt --output=/tmp/counts
> --embeddedExecution=false --configFile=beam-runners-apex.properties
> --runner=ApexRunner
>
> (this was on a single node cluster, hence the local file path)
>
> The quickstart instructions suggest to use *mvn exec:java* instead of
> *java*
> - it generally isn't valid to assume that mvn and a build environment
> exists on the edge node of a YARN cluster.
>
>
>
> On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:
>
> > Hi Eugene,
> >
> > I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
> > Cluster(Yarn). Both ran successfully. :)
> >
> > Regards,
> > Nishu
> >
> >  > source=link_campaign=sig-email_content=webmail_term=icon>
> > Virus-free.
> > www.avast.com
> >  > source=link_campaign=sig-email_content=webmail_term=link>
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
> > On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
> > kirpic...@google.com.invalid> wrote:
> >
> > > Thanks Nishu. So, if I understand correctly, your pipelines were
> running
> > on
> > > non-YARN, but you're planning to run with YARN?
> > >
> > > I meanwhile was able to get Flink running on Dataproc (YARN), and
> > validated
> > > quickstart and game examples.
> > > At this point we need validation for Spark and Flink non-YARN [I think
> if
> > > Nishu's runs were non-YARN, they'd give us enough confidence, combined
> > with
> > > the success of other validations of Spark and Flink runners?], and Apex
> > on
> > > YARN. However, it seems that in previous RCs we were not validating
> Apex
> > on
> > > YARN, only local cluster. Is it needed this time?
> > >
> > > On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:
> > >
> > > > Hi Eugene,
> > > >
> > > > No, I didn't try with those instead I have my custom pipeline where
> > Kafka
> > > > topic is the source. I have defined a Global Window and processing
> time
> > > > trigger to read the data. Further it runs some transformation i.e.
> > > > GroupByKey and CoGroupByKey. on the windowed collections.
> > > > I was running the same pipeline on direct runner and spark runner
> > > earlier..
> > > > Today gave it a try with Flink on Yarn.
> > > >
> > > > Best Regards,
> > > > Nishu.
> > > >
> > > > <
> > > > https://www.avast.com/sig-email?utm_medium=email_
> > > source=link_campaign=sig-email_content=webmail_term=icon
> > > > >
> > > > Virus-free.
> > > > www.avast.com
> > > > <
> > > > https://www.avast.com/sig-email?utm_medium=email_
> > > source=link_campaign=sig-email_content=webmail_term=link
> > > > >
> > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> > > >
> > > > On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov <
> > > > kirpic...@google.com.invalid> wrote:
> > > >
> > > > > Thanks Nishu! Can you clarify which pipeline you were running?
> > > > > The validation spreadsheet includes 1) the quickstart and 2) mobile
> > > game
> > > > > walkthroughs. Was it one of these, or your custom pipeline?
> > > > >
> > > > > On Wed, Nov 22, 2017 at 10:20 AM Nishu 
> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Typo in previous mail.  I meant Flink runner.
> > > > > >
> > > > > > Thanks,
> > > > > > Nishu
> > > > > > On Wed, 22 Nov 2017 at 19.17,
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I build a pipeline using RC 2.2 today and ran with runner on
> > yarn.
> > > > > > > It worked seamlessly for unbounded sources. Couldn’t see any
> > issues
> > > > > with
> > > > > > > my pipeline so far :)
> > > > > > >
> > > > > > >
> > > > > > > Thanks,Nishu
> > > > > > >
> > > > > > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax
> >  > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Who is validating Flink and Yarn?
> > > > > > >>
> > > > > > >> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Thomas Weise]

+1

Run quickstart with Apex runner in embedded mode and on YARN.

It needed couple tweaks to get there though.

1) Change quickstart pom.xml apex-runner profile:



  org.apache.hadoop
  hadoop-yarn-client
  ${hadoop.version}
  runtime


  org.apache.hadoop
  hadoop-common
  ${hadoop.version}
  runtime


2) After copying the fat jar to the cluster:

java -cp word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount
\
 --inputFile=file:///tmp/input.txt --output=/tmp/counts
--embeddedExecution=false --configFile=beam-runners-apex.properties
--runner=ApexRunner

(this was on a single node cluster, hence the local file path)

The quickstart instructions suggest to use *mvn exec:java* instead of *java*
- it generally isn't valid to assume that mvn and a build environment
exists on the edge node of a YARN cluster.



On Wed, Nov 22, 2017 at 2:12 PM, Nishu  wrote:

> Hi Eugene,
>
> I ran it on both  standalone flink(non Yarn) and  Flink on HDInsight
> Cluster(Yarn). Both ran successfully. :)
>
> Regards,
> Nishu
>
>  source=link_campaign=sig-email_content=webmail_term=icon>
> Virus-free.
> www.avast.com
>  source=link_campaign=sig-email_content=webmail_term=link>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Wed, Nov 22, 2017 at 9:40 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
> > Thanks Nishu. So, if I understand correctly, your pipelines were running
> on
> > non-YARN, but you're planning to run with YARN?
> >
> > I meanwhile was able to get Flink running on Dataproc (YARN), and
> validated
> > quickstart and game examples.
> > At this point we need validation for Spark and Flink non-YARN [I think if
> > Nishu's runs were non-YARN, they'd give us enough confidence, combined
> with
> > the success of other validations of Spark and Flink runners?], and Apex
> on
> > YARN. However, it seems that in previous RCs we were not validating Apex
> on
> > YARN, only local cluster. Is it needed this time?
> >
> > On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:
> >
> > > Hi Eugene,
> > >
> > > No, I didn't try with those instead I have my custom pipeline where
> Kafka
> > > topic is the source. I have defined a Global Window and processing time
> > > trigger to read the data. Further it runs some transformation i.e.
> > > GroupByKey and CoGroupByKey. on the windowed collections.
> > > I was running the same pipeline on direct runner and spark runner
> > earlier..
> > > Today gave it a try with Flink on Yarn.
> > >
> > > Best Regards,
> > > Nishu.
> > >
> > > <
> > > https://www.avast.com/sig-email?utm_medium=email_
> > source=link_campaign=sig-email_content=webmail_term=icon
> > > >
> > > Virus-free.
> > > www.avast.com
> > > <
> > > https://www.avast.com/sig-email?utm_medium=email_
> > source=link_campaign=sig-email_content=webmail_term=link
> > > >
> > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> > >
> > > On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov <
> > > kirpic...@google.com.invalid> wrote:
> > >
> > > > Thanks Nishu! Can you clarify which pipeline you were running?
> > > > The validation spreadsheet includes 1) the quickstart and 2) mobile
> > game
> > > > walkthroughs. Was it one of these, or your custom pipeline?
> > > >
> > > > On Wed, Nov 22, 2017 at 10:20 AM Nishu  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Typo in previous mail.  I meant Flink runner.
> > > > >
> > > > > Thanks,
> > > > > Nishu
> > > > > On Wed, 22 Nov 2017 at 19.17,
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I build a pipeline using RC 2.2 today and ran with runner on
> yarn.
> > > > > > It worked seamlessly for unbounded sources. Couldn’t see any
> issues
> > > > with
> > > > > > my pipeline so far :)
> > > > > >
> > > > > >
> > > > > > Thanks,Nishu
> > > > > >
> > > > > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax
>  > >
> > > > > wrote:
> > > > > >
> > > > > >> Who is validating Flink and Yarn?
> > > > > >>
> > > > > >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles
> > > >  > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> > > > > >> > kirpic...@google.com.invalid> wrote:
> > > > > >> >
> > > > > >> > > In the verification spreadsheet, I'm not sure I understand
> the
> > > > > >> difference
> > > > > >> > > between the "YARN" and "Standalone cluster/service". Which
> is
> > > > > >> Dataproc?
> > > > > >> > It
> > > > > >> > > definitely uses YARN, but it is also a standalone
> > > cluster/service.
> > > > > >> Does
> > > > > >> > it
> > > > > >> > > count for both?
> > > > > >> > >
> > > > > >> >
> > > > > >> > No, it doesn't. A number of runners have their own non-YARN
> > > cluster
> > > > > >> mode. I

Re: [VOTE] Release 2.2.0, release candidate #4

[Eugene Kirpichov]

Thanks Nishu. So, if I understand correctly, your pipelines were running on
non-YARN, but you're planning to run with YARN?

I meanwhile was able to get Flink running on Dataproc (YARN), and validated
quickstart and game examples.
At this point we need validation for Spark and Flink non-YARN [I think if
Nishu's runs were non-YARN, they'd give us enough confidence, combined with
the success of other validations of Spark and Flink runners?], and Apex on
YARN. However, it seems that in previous RCs we were not validating Apex on
YARN, only local cluster. Is it needed this time?

On Wed, Nov 22, 2017 at 12:28 PM Nishu  wrote:

> Hi Eugene,
>
> No, I didn't try with those instead I have my custom pipeline where Kafka
> topic is the source. I have defined a Global Window and processing time
> trigger to read the data. Further it runs some transformation i.e.
> GroupByKey and CoGroupByKey. on the windowed collections.
> I was running the same pipeline on direct runner and spark runner earlier..
> Today gave it a try with Flink on Yarn.
>
> Best Regards,
> Nishu.
>
> <
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=icon
> >
> Virus-free.
> www.avast.com
> <
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=link
> >
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
> > Thanks Nishu! Can you clarify which pipeline you were running?
> > The validation spreadsheet includes 1) the quickstart and 2) mobile game
> > walkthroughs. Was it one of these, or your custom pipeline?
> >
> > On Wed, Nov 22, 2017 at 10:20 AM Nishu  wrote:
> >
> > > Hi,
> > >
> > > Typo in previous mail.  I meant Flink runner.
> > >
> > > Thanks,
> > > Nishu
> > > On Wed, 22 Nov 2017 at 19.17,
> > >
> > > > Hi,
> > > >
> > > > I build a pipeline using RC 2.2 today and ran with runner on yarn.
> > > > It worked seamlessly for unbounded sources. Couldn’t see any issues
> > with
> > > > my pipeline so far :)
> > > >
> > > >
> > > > Thanks,Nishu
> > > >
> > > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax 
> > > wrote:
> > > >
> > > >> Who is validating Flink and Yarn?
> > > >>
> > > >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles
> >  > > >
> > > >> wrote:
> > > >>
> > > >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> > > >> > kirpic...@google.com.invalid> wrote:
> > > >> >
> > > >> > > In the verification spreadsheet, I'm not sure I understand the
> > > >> difference
> > > >> > > between the "YARN" and "Standalone cluster/service". Which is
> > > >> Dataproc?
> > > >> > It
> > > >> > > definitely uses YARN, but it is also a standalone
> cluster/service.
> > > >> Does
> > > >> > it
> > > >> > > count for both?
> > > >> > >
> > > >> >
> > > >> > No, it doesn't. A number of runners have their own non-YARN
> cluster
> > > >> mode. I
> > > >> > would expect that the launching experience might be different and
> > the
> > > >> > portable container management to differ. If they are identical,
> > > experts
> > > >> in
> > > >> > those systems should feel free to coalesce the rows. Conversely,
> as
> > > >> other
> > > >> > platforms become supported, they could be added or not based on
> > > whether
> > > >> > they are substantively different from a user experience or QA
> point
> > of
> > > >> > view.
> > > >> >
> > > >> > Kenn
> > > >> >
> > > >> >
> > > >> > > Seems now we're missing just Apex and Flink cluster
> verifications.
> > > >> > >
> > > >> > > *though Spark runner took 6x longer to run UserScore, partially
> I
> > > >> guess
> > > >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5
> > > >> workers
> > > >> > > whereas Spark runner used 2 workers). For some reason Spark
> runner
> > > >> chose
> > > >> > > not to split the 10GB input files into chunks.
> > > >> > >
> > > >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax
> >  > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Done
> > > >> > > >
> > > >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> > > >> > > > rober...@google.com.invalid> wrote:
> > > >> > > >
> > > >> > > > > Thanks. You need to re-sign as well.
> > > >> > > > >
> > > >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
> > > >> >  > > >> > > >
> > > >> > > > > wrote:
> > > >> > > > > > FYI these generated files have been removed from the
> source
> > > >> > > > distribution.
> > > >> > > > > >
> > > >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax <
> > re...@google.com
> > > >
> > > >> > > wrote:
> > > >> > > > > >
> > > >> > > > > >> hmmm, I thought I removed those generated files from the
> > zip
> > > >> file
> > > >> > > > before
> > > >> > > > > >> sending this email. Let me check again.
> > > >> > > > > >>
> > > >> > > 

Re: [VOTE] Release 2.2.0, release candidate #4

[Nishu]

Hi Eugene,

No, I didn't try with those instead I have my custom pipeline where Kafka
topic is the source. I have defined a Global Window and processing time
trigger to read the data. Further it runs some transformation i.e.
GroupByKey and CoGroupByKey. on the windowed collections.
I was running the same pipeline on direct runner and spark runner earlier..
Today gave it a try with Flink on Yarn.

Best Regards,
Nishu.


Virus-free.
www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> Thanks Nishu! Can you clarify which pipeline you were running?
> The validation spreadsheet includes 1) the quickstart and 2) mobile game
> walkthroughs. Was it one of these, or your custom pipeline?
>
> On Wed, Nov 22, 2017 at 10:20 AM Nishu  wrote:
>
> > Hi,
> >
> > Typo in previous mail.  I meant Flink runner.
> >
> > Thanks,
> > Nishu
> > On Wed, 22 Nov 2017 at 19.17,
> >
> > > Hi,
> > >
> > > I build a pipeline using RC 2.2 today and ran with runner on yarn.
> > > It worked seamlessly for unbounded sources. Couldn’t see any issues
> with
> > > my pipeline so far :)
> > >
> > >
> > > Thanks,Nishu
> > >
> > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax 
> > wrote:
> > >
> > >> Who is validating Flink and Yarn?
> > >>
> > >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles
>  > >
> > >> wrote:
> > >>
> > >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> > >> > kirpic...@google.com.invalid> wrote:
> > >> >
> > >> > > In the verification spreadsheet, I'm not sure I understand the
> > >> difference
> > >> > > between the "YARN" and "Standalone cluster/service". Which is
> > >> Dataproc?
> > >> > It
> > >> > > definitely uses YARN, but it is also a standalone cluster/service.
> > >> Does
> > >> > it
> > >> > > count for both?
> > >> > >
> > >> >
> > >> > No, it doesn't. A number of runners have their own non-YARN cluster
> > >> mode. I
> > >> > would expect that the launching experience might be different and
> the
> > >> > portable container management to differ. If they are identical,
> > experts
> > >> in
> > >> > those systems should feel free to coalesce the rows. Conversely, as
> > >> other
> > >> > platforms become supported, they could be added or not based on
> > whether
> > >> > they are substantively different from a user experience or QA point
> of
> > >> > view.
> > >> >
> > >> > Kenn
> > >> >
> > >> >
> > >> > > Seems now we're missing just Apex and Flink cluster verifications.
> > >> > >
> > >> > > *though Spark runner took 6x longer to run UserScore, partially I
> > >> guess
> > >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5
> > >> workers
> > >> > > whereas Spark runner used 2 workers). For some reason Spark runner
> > >> chose
> > >> > > not to split the 10GB input files into chunks.
> > >> > >
> > >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax
>  > >
> > >> > > wrote:
> > >> > >
> > >> > > > Done
> > >> > > >
> > >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> > >> > > > rober...@google.com.invalid> wrote:
> > >> > > >
> > >> > > > > Thanks. You need to re-sign as well.
> > >> > > > >
> > >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
> > >> >  > >> > > >
> > >> > > > > wrote:
> > >> > > > > > FYI these generated files have been removed from the source
> > >> > > > distribution.
> > >> > > > > >
> > >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax <
> re...@google.com
> > >
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > >> hmmm, I thought I removed those generated files from the
> zip
> > >> file
> > >> > > > before
> > >> > > > > >> sending this email. Let me check again.
> > >> > > > > >>
> > >> > > > > >> Reuven
> > >> > > > > >>
> > >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > >> > > > > >> rober...@google.com.invalid> wrote:
> > >> > > > > >>
> > >> > > > > >>> The source distribution contains a couple of files not on
> > >> github
> > >> > > > (e.g.
> > >> > > > > >>> folders that were added on master, Python generated
> files).
> > >> The
> > >> > pom
> > >> > > > > >>> files differed only by missing -SNAPSHOT, other than that
> > >> > > presumably
> > >> > > > > >>> the source release should just be "wget
> > >> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip
> "?
> > >> > > > > >>>
> > >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > >> > > > > >>>
> > >> > > > > >>> # OK?
> > >> > > > > >>>
> > >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> > >> > > > > >>>
> > >> > > > > >>> # Expected.
> > >> > > > > >>>
> > >> > > > > >>> Only 

Re: [VOTE] Release 2.2.0, release candidate #4

[Kenneth Knowles]

+1 (binding)

On Wed, Nov 22, 2017 at 11:43 AM, Max Barrios 
wrote:

> +1 (non-binding)
>
> Sent from my iPhone
>
> > On Nov 20, 2017, at 12:47 AM, Jean-Baptiste Onofré 
> wrote:
> >
> > Yeah, I have a Jira about that.
> >
> > You just have to update the existing symlink to point on the new release.
> >
> > I will update the release guide asap.
> >
> > Thanks !
> > Regards
> > JB
> >
> >> On 11/20/2017 09:39 AM, Reuven Lax wrote:
> >> Good point. How do I update that link? I think this step might be
> missing
> >> from our release guide (or I simply can't find it).
> >> Reuven
> >> On Mon, Nov 20, 2017 at 4:24 PM, Jean-Baptiste Onofré 
> >> wrote:
> >>> +1 (binding)
> >>>
> >>> Tested on Beam samples, it looks good.
> >>>
> >>> Checked the legal/license, content. OK for me.
> >>>
> >>> Just a side note: don't forget to update the latest link on
> >>> dist.apache.org.
> >>>
> >>> Thanks
> >>> Regards
> >>> JB
> >>>
> >>>
>  On 11/17/2017 07:08 AM, Reuven Lax wrote:
> 
>  Hi everyone,
> 
>  Please review and vote on the release candidate #4 for the version
> 2.2.0,
>  as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific
> comments)
> 
> 
>  The complete staging area is available for your review, which
> includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to
> dist.apache.org
>  [2],
>  which is signed with the key with fingerprint B98B7708 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.2.0-RC4" [5],
> * website pull request listing the release and publishing the API
>  reference manual [6].
> * Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
>  1.8.0_144.
> * Python artifacts are deployed along with the source release to
> the
>  dist.apache.org [2].
> 
>  The vote will be open for at least 72 hours. It is adopted by majority
>  approval, with at least 3 PMC affirmative votes.
> 
>  Thanks,
>  Reuven
> 
>  [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
>  rojectId=12319527=12341044
>  [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
>  [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>  [4] https://repository.apache.org/content/repositories/orgapache
>  beam-1025/
>  [5] https://github.com/apache/beam/tree/v2.2.0-RC4
>  
>  [6] https://github.com/apache/beam-site/pull/337
> 
> 
> >>> --
> >>> Jean-Baptiste Onofré
> >>> jbono...@apache.org
> >>> http://blog.nanthrax.net
> >>> Talend - http://www.talend.com
> >>>
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>

Re: [VOTE] Release 2.2.0, release candidate #4

[Max Barrios]

+1 (non-binding)

Sent from my iPhone

> On Nov 20, 2017, at 12:47 AM, Jean-Baptiste Onofré  wrote:
> 
> Yeah, I have a Jira about that.
> 
> You just have to update the existing symlink to point on the new release.
> 
> I will update the release guide asap.
> 
> Thanks !
> Regards
> JB
> 
>> On 11/20/2017 09:39 AM, Reuven Lax wrote:
>> Good point. How do I update that link? I think this step might be missing
>> from our release guide (or I simply can't find it).
>> Reuven
>> On Mon, Nov 20, 2017 at 4:24 PM, Jean-Baptiste Onofré 
>> wrote:
>>> +1 (binding)
>>> 
>>> Tested on Beam samples, it looks good.
>>> 
>>> Checked the legal/license, content. OK for me.
>>> 
>>> Just a side note: don't forget to update the latest link on
>>> dist.apache.org.
>>> 
>>> Thanks
>>> Regards
>>> JB
>>> 
>>> 
 On 11/17/2017 07:08 AM, Reuven Lax wrote:
 
 Hi everyone,
 
 Please review and vote on the release candidate #4 for the version 2.2.0,
 as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)
 
 
 The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org
 [2],
 which is signed with the key with fingerprint B98B7708 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.2.0-RC4" [5],
* website pull request listing the release and publishing the API
 reference manual [6].
* Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
 1.8.0_144.
* Python artifacts are deployed along with the source release to the
 dist.apache.org [2].
 
 The vote will be open for at least 72 hours. It is adopted by majority
 approval, with at least 3 PMC affirmative votes.
 
 Thanks,
 Reuven
 
 [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
 rojectId=12319527=12341044
 [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
 [3] https://dist.apache.org/repos/dist/release/beam/KEYS
 [4] https://repository.apache.org/content/repositories/orgapache
 beam-1025/
 [5] https://github.com/apache/beam/tree/v2.2.0-RC4
 
 [6] https://github.com/apache/beam-site/pull/337
 
 
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>> 
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com

Re: [VOTE] Release 2.2.0, release candidate #4

[Eugene Kirpichov]

Thanks Nishu! Can you clarify which pipeline you were running?
The validation spreadsheet includes 1) the quickstart and 2) mobile game
walkthroughs. Was it one of these, or your custom pipeline?

On Wed, Nov 22, 2017 at 10:20 AM Nishu  wrote:

> Hi,
>
> Typo in previous mail.  I meant Flink runner.
>
> Thanks,
> Nishu
> On Wed, 22 Nov 2017 at 19.17,
>
> > Hi,
> >
> > I build a pipeline using RC 2.2 today and ran with runner on yarn.
> > It worked seamlessly for unbounded sources. Couldn’t see any issues with
> > my pipeline so far :)
> >
> >
> > Thanks,Nishu
> >
> > On Wed, 22 Nov 2017 at 18.57, Reuven Lax 
> wrote:
> >
> >> Who is validating Flink and Yarn?
> >>
> >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles  >
> >> wrote:
> >>
> >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> >> > kirpic...@google.com.invalid> wrote:
> >> >
> >> > > In the verification spreadsheet, I'm not sure I understand the
> >> difference
> >> > > between the "YARN" and "Standalone cluster/service". Which is
> >> Dataproc?
> >> > It
> >> > > definitely uses YARN, but it is also a standalone cluster/service.
> >> Does
> >> > it
> >> > > count for both?
> >> > >
> >> >
> >> > No, it doesn't. A number of runners have their own non-YARN cluster
> >> mode. I
> >> > would expect that the launching experience might be different and the
> >> > portable container management to differ. If they are identical,
> experts
> >> in
> >> > those systems should feel free to coalesce the rows. Conversely, as
> >> other
> >> > platforms become supported, they could be added or not based on
> whether
> >> > they are substantively different from a user experience or QA point of
> >> > view.
> >> >
> >> > Kenn
> >> >
> >> >
> >> > > Seems now we're missing just Apex and Flink cluster verifications.
> >> > >
> >> > > *though Spark runner took 6x longer to run UserScore, partially I
> >> guess
> >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5
> >> workers
> >> > > whereas Spark runner used 2 workers). For some reason Spark runner
> >> chose
> >> > > not to split the 10GB input files into chunks.
> >> > >
> >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax  >
> >> > > wrote:
> >> > >
> >> > > > Done
> >> > > >
> >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> >> > > > rober...@google.com.invalid> wrote:
> >> > > >
> >> > > > > Thanks. You need to re-sign as well.
> >> > > > >
> >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
> >> >  >> > > >
> >> > > > > wrote:
> >> > > > > > FYI these generated files have been removed from the source
> >> > > > distribution.
> >> > > > > >
> >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax  >
> >> > > wrote:
> >> > > > > >
> >> > > > > >> hmmm, I thought I removed those generated files from the zip
> >> file
> >> > > > before
> >> > > > > >> sending this email. Let me check again.
> >> > > > > >>
> >> > > > > >> Reuven
> >> > > > > >>
> >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> >> > > > > >> rober...@google.com.invalid> wrote:
> >> > > > > >>
> >> > > > > >>> The source distribution contains a couple of files not on
> >> github
> >> > > > (e.g.
> >> > > > > >>> folders that were added on master, Python generated files).
> >> The
> >> > pom
> >> > > > > >>> files differed only by missing -SNAPSHOT, other than that
> >> > > presumably
> >> > > > > >>> the source release should just be "wget
> >> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> >> > > > > >>>
> >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> >> > > > > >>>
> >> > > > > >>> # OK?
> >> > > > > >>>
> >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> >> > > > > >>>
> >> > > > > >>> # Expected.
> >> > > > > >>>
> >> > > > > >>> Only in beam/: .git
> >> > > > > >>> Only in beam/: .gitattributes
> >> > > > > >>> Only in beam/: .gitignore
> >> > > > > >>>
> >> > > > > >>> # These folders are probably from switching around between
> >> master
> >> > > and
> >> > > > > >>> git branches.
> >> > > > > >>>
> >> > > > > >>> Only in apache-beam-2.2.0: model
> >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples
> >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner
> >> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
> >> > maven-archetypes/examples/src/
> >> > > ma
> >> > > > > >>> in/resources/archetype-resources:
> >> > > > > >>> src
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
> >> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Please update the spreadsheet.

On Wed, Nov 22, 2017 at 10:19 AM, Nishu  wrote:

> Hi,
>
> Typo in previous mail.  I meant Flink runner.
>
> Thanks,
> Nishu
> On Wed, 22 Nov 2017 at 19.17,
>
> > Hi,
> >
> > I build a pipeline using RC 2.2 today and ran with runner on yarn.
> > It worked seamlessly for unbounded sources. Couldn’t see any issues with
> > my pipeline so far :)
> >
> >
> > Thanks,Nishu
> >
> > On Wed, 22 Nov 2017 at 18.57, Reuven Lax 
> wrote:
> >
> >> Who is validating Flink and Yarn?
> >>
> >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles  >
> >> wrote:
> >>
> >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> >> > kirpic...@google.com.invalid> wrote:
> >> >
> >> > > In the verification spreadsheet, I'm not sure I understand the
> >> difference
> >> > > between the "YARN" and "Standalone cluster/service". Which is
> >> Dataproc?
> >> > It
> >> > > definitely uses YARN, but it is also a standalone cluster/service.
> >> Does
> >> > it
> >> > > count for both?
> >> > >
> >> >
> >> > No, it doesn't. A number of runners have their own non-YARN cluster
> >> mode. I
> >> > would expect that the launching experience might be different and the
> >> > portable container management to differ. If they are identical,
> experts
> >> in
> >> > those systems should feel free to coalesce the rows. Conversely, as
> >> other
> >> > platforms become supported, they could be added or not based on
> whether
> >> > they are substantively different from a user experience or QA point of
> >> > view.
> >> >
> >> > Kenn
> >> >
> >> >
> >> > > Seems now we're missing just Apex and Flink cluster verifications.
> >> > >
> >> > > *though Spark runner took 6x longer to run UserScore, partially I
> >> guess
> >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5
> >> workers
> >> > > whereas Spark runner used 2 workers). For some reason Spark runner
> >> chose
> >> > > not to split the 10GB input files into chunks.
> >> > >
> >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax  >
> >> > > wrote:
> >> > >
> >> > > > Done
> >> > > >
> >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> >> > > > rober...@google.com.invalid> wrote:
> >> > > >
> >> > > > > Thanks. You need to re-sign as well.
> >> > > > >
> >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
> >> >  >> > > >
> >> > > > > wrote:
> >> > > > > > FYI these generated files have been removed from the source
> >> > > > distribution.
> >> > > > > >
> >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax  >
> >> > > wrote:
> >> > > > > >
> >> > > > > >> hmmm, I thought I removed those generated files from the zip
> >> file
> >> > > > before
> >> > > > > >> sending this email. Let me check again.
> >> > > > > >>
> >> > > > > >> Reuven
> >> > > > > >>
> >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> >> > > > > >> rober...@google.com.invalid> wrote:
> >> > > > > >>
> >> > > > > >>> The source distribution contains a couple of files not on
> >> github
> >> > > > (e.g.
> >> > > > > >>> folders that were added on master, Python generated files).
> >> The
> >> > pom
> >> > > > > >>> files differed only by missing -SNAPSHOT, other than that
> >> > > presumably
> >> > > > > >>> the source release should just be "wget
> >> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> >> > > > > >>>
> >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> >> > > > > >>>
> >> > > > > >>> # OK?
> >> > > > > >>>
> >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> >> > > > > >>>
> >> > > > > >>> # Expected.
> >> > > > > >>>
> >> > > > > >>> Only in beam/: .git
> >> > > > > >>> Only in beam/: .gitattributes
> >> > > > > >>> Only in beam/: .gitignore
> >> > > > > >>>
> >> > > > > >>> # These folders are probably from switching around between
> >> master
> >> > > and
> >> > > > > >>> git branches.
> >> > > > > >>>
> >> > > > > >>> Only in apache-beam-2.2.0: model
> >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples
> >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner
> >> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
> >> > maven-archetypes/examples/src/
> >> > > ma
> >> > > > > >>> in/resources/archetype-resources:
> >> > > > > >>> src
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
> >> maven-archetypes/examples-
> >> > > java8/
> >> > > > > >>> src/main/resources/archetype-resources:
> >> > > > > >>> src
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java: 

Re: [VOTE] Release 2.2.0, release candidate #4

[Konstantinos Katsiapis]

+1 (non-binding)

Since Beam 2.2 is blocking release of tensorflow.transform
 0.4.

On Wed, Nov 22, 2017 at 10:19 AM, Nishu  wrote:

> Hi,
>
> Typo in previous mail.  I meant Flink runner.
>
> Thanks,
> Nishu
> On Wed, 22 Nov 2017 at 19.17,
>
> > Hi,
> >
> > I build a pipeline using RC 2.2 today and ran with runner on yarn.
> > It worked seamlessly for unbounded sources. Couldn’t see any issues with
> > my pipeline so far :)
> >
> >
> > Thanks,Nishu
> >
> > On Wed, 22 Nov 2017 at 18.57, Reuven Lax 
> wrote:
> >
> >> Who is validating Flink and Yarn?
> >>
> >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles  >
> >> wrote:
> >>
> >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> >> > kirpic...@google.com.invalid> wrote:
> >> >
> >> > > In the verification spreadsheet, I'm not sure I understand the
> >> difference
> >> > > between the "YARN" and "Standalone cluster/service". Which is
> >> Dataproc?
> >> > It
> >> > > definitely uses YARN, but it is also a standalone cluster/service.
> >> Does
> >> > it
> >> > > count for both?
> >> > >
> >> >
> >> > No, it doesn't. A number of runners have their own non-YARN cluster
> >> mode. I
> >> > would expect that the launching experience might be different and the
> >> > portable container management to differ. If they are identical,
> experts
> >> in
> >> > those systems should feel free to coalesce the rows. Conversely, as
> >> other
> >> > platforms become supported, they could be added or not based on
> whether
> >> > they are substantively different from a user experience or QA point of
> >> > view.
> >> >
> >> > Kenn
> >> >
> >> >
> >> > > Seems now we're missing just Apex and Flink cluster verifications.
> >> > >
> >> > > *though Spark runner took 6x longer to run UserScore, partially I
> >> guess
> >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5
> >> workers
> >> > > whereas Spark runner used 2 workers). For some reason Spark runner
> >> chose
> >> > > not to split the 10GB input files into chunks.
> >> > >
> >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax  >
> >> > > wrote:
> >> > >
> >> > > > Done
> >> > > >
> >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> >> > > > rober...@google.com.invalid> wrote:
> >> > > >
> >> > > > > Thanks. You need to re-sign as well.
> >> > > > >
> >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
> >> >  >> > > >
> >> > > > > wrote:
> >> > > > > > FYI these generated files have been removed from the source
> >> > > > distribution.
> >> > > > > >
> >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax  >
> >> > > wrote:
> >> > > > > >
> >> > > > > >> hmmm, I thought I removed those generated files from the zip
> >> file
> >> > > > before
> >> > > > > >> sending this email. Let me check again.
> >> > > > > >>
> >> > > > > >> Reuven
> >> > > > > >>
> >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> >> > > > > >> rober...@google.com.invalid> wrote:
> >> > > > > >>
> >> > > > > >>> The source distribution contains a couple of files not on
> >> github
> >> > > > (e.g.
> >> > > > > >>> folders that were added on master, Python generated files).
> >> The
> >> > pom
> >> > > > > >>> files differed only by missing -SNAPSHOT, other than that
> >> > > presumably
> >> > > > > >>> the source release should just be "wget
> >> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> >> > > > > >>>
> >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> >> > > > > >>>
> >> > > > > >>> # OK?
> >> > > > > >>>
> >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> >> > > > > >>>
> >> > > > > >>> # Expected.
> >> > > > > >>>
> >> > > > > >>> Only in beam/: .git
> >> > > > > >>> Only in beam/: .gitattributes
> >> > > > > >>> Only in beam/: .gitignore
> >> > > > > >>>
> >> > > > > >>> # These folders are probably from switching around between
> >> master
> >> > > and
> >> > > > > >>> git branches.
> >> > > > > >>>
> >> > > > > >>> Only in apache-beam-2.2.0: model
> >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples
> >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner
> >> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
> >> > maven-archetypes/examples/src/
> >> > > ma
> >> > > > > >>> in/resources/archetype-resources:
> >> > > > > >>> src
> >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
> >> maven-archetypes/examples-
> >> > > java8/
> >> > > > > >>> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Nishu]

Hi,

Typo in previous mail.  I meant Flink runner.

Thanks,
Nishu
On Wed, 22 Nov 2017 at 19.17,

> Hi,
>
> I build a pipeline using RC 2.2 today and ran with runner on yarn.
> It worked seamlessly for unbounded sources. Couldn’t see any issues with
> my pipeline so far :)
>
>
> Thanks,Nishu
>
> On Wed, 22 Nov 2017 at 18.57, Reuven Lax  wrote:
>
>> Who is validating Flink and Yarn?
>>
>> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles 
>> wrote:
>>
>> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
>> > kirpic...@google.com.invalid> wrote:
>> >
>> > > In the verification spreadsheet, I'm not sure I understand the
>> difference
>> > > between the "YARN" and "Standalone cluster/service". Which is
>> Dataproc?
>> > It
>> > > definitely uses YARN, but it is also a standalone cluster/service.
>> Does
>> > it
>> > > count for both?
>> > >
>> >
>> > No, it doesn't. A number of runners have their own non-YARN cluster
>> mode. I
>> > would expect that the launching experience might be different and the
>> > portable container management to differ. If they are identical, experts
>> in
>> > those systems should feel free to coalesce the rows. Conversely, as
>> other
>> > platforms become supported, they could be added or not based on whether
>> > they are substantively different from a user experience or QA point of
>> > view.
>> >
>> > Kenn
>> >
>> >
>> > > Seems now we're missing just Apex and Flink cluster verifications.
>> > >
>> > > *though Spark runner took 6x longer to run UserScore, partially I
>> guess
>> > > because it didn't do autoscaling (Dataflow runner ramped up to 5
>> workers
>> > > whereas Spark runner used 2 workers). For some reason Spark runner
>> chose
>> > > not to split the 10GB input files into chunks.
>> > >
>> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax 
>> > > wrote:
>> > >
>> > > > Done
>> > > >
>> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
>> > > > rober...@google.com.invalid> wrote:
>> > > >
>> > > > > Thanks. You need to re-sign as well.
>> > > > >
>> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
>> > > > > >
>> > > > > wrote:
>> > > > > > FYI these generated files have been removed from the source
>> > > > distribution.
>> > > > > >
>> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax 
>> > > wrote:
>> > > > > >
>> > > > > >> hmmm, I thought I removed those generated files from the zip
>> file
>> > > > before
>> > > > > >> sending this email. Let me check again.
>> > > > > >>
>> > > > > >> Reuven
>> > > > > >>
>> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
>> > > > > >> rober...@google.com.invalid> wrote:
>> > > > > >>
>> > > > > >>> The source distribution contains a couple of files not on
>> github
>> > > > (e.g.
>> > > > > >>> folders that were added on master, Python generated files).
>> The
>> > pom
>> > > > > >>> files differed only by missing -SNAPSHOT, other than that
>> > > presumably
>> > > > > >>> the source release should just be "wget
>> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
>> > > > > >>>
>> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
>> > > > > >>>
>> > > > > >>> # OK?
>> > > > > >>>
>> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
>> > > > > >>>
>> > > > > >>> # Expected.
>> > > > > >>>
>> > > > > >>> Only in beam/: .git
>> > > > > >>> Only in beam/: .gitattributes
>> > > > > >>> Only in beam/: .gitignore
>> > > > > >>>
>> > > > > >>> # These folders are probably from switching around between
>> master
>> > > and
>> > > > > >>> git branches.
>> > > > > >>>
>> > > > > >>> Only in apache-beam-2.2.0: model
>> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples
>> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner
>> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
>> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
>> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
>> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
>> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
>> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
>> > maven-archetypes/examples/src/
>> > > ma
>> > > > > >>> in/resources/archetype-resources:
>> > > > > >>> src
>> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
>> maven-archetypes/examples-
>> > > java8/
>> > > > > >>> src/main/resources/archetype-resources:
>> > > > > >>> src
>> > > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
>> > > > > >>>
>> > > > > >>> # Here's the generated protos.
>> > > > > >>>
>> > > > > >>> Only in apache-beam-2.2.0/sdks/python/
>> > apache_beam/portability/api:
>> > > > > >>> beam_artifact_api_pb2_grpc.py
>> > > > > >>> Only in apache-beam-2.2.0/sdks/python/
>> > apache_beam/portability/api:
>> > > > > >>> beam_artifact_api_pb2.py
>> > > > > >>> Only in 

Re: [VOTE] Release 2.2.0, release candidate #4

[Nishu]

Hi,

I build a pipeline using RC 2.2 today and ran with runner on yarn.
It worked seamlessly for unbounded sources. Couldn’t see any issues with my
pipeline so far :)


Thanks,Nishu

On Wed, 22 Nov 2017 at 18.57, Reuven Lax  wrote:

> Who is validating Flink and Yarn?
>
> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles 
> wrote:
>
> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> > kirpic...@google.com.invalid> wrote:
> >
> > > In the verification spreadsheet, I'm not sure I understand the
> difference
> > > between the "YARN" and "Standalone cluster/service". Which is Dataproc?
> > It
> > > definitely uses YARN, but it is also a standalone cluster/service. Does
> > it
> > > count for both?
> > >
> >
> > No, it doesn't. A number of runners have their own non-YARN cluster
> mode. I
> > would expect that the launching experience might be different and the
> > portable container management to differ. If they are identical, experts
> in
> > those systems should feel free to coalesce the rows. Conversely, as other
> > platforms become supported, they could be added or not based on whether
> > they are substantively different from a user experience or QA point of
> > view.
> >
> > Kenn
> >
> >
> > > Seems now we're missing just Apex and Flink cluster verifications.
> > >
> > > *though Spark runner took 6x longer to run UserScore, partially I guess
> > > because it didn't do autoscaling (Dataflow runner ramped up to 5
> workers
> > > whereas Spark runner used 2 workers). For some reason Spark runner
> chose
> > > not to split the 10GB input files into chunks.
> > >
> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax 
> > > wrote:
> > >
> > > > Done
> > > >
> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> > > > rober...@google.com.invalid> wrote:
> > > >
> > > > > Thanks. You need to re-sign as well.
> > > > >
> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
> >  > > >
> > > > > wrote:
> > > > > > FYI these generated files have been removed from the source
> > > > distribution.
> > > > > >
> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax 
> > > wrote:
> > > > > >
> > > > > >> hmmm, I thought I removed those generated files from the zip
> file
> > > > before
> > > > > >> sending this email. Let me check again.
> > > > > >>
> > > > > >> Reuven
> > > > > >>
> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > > > > >> rober...@google.com.invalid> wrote:
> > > > > >>
> > > > > >>> The source distribution contains a couple of files not on
> github
> > > > (e.g.
> > > > > >>> folders that were added on master, Python generated files). The
> > pom
> > > > > >>> files differed only by missing -SNAPSHOT, other than that
> > > presumably
> > > > > >>> the source release should just be "wget
> > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> > > > > >>>
> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > > > > >>>
> > > > > >>> # OK?
> > > > > >>>
> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> > > > > >>>
> > > > > >>> # Expected.
> > > > > >>>
> > > > > >>> Only in beam/: .git
> > > > > >>> Only in beam/: .gitattributes
> > > > > >>> Only in beam/: .gitignore
> > > > > >>>
> > > > > >>> # These folders are probably from switching around between
> master
> > > and
> > > > > >>> git branches.
> > > > > >>>
> > > > > >>> Only in apache-beam-2.2.0: model
> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples
> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner
> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > > > > >>> Only in apache-beam-2.2.0/sdks/java/
> > maven-archetypes/examples/src/
> > > ma
> > > > > >>> in/resources/archetype-resources:
> > > > > >>> src
> > > > > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> > > java8/
> > > > > >>> src/main/resources/archetype-resources:
> > > > > >>> src
> > > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> > > > > >>>
> > > > > >>> # Here's the generated protos.
> > > > > >>>
> > > > > >>> Only in apache-beam-2.2.0/sdks/python/
> > apache_beam/portability/api:
> > > > > >>> beam_artifact_api_pb2_grpc.py
> > > > > >>> Only in apache-beam-2.2.0/sdks/python/
> > apache_beam/portability/api:
> > > > > >>> beam_artifact_api_pb2.py
> > > > > >>> Only in apache-beam-2.2.0/sdks/python/
> > apache_beam/portability/api:
> > > > > >>> beam_fn_api_pb2_grpc.py
> > > > > >>> Only in apache-beam-2.2.0/sdks/python/
> > apache_beam/portability/api:
> > > > > >>> beam_fn_api_pb2.py
> > > > > >>> Only in apache-beam-2.2.0/sdks/python/
> > 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Who is validating Flink and Yarn?

On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles 
wrote:

> On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
> > In the verification spreadsheet, I'm not sure I understand the difference
> > between the "YARN" and "Standalone cluster/service". Which is Dataproc?
> It
> > definitely uses YARN, but it is also a standalone cluster/service. Does
> it
> > count for both?
> >
>
> No, it doesn't. A number of runners have their own non-YARN cluster mode. I
> would expect that the launching experience might be different and the
> portable container management to differ. If they are identical, experts in
> those systems should feel free to coalesce the rows. Conversely, as other
> platforms become supported, they could be added or not based on whether
> they are substantively different from a user experience or QA point of
> view.
>
> Kenn
>
>
> > Seems now we're missing just Apex and Flink cluster verifications.
> >
> > *though Spark runner took 6x longer to run UserScore, partially I guess
> > because it didn't do autoscaling (Dataflow runner ramped up to 5 workers
> > whereas Spark runner used 2 workers). For some reason Spark runner chose
> > not to split the 10GB input files into chunks.
> >
> > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax 
> > wrote:
> >
> > > Done
> > >
> > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> > > rober...@google.com.invalid> wrote:
> > >
> > > > Thanks. You need to re-sign as well.
> > > >
> > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
>  > >
> > > > wrote:
> > > > > FYI these generated files have been removed from the source
> > > distribution.
> > > > >
> > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax 
> > wrote:
> > > > >
> > > > >> hmmm, I thought I removed those generated files from the zip file
> > > before
> > > > >> sending this email. Let me check again.
> > > > >>
> > > > >> Reuven
> > > > >>
> > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > > > >> rober...@google.com.invalid> wrote:
> > > > >>
> > > > >>> The source distribution contains a couple of files not on github
> > > (e.g.
> > > > >>> folders that were added on master, Python generated files). The
> pom
> > > > >>> files differed only by missing -SNAPSHOT, other than that
> > presumably
> > > > >>> the source release should just be "wget
> > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> > > > >>>
> > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > > > >>>
> > > > >>> # OK?
> > > > >>>
> > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> > > > >>>
> > > > >>> # Expected.
> > > > >>>
> > > > >>> Only in beam/: .git
> > > > >>> Only in beam/: .gitattributes
> > > > >>> Only in beam/: .gitignore
> > > > >>>
> > > > >>> # These folders are probably from switching around between master
> > and
> > > > >>> git branches.
> > > > >>>
> > > > >>> Only in apache-beam-2.2.0: model
> > > > >>> Only in apache-beam-2.2.0/runners/flink: examples
> > > > >>> Only in apache-beam-2.2.0/runners/flink: runner
> > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > > > >>> Only in apache-beam-2.2.0/sdks/java/
> maven-archetypes/examples/src/
> > ma
> > > > >>> in/resources/archetype-resources:
> > > > >>> src
> > > > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> > java8/
> > > > >>> src/main/resources/archetype-resources:
> > > > >>> src
> > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> > > > >>>
> > > > >>> # Here's the generated protos.
> > > > >>>
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_artifact_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_artifact_api_pb2.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_fn_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_fn_api_pb2.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_job_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_job_api_pb2.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_provision_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_provision_api_pb2.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Kenneth Knowles]

On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> In the verification spreadsheet, I'm not sure I understand the difference
> between the "YARN" and "Standalone cluster/service". Which is Dataproc? It
> definitely uses YARN, but it is also a standalone cluster/service. Does it
> count for both?
>

No, it doesn't. A number of runners have their own non-YARN cluster mode. I
would expect that the launching experience might be different and the
portable container management to differ. If they are identical, experts in
those systems should feel free to coalesce the rows. Conversely, as other
platforms become supported, they could be added or not based on whether
they are substantively different from a user experience or QA point of view.

Kenn


> Seems now we're missing just Apex and Flink cluster verifications.
>
> *though Spark runner took 6x longer to run UserScore, partially I guess
> because it didn't do autoscaling (Dataflow runner ramped up to 5 workers
> whereas Spark runner used 2 workers). For some reason Spark runner chose
> not to split the 10GB input files into chunks.
>
> On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax 
> wrote:
>
> > Done
> >
> > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> > rober...@google.com.invalid> wrote:
> >
> > > Thanks. You need to re-sign as well.
> > >
> > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax  >
> > > wrote:
> > > > FYI these generated files have been removed from the source
> > distribution.
> > > >
> > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax 
> wrote:
> > > >
> > > >> hmmm, I thought I removed those generated files from the zip file
> > before
> > > >> sending this email. Let me check again.
> > > >>
> > > >> Reuven
> > > >>
> > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > > >> rober...@google.com.invalid> wrote:
> > > >>
> > > >>> The source distribution contains a couple of files not on github
> > (e.g.
> > > >>> folders that were added on master, Python generated files). The pom
> > > >>> files differed only by missing -SNAPSHOT, other than that
> presumably
> > > >>> the source release should just be "wget
> > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> > > >>>
> > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > > >>>
> > > >>> # OK?
> > > >>>
> > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> > > >>>
> > > >>> # Expected.
> > > >>>
> > > >>> Only in beam/: .git
> > > >>> Only in beam/: .gitattributes
> > > >>> Only in beam/: .gitignore
> > > >>>
> > > >>> # These folders are probably from switching around between master
> and
> > > >>> git branches.
> > > >>>
> > > >>> Only in apache-beam-2.2.0: model
> > > >>> Only in apache-beam-2.2.0/runners/flink: examples
> > > >>> Only in apache-beam-2.2.0/runners/flink: runner
> > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/
> ma
> > > >>> in/resources/archetype-resources:
> > > >>> src
> > > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> java8/
> > > >>> src/main/resources/archetype-resources:
> > > >>> src
> > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> > > >>>
> > > >>> # Here's the generated protos.
> > > >>>
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_artifact_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_artifact_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_fn_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_fn_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_job_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_job_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_provision_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_provision_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_runner_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_runner_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> endpoints_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> endpoints_pb2.py
> > > >>> Only in 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

We now have 3 PMC votes for yes. I'm just waiting for Flink runner
validation (still missing on the spreadsheet) before finalizing the release.

On Tue, Nov 21, 2017 at 9:39 AM, Robert Bradshaw <
rober...@google.com.invalid> wrote:

> +1 (binding)
>
> On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
> > I've additionally verified the game examples against Dataflow runner, and
> > verified UserScore and Leaderboard against Spark on Dataproc (YARN). All
> of
> > these worked*
> > So my +1 stands.
> >
> > In the verification spreadsheet, I'm not sure I understand the difference
> > between the "YARN" and "Standalone cluster/service". Which is Dataproc?
> It
> > definitely uses YARN, but it is also a standalone cluster/service. Does
> it
> > count for both?
> >
> > Seems now we're missing just Apex and Flink cluster verifications.
> >
> > *though Spark runner took 6x longer to run UserScore, partially I guess
> > because it didn't do autoscaling (Dataflow runner ramped up to 5 workers
> > whereas Spark runner used 2 workers). For some reason Spark runner chose
> > not to split the 10GB input files into chunks.
> >
> > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax 
> > wrote:
> >
> > > Done
> > >
> > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> > > rober...@google.com.invalid> wrote:
> > >
> > > > Thanks. You need to re-sign as well.
> > > >
> > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax
>  > >
> > > > wrote:
> > > > > FYI these generated files have been removed from the source
> > > distribution.
> > > > >
> > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax 
> > wrote:
> > > > >
> > > > >> hmmm, I thought I removed those generated files from the zip file
> > > before
> > > > >> sending this email. Let me check again.
> > > > >>
> > > > >> Reuven
> > > > >>
> > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > > > >> rober...@google.com.invalid> wrote:
> > > > >>
> > > > >>> The source distribution contains a couple of files not on github
> > > (e.g.
> > > > >>> folders that were added on master, Python generated files). The
> pom
> > > > >>> files differed only by missing -SNAPSHOT, other than that
> > presumably
> > > > >>> the source release should just be "wget
> > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> > > > >>>
> > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > > > >>>
> > > > >>> # OK?
> > > > >>>
> > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> > > > >>>
> > > > >>> # Expected.
> > > > >>>
> > > > >>> Only in beam/: .git
> > > > >>> Only in beam/: .gitattributes
> > > > >>> Only in beam/: .gitignore
> > > > >>>
> > > > >>> # These folders are probably from switching around between master
> > and
> > > > >>> git branches.
> > > > >>>
> > > > >>> Only in apache-beam-2.2.0: model
> > > > >>> Only in apache-beam-2.2.0/runners/flink: examples
> > > > >>> Only in apache-beam-2.2.0/runners/flink: runner
> > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > > > >>> Only in apache-beam-2.2.0/sdks/java/
> maven-archetypes/examples/src/
> > ma
> > > > >>> in/resources/archetype-resources:
> > > > >>> src
> > > > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> > java8/
> > > > >>> src/main/resources/archetype-resources:
> > > > >>> src
> > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> > > > >>>
> > > > >>> # Here's the generated protos.
> > > > >>>
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_artifact_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_artifact_api_pb2.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_fn_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_fn_api_pb2.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_job_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_job_api_pb2.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_provision_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_provision_api_pb2.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> beam_runner_api_pb2_grpc.py
> > > > >>> Only in apache-beam-2.2.0/sdks/python/
> apache_beam/portability/api:
> > > > >>> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Robert Bradshaw]

+1 (binding)

On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> I've additionally verified the game examples against Dataflow runner, and
> verified UserScore and Leaderboard against Spark on Dataproc (YARN). All of
> these worked*
> So my +1 stands.
>
> In the verification spreadsheet, I'm not sure I understand the difference
> between the "YARN" and "Standalone cluster/service". Which is Dataproc? It
> definitely uses YARN, but it is also a standalone cluster/service. Does it
> count for both?
>
> Seems now we're missing just Apex and Flink cluster verifications.
>
> *though Spark runner took 6x longer to run UserScore, partially I guess
> because it didn't do autoscaling (Dataflow runner ramped up to 5 workers
> whereas Spark runner used 2 workers). For some reason Spark runner chose
> not to split the 10GB input files into chunks.
>
> On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax 
> wrote:
>
> > Done
> >
> > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> > rober...@google.com.invalid> wrote:
> >
> > > Thanks. You need to re-sign as well.
> > >
> > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax  >
> > > wrote:
> > > > FYI these generated files have been removed from the source
> > distribution.
> > > >
> > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax 
> wrote:
> > > >
> > > >> hmmm, I thought I removed those generated files from the zip file
> > before
> > > >> sending this email. Let me check again.
> > > >>
> > > >> Reuven
> > > >>
> > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > > >> rober...@google.com.invalid> wrote:
> > > >>
> > > >>> The source distribution contains a couple of files not on github
> > (e.g.
> > > >>> folders that were added on master, Python generated files). The pom
> > > >>> files differed only by missing -SNAPSHOT, other than that
> presumably
> > > >>> the source release should just be "wget
> > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> > > >>>
> > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > > >>>
> > > >>> # OK?
> > > >>>
> > > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> > > >>>
> > > >>> # Expected.
> > > >>>
> > > >>> Only in beam/: .git
> > > >>> Only in beam/: .gitattributes
> > > >>> Only in beam/: .gitignore
> > > >>>
> > > >>> # These folders are probably from switching around between master
> and
> > > >>> git branches.
> > > >>>
> > > >>> Only in apache-beam-2.2.0: model
> > > >>> Only in apache-beam-2.2.0/runners/flink: examples
> > > >>> Only in apache-beam-2.2.0/runners/flink: runner
> > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/
> ma
> > > >>> in/resources/archetype-resources:
> > > >>> src
> > > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> java8/
> > > >>> src/main/resources/archetype-resources:
> > > >>> src
> > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> > > >>>
> > > >>> # Here's the generated protos.
> > > >>>
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_artifact_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_artifact_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_fn_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_fn_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_job_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_job_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_provision_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_provision_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_runner_api_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> beam_runner_api_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> endpoints_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> endpoints_pb2.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> standard_window_fns_pb2_grpc.py
> > > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > >>> standard_window_fns_pb2.py
> > > >>>
> > > >>> And some other sdist 

Re: [VOTE] Release 2.2.0, release candidate #4

[Eugene Kirpichov]

I've additionally verified the game examples against Dataflow runner, and
verified UserScore and Leaderboard against Spark on Dataproc (YARN). All of
these worked*
So my +1 stands.

In the verification spreadsheet, I'm not sure I understand the difference
between the "YARN" and "Standalone cluster/service". Which is Dataproc? It
definitely uses YARN, but it is also a standalone cluster/service. Does it
count for both?

Seems now we're missing just Apex and Flink cluster verifications.

*though Spark runner took 6x longer to run UserScore, partially I guess
because it didn't do autoscaling (Dataflow runner ramped up to 5 workers
whereas Spark runner used 2 workers). For some reason Spark runner chose
not to split the 10GB input files into chunks.

On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax  wrote:

> Done
>
> On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
> rober...@google.com.invalid> wrote:
>
> > Thanks. You need to re-sign as well.
> >
> > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax 
> > wrote:
> > > FYI these generated files have been removed from the source
> distribution.
> > >
> > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax  wrote:
> > >
> > >> hmmm, I thought I removed those generated files from the zip file
> before
> > >> sending this email. Let me check again.
> > >>
> > >> Reuven
> > >>
> > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > >> rober...@google.com.invalid> wrote:
> > >>
> > >>> The source distribution contains a couple of files not on github
> (e.g.
> > >>> folders that were added on master, Python generated files). The pom
> > >>> files differed only by missing -SNAPSHOT, other than that presumably
> > >>> the source release should just be "wget
> > >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> > >>>
> > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > >>>
> > >>> # OK?
> > >>>
> > >>> Only in apache-beam-2.2.0: DEPENDENCIES
> > >>>
> > >>> # Expected.
> > >>>
> > >>> Only in beam/: .git
> > >>> Only in beam/: .gitattributes
> > >>> Only in beam/: .gitignore
> > >>>
> > >>> # These folders are probably from switching around between master and
> > >>> git branches.
> > >>>
> > >>> Only in apache-beam-2.2.0: model
> > >>> Only in apache-beam-2.2.0/runners/flink: examples
> > >>> Only in apache-beam-2.2.0/runners/flink: runner
> > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/ma
> > >>> in/resources/archetype-resources:
> > >>> src
> > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-java8/
> > >>> src/main/resources/archetype-resources:
> > >>> src
> > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> > >>>
> > >>> # Here's the generated protos.
> > >>>
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_artifact_api_pb2_grpc.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_artifact_api_pb2.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_fn_api_pb2_grpc.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_fn_api_pb2.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_job_api_pb2_grpc.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_job_api_pb2.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_provision_api_pb2_grpc.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_provision_api_pb2.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_runner_api_pb2_grpc.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> beam_runner_api_pb2.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> endpoints_pb2_grpc.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> endpoints_pb2.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> standard_window_fns_pb2_grpc.py
> > >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > >>> standard_window_fns_pb2.py
> > >>>
> > >>> And some other sdist generated Python files.
> > >>>
> > >>> Only in apache-beam-2.2.0/sdks/python: .eggs
> > >>> Only in apache-beam-2.2.0/sdks/python: LICENSE
> > >>> Only in apache-beam-2.2.0/sdks/python: NOTICE
> > >>> Only in apache-beam-2.2.0/sdks/python: README.md
> > >>>
> > >>> Presumably we should just purge these files from the rc?
> > >>>
> > >>>
> > >>> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Done

On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw <
rober...@google.com.invalid> wrote:

> Thanks. You need to re-sign as well.
>
> On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax 
> wrote:
> > FYI these generated files have been removed from the source distribution.
> >
> > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax  wrote:
> >
> >> hmmm, I thought I removed those generated files from the zip file before
> >> sending this email. Let me check again.
> >>
> >> Reuven
> >>
> >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> >> rober...@google.com.invalid> wrote:
> >>
> >>> The source distribution contains a couple of files not on github (e.g.
> >>> folders that were added on master, Python generated files). The pom
> >>> files differed only by missing -SNAPSHOT, other than that presumably
> >>> the source release should just be "wget
> >>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
> >>>
> >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> >>>
> >>> # OK?
> >>>
> >>> Only in apache-beam-2.2.0: DEPENDENCIES
> >>>
> >>> # Expected.
> >>>
> >>> Only in beam/: .git
> >>> Only in beam/: .gitattributes
> >>> Only in beam/: .gitignore
> >>>
> >>> # These folders are probably from switching around between master and
> >>> git branches.
> >>>
> >>> Only in apache-beam-2.2.0: model
> >>> Only in apache-beam-2.2.0/runners/flink: examples
> >>> Only in apache-beam-2.2.0/runners/flink: runner
> >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/ma
> >>> in/resources/archetype-resources:
> >>> src
> >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-java8/
> >>> src/main/resources/archetype-resources:
> >>> src
> >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> >>>
> >>> # Here's the generated protos.
> >>>
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_artifact_api_pb2_grpc.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_artifact_api_pb2.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_fn_api_pb2_grpc.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_fn_api_pb2.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_job_api_pb2_grpc.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_job_api_pb2.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_provision_api_pb2_grpc.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_provision_api_pb2.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_runner_api_pb2_grpc.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> beam_runner_api_pb2.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> endpoints_pb2_grpc.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> endpoints_pb2.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> standard_window_fns_pb2_grpc.py
> >>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> >>> standard_window_fns_pb2.py
> >>>
> >>> And some other sdist generated Python files.
> >>>
> >>> Only in apache-beam-2.2.0/sdks/python: .eggs
> >>> Only in apache-beam-2.2.0/sdks/python: LICENSE
> >>> Only in apache-beam-2.2.0/sdks/python: NOTICE
> >>> Only in apache-beam-2.2.0/sdks/python: README.md
> >>>
> >>> Presumably we should just purge these files from the rc?
> >>>
> >>>
> >>> FWIW, the Python tarball looks fine.
> >>>
> >>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov
> >>>  wrote:
> >>> > How can I specify a dependency on the staged RC? E.g. I'm trying to
> >>> > validate the quickstart per
> >>> > https://beam.apache.org/get-started/quickstart-java/ and specifying
> >>> version
> >>> > 2.2.0 doesn't work I suppose because it's not released yet. Should I
> >>> pass
> >>> > some command-line flag to mvn to make it fetch the version from the
> >>> staging
> >>> > area?
> >>> >
> >>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik  >
> >>> > wrote:
> >>> >
> >>> >> Its open to all, its just that there are binding votes and
> non-binding
> >>> >> votes.
> >>> >>
> >>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
> >>> >> valen...@google.com.invalid> wrote:
> >>> >>
> >>> >> > I have a process question: is the vote open for committers only or
> >>> for
> >>> >> all
> >>> >> > contributors?
> >>> >> >
> >>> >> > On Fri, Nov 17, 

Re: [VOTE] Release 2.2.0, release candidate #4

[Ahmet Altay]

+1

I verified the python quick start on Windows. I could not verify the
documentation changes because the staged version expired.


On Mon, Nov 20, 2017 at 12:08 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> Thanks Luke. I was able to validate quickstart on Dataflow and on Spark
> cluster (using Cloud Dataproc). So +1 from me so far.
>
> On Sun, Nov 19, 2017 at 4:28 PM Lukasz Cwik 
> wrote:
>
> > Eugene, you can setup your ~/.m2/settings.xml to point to the repository
> > containing the release candidate.
> > 
> >   
> >
> >  release-repo
> >  
> >true
> >  
> >  
> >
> >  Release 2.2.0 RC4
> >  Release 2.2.0 RC4
> >  
> > https://repository.apache.org/content/repositories/orgapachebeam-1025/
> > 
> >
> >  
> >
> >   
> > 
> >
> > The URL for the release candidate is always part of the vote e-mail.
> > For more details about having multiple repositories, take a look at
> > https://maven.apache.org/guides/mini/guide-multiple-repositories.html
> >
> > On Fri, Nov 17, 2017 at 5:09 PM, Reuven Lax 
> > wrote:
> >
> > > hmmm, I thought I removed those generated files from the zip file
> before
> > > sending this email. Let me check again.
> > >
> > > Reuven
> > >
> > > On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > > rober...@google.com.invalid> wrote:
> > >
> > > > The source distribution contains a couple of files not on github
> (e.g.
> > > > folders that were added on master, Python generated files). The pom
> > > > files differed only by missing -SNAPSHOT, other than that presumably
> > > > the source release should just be "wget
> > > > https://github.com/apache/beam/archive/release-2.2.0.zip;?
> > > >
> > > > diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > > >
> > > > # OK?
> > > >
> > > > Only in apache-beam-2.2.0: DEPENDENCIES
> > > >
> > > > # Expected.
> > > >
> > > > Only in beam/: .git
> > > > Only in beam/: .gitattributes
> > > > Only in beam/: .gitignore
> > > >
> > > > # These folders are probably from switching around between master and
> > > > git branches.
> > > >
> > > > Only in apache-beam-2.2.0: model
> > > > Only in apache-beam-2.2.0/runners/flink: examples
> > > > Only in apache-beam-2.2.0/runners/flink: runner
> > > > Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > > > Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > > > Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > > > Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > > > Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > > > Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/
> > > > main/resources/archetype-resources:
> > > > src
> > > > Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> > > > java8/src/main/resources/archetype-resources:
> > > > src
> > > > Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> > > >
> > > > # Here's the generated protos.
> > > >
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_artifact_api_pb2_grpc.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_artifact_api_pb2.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_fn_api_pb2_grpc.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_fn_api_pb2.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_job_api_pb2_grpc.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_job_api_pb2.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_provision_api_pb2_grpc.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_provision_api_pb2.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_runner_api_pb2_grpc.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > beam_runner_api_pb2.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > endpoints_pb2_grpc.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > endpoints_pb2.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > standard_window_fns_pb2_grpc.py
> > > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > > standard_window_fns_pb2.py
> > > >
> > > > And some other sdist generated Python files.
> > > >
> > > > Only in apache-beam-2.2.0/sdks/python: .eggs
> > > > Only in apache-beam-2.2.0/sdks/python: LICENSE
> > > > Only in apache-beam-2.2.0/sdks/python: NOTICE
> > > > Only in apache-beam-2.2.0/sdks/python: README.md
> > > >
> > > > Presumably we should just purge these files from the rc?
> > > >
> > > >
> > > > FWIW, the Python tarball looks fine.
> > > >
> > > > On Fri, Nov 17, 2017 at 

Re: [VOTE] Release 2.2.0, release candidate #4

[Eugene Kirpichov]

Thanks Luke. I was able to validate quickstart on Dataflow and on Spark
cluster (using Cloud Dataproc). So +1 from me so far.

On Sun, Nov 19, 2017 at 4:28 PM Lukasz Cwik 
wrote:

> Eugene, you can setup your ~/.m2/settings.xml to point to the repository
> containing the release candidate.
> 
>   
>
>  release-repo
>  
>true
>  
>  
>
>  Release 2.2.0 RC4
>  Release 2.2.0 RC4
>  
> https://repository.apache.org/content/repositories/orgapachebeam-1025/
> 
>
>  
>
>   
> 
>
> The URL for the release candidate is always part of the vote e-mail.
> For more details about having multiple repositories, take a look at
> https://maven.apache.org/guides/mini/guide-multiple-repositories.html
>
> On Fri, Nov 17, 2017 at 5:09 PM, Reuven Lax 
> wrote:
>
> > hmmm, I thought I removed those generated files from the zip file before
> > sending this email. Let me check again.
> >
> > Reuven
> >
> > On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> > rober...@google.com.invalid> wrote:
> >
> > > The source distribution contains a couple of files not on github (e.g.
> > > folders that were added on master, Python generated files). The pom
> > > files differed only by missing -SNAPSHOT, other than that presumably
> > > the source release should just be "wget
> > > https://github.com/apache/beam/archive/release-2.2.0.zip;?
> > >
> > > diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> > >
> > > # OK?
> > >
> > > Only in apache-beam-2.2.0: DEPENDENCIES
> > >
> > > # Expected.
> > >
> > > Only in beam/: .git
> > > Only in beam/: .gitattributes
> > > Only in beam/: .gitignore
> > >
> > > # These folders are probably from switching around between master and
> > > git branches.
> > >
> > > Only in apache-beam-2.2.0: model
> > > Only in apache-beam-2.2.0/runners/flink: examples
> > > Only in apache-beam-2.2.0/runners/flink: runner
> > > Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > > Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > > Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > > Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > > Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > > Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/
> > > main/resources/archetype-resources:
> > > src
> > > Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> > > java8/src/main/resources/archetype-resources:
> > > src
> > > Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> > >
> > > # Here's the generated protos.
> > >
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_artifact_api_pb2_grpc.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_artifact_api_pb2.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_fn_api_pb2_grpc.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_fn_api_pb2.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_job_api_pb2_grpc.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_job_api_pb2.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_provision_api_pb2_grpc.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_provision_api_pb2.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_runner_api_pb2_grpc.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > beam_runner_api_pb2.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > endpoints_pb2_grpc.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > endpoints_pb2.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > standard_window_fns_pb2_grpc.py
> > > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > > standard_window_fns_pb2.py
> > >
> > > And some other sdist generated Python files.
> > >
> > > Only in apache-beam-2.2.0/sdks/python: .eggs
> > > Only in apache-beam-2.2.0/sdks/python: LICENSE
> > > Only in apache-beam-2.2.0/sdks/python: NOTICE
> > > Only in apache-beam-2.2.0/sdks/python: README.md
> > >
> > > Presumably we should just purge these files from the rc?
> > >
> > >
> > > FWIW, the Python tarball looks fine.
> > >
> > > On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov
> > >  wrote:
> > > > How can I specify a dependency on the staged RC? E.g. I'm trying to
> > > > validate the quickstart per
> > > > https://beam.apache.org/get-started/quickstart-java/ and specifying
> > > version
> > > > 2.2.0 doesn't work I suppose because it's not released yet. Should I
> > pass
> > > > some command-line flag to mvn to make it fetch the version from the
> > > staging
> > > > area?
> > > >
> > 

Re: [VOTE] Release 2.2.0, release candidate #4

[Robert Bradshaw]

Thanks. You need to re-sign as well.

On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax  wrote:
> FYI these generated files have been removed from the source distribution.
>
> On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax  wrote:
>
>> hmmm, I thought I removed those generated files from the zip file before
>> sending this email. Let me check again.
>>
>> Reuven
>>
>> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
>> rober...@google.com.invalid> wrote:
>>
>>> The source distribution contains a couple of files not on github (e.g.
>>> folders that were added on master, Python generated files). The pom
>>> files differed only by missing -SNAPSHOT, other than that presumably
>>> the source release should just be "wget
>>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
>>>
>>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
>>>
>>> # OK?
>>>
>>> Only in apache-beam-2.2.0: DEPENDENCIES
>>>
>>> # Expected.
>>>
>>> Only in beam/: .git
>>> Only in beam/: .gitattributes
>>> Only in beam/: .gitignore
>>>
>>> # These folders are probably from switching around between master and
>>> git branches.
>>>
>>> Only in apache-beam-2.2.0: model
>>> Only in apache-beam-2.2.0/runners/flink: examples
>>> Only in apache-beam-2.2.0/runners/flink: runner
>>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
>>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
>>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
>>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
>>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
>>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/ma
>>> in/resources/archetype-resources:
>>> src
>>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-java8/
>>> src/main/resources/archetype-resources:
>>> src
>>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
>>>
>>> # Here's the generated protos.
>>>
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_artifact_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_artifact_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_fn_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_fn_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_job_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_job_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_provision_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_provision_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_runner_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_runner_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> endpoints_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> endpoints_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> standard_window_fns_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> standard_window_fns_pb2.py
>>>
>>> And some other sdist generated Python files.
>>>
>>> Only in apache-beam-2.2.0/sdks/python: .eggs
>>> Only in apache-beam-2.2.0/sdks/python: LICENSE
>>> Only in apache-beam-2.2.0/sdks/python: NOTICE
>>> Only in apache-beam-2.2.0/sdks/python: README.md
>>>
>>> Presumably we should just purge these files from the rc?
>>>
>>>
>>> FWIW, the Python tarball looks fine.
>>>
>>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov
>>>  wrote:
>>> > How can I specify a dependency on the staged RC? E.g. I'm trying to
>>> > validate the quickstart per
>>> > https://beam.apache.org/get-started/quickstart-java/ and specifying
>>> version
>>> > 2.2.0 doesn't work I suppose because it's not released yet. Should I
>>> pass
>>> > some command-line flag to mvn to make it fetch the version from the
>>> staging
>>> > area?
>>> >
>>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik 
>>> > wrote:
>>> >
>>> >> Its open to all, its just that there are binding votes and non-binding
>>> >> votes.
>>> >>
>>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
>>> >> valen...@google.com.invalid> wrote:
>>> >>
>>> >> > I have a process question: is the vote open for committers only or
>>> for
>>> >> all
>>> >> > contributors?
>>> >> >
>>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik
>>> 
>>> >> > wrote:
>>> >> >
>>> >> > > +1, Approve the release
>>> >> > >
>>> >> > > I have verified the wordcount quickstart on the Apache Beam website
>>> >> using
>>> >> > > Apex, DirectRunner, Flink & Spark on Linux.
>>> >> > >
>>> >> > > The Gearpump runner is yet to have a 

Re: [VOTE] Release 2.2.0, release candidate #4

[Jean-Baptiste Onofré]

Yeah, I have a Jira about that.

You just have to update the existing symlink to point on the new release.

I will update the release guide asap.

Thanks !
Regards
JB

On 11/20/2017 09:39 AM, Reuven Lax wrote:

Good point. How do I update that link? I think this step might be missing
from our release guide (or I simply can't find it).

Reuven

On Mon, Nov 20, 2017 at 4:24 PM, Jean-Baptiste Onofré 
wrote:


+1 (binding)

Tested on Beam samples, it looks good.

Checked the legal/license, content. OK for me.

Just a side note: don't forget to update the latest link on
dist.apache.org.

Thanks
Regards
JB


On 11/17/2017 07:08 AM, Reuven Lax wrote:


Hi everyone,

Please review and vote on the release candidate #4 for the version 2.2.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org
[2],
which is signed with the key with fingerprint B98B7708 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.2.0-RC4" [5],
* website pull request listing the release and publishing the API
reference manual [6].
* Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
1.8.0_144.
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Reuven

[1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
rojectId=12319527=12341044
[2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapache
beam-1025/
[5] https://github.com/apache/beam/tree/v2.2.0-RC4

[6] https://github.com/apache/beam-site/pull/337



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com





--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Or is this simply a matter of copying from the dev to the release
repository?

On Mon, Nov 20, 2017 at 4:39 PM, Reuven Lax  wrote:

> Good point. How do I update that link? I think this step might be missing
> from our release guide (or I simply can't find it).
>
> Reuven
>
> On Mon, Nov 20, 2017 at 4:24 PM, Jean-Baptiste Onofré 
> wrote:
>
>> +1 (binding)
>>
>> Tested on Beam samples, it looks good.
>>
>> Checked the legal/license, content. OK for me.
>>
>> Just a side note: don't forget to update the latest link on
>> dist.apache.org.
>>
>> Thanks
>> Regards
>> JB
>>
>>
>> On 11/17/2017 07:08 AM, Reuven Lax wrote:
>>
>>> Hi everyone,
>>>
>>> Please review and vote on the release candidate #4 for the version 2.2.0,
>>> as follows:
>>>[ ] +1, Approve the release
>>>[ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> The complete staging area is available for your review, which includes:
>>>* JIRA release notes [1],
>>>* the official Apache source release to be deployed to
>>> dist.apache.org [2],
>>> which is signed with the key with fingerprint B98B7708 [3],
>>>* all artifacts to be deployed to the Maven Central Repository [4],
>>>* source code tag "v2.2.0-RC4" [5],
>>>* website pull request listing the release and publishing the API
>>> reference manual [6].
>>>* Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
>>> 1.8.0_144.
>>>* Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> Reuven
>>>
>>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
>>> rojectId=12319527=12341044
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4] https://repository.apache.org/content/repositories/orgapache
>>> beam-1025/
>>> [5] https://github.com/apache/beam/tree/v2.2.0-RC4
>>> 
>>> [6] https://github.com/apache/beam-site/pull/337
>>>
>>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

Good point. How do I update that link? I think this step might be missing
from our release guide (or I simply can't find it).

Reuven

On Mon, Nov 20, 2017 at 4:24 PM, Jean-Baptiste Onofré 
wrote:

> +1 (binding)
>
> Tested on Beam samples, it looks good.
>
> Checked the legal/license, content. OK for me.
>
> Just a side note: don't forget to update the latest link on
> dist.apache.org.
>
> Thanks
> Regards
> JB
>
>
> On 11/17/2017 07:08 AM, Reuven Lax wrote:
>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #4 for the version 2.2.0,
>> as follows:
>>[ ] +1, Approve the release
>>[ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> The complete staging area is available for your review, which includes:
>>* JIRA release notes [1],
>>* the official Apache source release to be deployed to dist.apache.org
>> [2],
>> which is signed with the key with fingerprint B98B7708 [3],
>>* all artifacts to be deployed to the Maven Central Repository [4],
>>* source code tag "v2.2.0-RC4" [5],
>>* website pull request listing the release and publishing the API
>> reference manual [6].
>>* Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
>> 1.8.0_144.
>>* Python artifacts are deployed along with the source release to the
>> dist.apache.org [2].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Reuven
>>
>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
>> rojectId=12319527=12341044
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapache
>> beam-1025/
>> [5] https://github.com/apache/beam/tree/v2.2.0-RC4
>> 
>> [6] https://github.com/apache/beam-site/pull/337
>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [VOTE] Release 2.2.0, release candidate #4

[Jean-Baptiste Onofré]

+1 (binding)

Tested on Beam samples, it looks good.

Checked the legal/license, content. OK for me.

Just a side note: don't forget to update the latest link on dist.apache.org.

Thanks
Regards
JB

On 11/17/2017 07:08 AM, Reuven Lax wrote:

Hi everyone,

Please review and vote on the release candidate #4 for the version 2.2.0,
as follows:
   [ ] +1, Approve the release
   [ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
   * JIRA release notes [1],
   * the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint B98B7708 [3],
   * all artifacts to be deployed to the Maven Central Repository [4],
   * source code tag "v2.2.0-RC4" [5],
   * website pull request listing the release and publishing the API
reference manual [6].
   * Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
1.8.0_144.
   * Python artifacts are deployed along with the source release to the
dist.apache.org [2].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Reuven

[1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
rojectId=12319527=12341044
[2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1025/
[5] https://github.com/apache/beam/tree/v2.2.0-RC4

[6] https://github.com/apache/beam-site/pull/337



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.2.0, release candidate #4

[Romain Manni-Bucau]

Tested on some custom transforms and integrations,
+1 (non-binding),
thanks for the perseverance Reuven!

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-20 9:14 GMT+01:00 Reuven Lax :
> FYI these generated files have been removed from the source distribution.
>
> On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax  wrote:
>
>> hmmm, I thought I removed those generated files from the zip file before
>> sending this email. Let me check again.
>>
>> Reuven
>>
>> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
>> rober...@google.com.invalid> wrote:
>>
>>> The source distribution contains a couple of files not on github (e.g.
>>> folders that were added on master, Python generated files). The pom
>>> files differed only by missing -SNAPSHOT, other than that presumably
>>> the source release should just be "wget
>>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
>>>
>>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
>>>
>>> # OK?
>>>
>>> Only in apache-beam-2.2.0: DEPENDENCIES
>>>
>>> # Expected.
>>>
>>> Only in beam/: .git
>>> Only in beam/: .gitattributes
>>> Only in beam/: .gitignore
>>>
>>> # These folders are probably from switching around between master and
>>> git branches.
>>>
>>> Only in apache-beam-2.2.0: model
>>> Only in apache-beam-2.2.0/runners/flink: examples
>>> Only in apache-beam-2.2.0/runners/flink: runner
>>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
>>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
>>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
>>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
>>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
>>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/ma
>>> in/resources/archetype-resources:
>>> src
>>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-java8/
>>> src/main/resources/archetype-resources:
>>> src
>>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
>>>
>>> # Here's the generated protos.
>>>
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_artifact_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_artifact_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_fn_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_fn_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_job_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_job_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_provision_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_provision_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_runner_api_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> beam_runner_api_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> endpoints_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> endpoints_pb2.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> standard_window_fns_pb2_grpc.py
>>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>>> standard_window_fns_pb2.py
>>>
>>> And some other sdist generated Python files.
>>>
>>> Only in apache-beam-2.2.0/sdks/python: .eggs
>>> Only in apache-beam-2.2.0/sdks/python: LICENSE
>>> Only in apache-beam-2.2.0/sdks/python: NOTICE
>>> Only in apache-beam-2.2.0/sdks/python: README.md
>>>
>>> Presumably we should just purge these files from the rc?
>>>
>>>
>>> FWIW, the Python tarball looks fine.
>>>
>>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov
>>>  wrote:
>>> > How can I specify a dependency on the staged RC? E.g. I'm trying to
>>> > validate the quickstart per
>>> > https://beam.apache.org/get-started/quickstart-java/ and specifying
>>> version
>>> > 2.2.0 doesn't work I suppose because it's not released yet. Should I
>>> pass
>>> > some command-line flag to mvn to make it fetch the version from the
>>> staging
>>> > area?
>>> >
>>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik 
>>> > wrote:
>>> >
>>> >> Its open to all, its just that there are binding votes and non-binding
>>> >> votes.
>>> >>
>>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
>>> >> valen...@google.com.invalid> wrote:
>>> >>
>>> >> > I have a process question: is the vote open for committers only or
>>> for
>>> >> all
>>> >> > contributors?
>>> >> >
>>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik
>>> 
>>> >> > wrote:
>>> >> >
>>> >> > > +1, Approve the release
>>> >> > >
>>> >> > > I have verified the wordcount quickstart on the Apache Beam website
>>> >> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

FYI these generated files have been removed from the source distribution.

On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax  wrote:

> hmmm, I thought I removed those generated files from the zip file before
> sending this email. Let me check again.
>
> Reuven
>
> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> rober...@google.com.invalid> wrote:
>
>> The source distribution contains a couple of files not on github (e.g.
>> folders that were added on master, Python generated files). The pom
>> files differed only by missing -SNAPSHOT, other than that presumably
>> the source release should just be "wget
>> https://github.com/apache/beam/archive/release-2.2.0.zip;?
>>
>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
>>
>> # OK?
>>
>> Only in apache-beam-2.2.0: DEPENDENCIES
>>
>> # Expected.
>>
>> Only in beam/: .git
>> Only in beam/: .gitattributes
>> Only in beam/: .gitignore
>>
>> # These folders are probably from switching around between master and
>> git branches.
>>
>> Only in apache-beam-2.2.0: model
>> Only in apache-beam-2.2.0/runners/flink: examples
>> Only in apache-beam-2.2.0/runners/flink: runner
>> Only in apache-beam-2.2.0/runners/gearpump: jarstore
>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
>> Only in apache-beam-2.2.0/sdks/java/io: hdfs
>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/ma
>> in/resources/archetype-resources:
>> src
>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-java8/
>> src/main/resources/archetype-resources:
>> src
>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
>>
>> # Here's the generated protos.
>>
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_artifact_api_pb2_grpc.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_artifact_api_pb2.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_fn_api_pb2_grpc.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_fn_api_pb2.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_job_api_pb2_grpc.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_job_api_pb2.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_provision_api_pb2_grpc.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_provision_api_pb2.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_runner_api_pb2_grpc.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> beam_runner_api_pb2.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> endpoints_pb2_grpc.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> endpoints_pb2.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> standard_window_fns_pb2_grpc.py
>> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
>> standard_window_fns_pb2.py
>>
>> And some other sdist generated Python files.
>>
>> Only in apache-beam-2.2.0/sdks/python: .eggs
>> Only in apache-beam-2.2.0/sdks/python: LICENSE
>> Only in apache-beam-2.2.0/sdks/python: NOTICE
>> Only in apache-beam-2.2.0/sdks/python: README.md
>>
>> Presumably we should just purge these files from the rc?
>>
>>
>> FWIW, the Python tarball looks fine.
>>
>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov
>>  wrote:
>> > How can I specify a dependency on the staged RC? E.g. I'm trying to
>> > validate the quickstart per
>> > https://beam.apache.org/get-started/quickstart-java/ and specifying
>> version
>> > 2.2.0 doesn't work I suppose because it's not released yet. Should I
>> pass
>> > some command-line flag to mvn to make it fetch the version from the
>> staging
>> > area?
>> >
>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik 
>> > wrote:
>> >
>> >> Its open to all, its just that there are binding votes and non-binding
>> >> votes.
>> >>
>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
>> >> valen...@google.com.invalid> wrote:
>> >>
>> >> > I have a process question: is the vote open for committers only or
>> for
>> >> all
>> >> > contributors?
>> >> >
>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik
>> 
>> >> > wrote:
>> >> >
>> >> > > +1, Approve the release
>> >> > >
>> >> > > I have verified the wordcount quickstart on the Apache Beam website
>> >> using
>> >> > > Apex, DirectRunner, Flink & Spark on Linux.
>> >> > >
>> >> > > The Gearpump runner is yet to have a quickstart listed on our
>> website.
>> >> > > Adding the quickstart is already represented by this existing
>> issue:
>> >> > > https://issues.apache.org/jira/browse/BEAM-2692
>> >> > >
>> >> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev <
>> 

Re: [VOTE] Release 2.2.0, release candidate #4

[Lukasz Cwik]

Eugene, you can setup your ~/.m2/settings.xml to point to the repository
containing the release candidate.

  
   
 release-repo
 
   true
 
 
   
 Release 2.2.0 RC4
 Release 2.2.0 RC4
 
https://repository.apache.org/content/repositories/orgapachebeam-1025/
   
 
   
  


The URL for the release candidate is always part of the vote e-mail.
For more details about having multiple repositories, take a look at
https://maven.apache.org/guides/mini/guide-multiple-repositories.html

On Fri, Nov 17, 2017 at 5:09 PM, Reuven Lax 
wrote:

> hmmm, I thought I removed those generated files from the zip file before
> sending this email. Let me check again.
>
> Reuven
>
> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
> rober...@google.com.invalid> wrote:
>
> > The source distribution contains a couple of files not on github (e.g.
> > folders that were added on master, Python generated files). The pom
> > files differed only by missing -SNAPSHOT, other than that presumably
> > the source release should just be "wget
> > https://github.com/apache/beam/archive/release-2.2.0.zip;?
> >
> > diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
> >
> > # OK?
> >
> > Only in apache-beam-2.2.0: DEPENDENCIES
> >
> > # Expected.
> >
> > Only in beam/: .git
> > Only in beam/: .gitattributes
> > Only in beam/: .gitignore
> >
> > # These folders are probably from switching around between master and
> > git branches.
> >
> > Only in apache-beam-2.2.0: model
> > Only in apache-beam-2.2.0/runners/flink: examples
> > Only in apache-beam-2.2.0/runners/flink: runner
> > Only in apache-beam-2.2.0/runners/gearpump: jarstore
> > Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> > Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> > Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> > Only in apache-beam-2.2.0/sdks/java/io: hdfs
> > Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/
> > main/resources/archetype-resources:
> > src
> > Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> > java8/src/main/resources/archetype-resources:
> > src
> > Only in apache-beam-2.2.0/sdks/java: microbenchmarks
> >
> > # Here's the generated protos.
> >
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_artifact_api_pb2_grpc.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_artifact_api_pb2.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_fn_api_pb2_grpc.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_fn_api_pb2.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_job_api_pb2_grpc.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_job_api_pb2.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_provision_api_pb2_grpc.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_provision_api_pb2.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_runner_api_pb2_grpc.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > beam_runner_api_pb2.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > endpoints_pb2_grpc.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > endpoints_pb2.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > standard_window_fns_pb2_grpc.py
> > Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> > standard_window_fns_pb2.py
> >
> > And some other sdist generated Python files.
> >
> > Only in apache-beam-2.2.0/sdks/python: .eggs
> > Only in apache-beam-2.2.0/sdks/python: LICENSE
> > Only in apache-beam-2.2.0/sdks/python: NOTICE
> > Only in apache-beam-2.2.0/sdks/python: README.md
> >
> > Presumably we should just purge these files from the rc?
> >
> >
> > FWIW, the Python tarball looks fine.
> >
> > On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov
> >  wrote:
> > > How can I specify a dependency on the staged RC? E.g. I'm trying to
> > > validate the quickstart per
> > > https://beam.apache.org/get-started/quickstart-java/ and specifying
> > version
> > > 2.2.0 doesn't work I suppose because it's not released yet. Should I
> pass
> > > some command-line flag to mvn to make it fetch the version from the
> > staging
> > > area?
> > >
> > > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik 
> > > wrote:
> > >
> > >> Its open to all, its just that there are binding votes and non-binding
> > >> votes.
> > >>
> > >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
> > >> valen...@google.com.invalid> wrote:
> > >>
> > >> > I have a process question: is the vote open for committers only or
> for
> > >> all
> > >> > contributors?
> > >> >
> > >> > On Fri, Nov 17, 2017 at 

Re: [VOTE] Release 2.2.0, release candidate #4

[Reuven Lax]

hmmm, I thought I removed those generated files from the zip file before
sending this email. Let me check again.

Reuven

On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw <
rober...@google.com.invalid> wrote:

> The source distribution contains a couple of files not on github (e.g.
> folders that were added on master, Python generated files). The pom
> files differed only by missing -SNAPSHOT, other than that presumably
> the source release should just be "wget
> https://github.com/apache/beam/archive/release-2.2.0.zip;?
>
> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml
>
> # OK?
>
> Only in apache-beam-2.2.0: DEPENDENCIES
>
> # Expected.
>
> Only in beam/: .git
> Only in beam/: .gitattributes
> Only in beam/: .gitignore
>
> # These folders are probably from switching around between master and
> git branches.
>
> Only in apache-beam-2.2.0: model
> Only in apache-beam-2.2.0/runners/flink: examples
> Only in apache-beam-2.2.0/runners/flink: runner
> Only in apache-beam-2.2.0/runners/gearpump: jarstore
> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
> Only in apache-beam-2.2.0/sdks/java/extensions: sketching
> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
> Only in apache-beam-2.2.0/sdks/java/io: hdfs
> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/
> main/resources/archetype-resources:
> src
> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples-
> java8/src/main/resources/archetype-resources:
> src
> Only in apache-beam-2.2.0/sdks/java: microbenchmarks
>
> # Here's the generated protos.
>
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_artifact_api_pb2_grpc.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_artifact_api_pb2.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_fn_api_pb2_grpc.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_fn_api_pb2.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_job_api_pb2_grpc.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_job_api_pb2.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_provision_api_pb2_grpc.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_provision_api_pb2.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_runner_api_pb2_grpc.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> beam_runner_api_pb2.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> endpoints_pb2_grpc.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> endpoints_pb2.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> standard_window_fns_pb2_grpc.py
> Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
> standard_window_fns_pb2.py
>
> And some other sdist generated Python files.
>
> Only in apache-beam-2.2.0/sdks/python: .eggs
> Only in apache-beam-2.2.0/sdks/python: LICENSE
> Only in apache-beam-2.2.0/sdks/python: NOTICE
> Only in apache-beam-2.2.0/sdks/python: README.md
>
> Presumably we should just purge these files from the rc?
>
>
> FWIW, the Python tarball looks fine.
>
> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov
>  wrote:
> > How can I specify a dependency on the staged RC? E.g. I'm trying to
> > validate the quickstart per
> > https://beam.apache.org/get-started/quickstart-java/ and specifying
> version
> > 2.2.0 doesn't work I suppose because it's not released yet. Should I pass
> > some command-line flag to mvn to make it fetch the version from the
> staging
> > area?
> >
> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik 
> > wrote:
> >
> >> Its open to all, its just that there are binding votes and non-binding
> >> votes.
> >>
> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
> >> valen...@google.com.invalid> wrote:
> >>
> >> > I have a process question: is the vote open for committers only or for
> >> all
> >> > contributors?
> >> >
> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik  >
> >> > wrote:
> >> >
> >> > > +1, Approve the release
> >> > >
> >> > > I have verified the wordcount quickstart on the Apache Beam website
> >> using
> >> > > Apex, DirectRunner, Flink & Spark on Linux.
> >> > >
> >> > > The Gearpump runner is yet to have a quickstart listed on our
> website.
> >> > > Adding the quickstart is already represented by this existing issue:
> >> > > https://issues.apache.org/jira/browse/BEAM-2692
> >> > >
> >> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev <
> >> > > valen...@google.com.invalid> wrote:
> >> > >
> >> > > > I have verified: SHA & MD5 signatures of Python artifacts in [2],
> and
> >> > > > checked Python side of the validation checklist on Linux.
> >> > > >
> >> > > > There is one known issue in UserScore example for Dataflow 

Re: [VOTE] Release 2.2.0, release candidate #4

[Robert Bradshaw]

The source distribution contains a couple of files not on github (e.g.
folders that were added on master, Python generated files). The pom
files differed only by missing -SNAPSHOT, other than that presumably
the source release should just be "wget
https://github.com/apache/beam/archive/release-2.2.0.zip;?

diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml

# OK?

Only in apache-beam-2.2.0: DEPENDENCIES

# Expected.

Only in beam/: .git
Only in beam/: .gitattributes
Only in beam/: .gitignore

# These folders are probably from switching around between master and
git branches.

Only in apache-beam-2.2.0: model
Only in apache-beam-2.2.0/runners/flink: examples
Only in apache-beam-2.2.0/runners/flink: runner
Only in apache-beam-2.2.0/runners/gearpump: jarstore
Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core
Only in apache-beam-2.2.0/sdks/java/extensions: sketching
Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests
Only in apache-beam-2.2.0/sdks/java/io: hdfs
Only in 
apache-beam-2.2.0/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources:
src
Only in 
apache-beam-2.2.0/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources:
src
Only in apache-beam-2.2.0/sdks/java: microbenchmarks

# Here's the generated protos.

Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_artifact_api_pb2_grpc.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_artifact_api_pb2.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_fn_api_pb2_grpc.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_fn_api_pb2.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_job_api_pb2_grpc.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_job_api_pb2.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_provision_api_pb2_grpc.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_provision_api_pb2.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_runner_api_pb2_grpc.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
beam_runner_api_pb2.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
endpoints_pb2_grpc.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
endpoints_pb2.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
standard_window_fns_pb2_grpc.py
Only in apache-beam-2.2.0/sdks/python/apache_beam/portability/api:
standard_window_fns_pb2.py

And some other sdist generated Python files.

Only in apache-beam-2.2.0/sdks/python: .eggs
Only in apache-beam-2.2.0/sdks/python: LICENSE
Only in apache-beam-2.2.0/sdks/python: NOTICE
Only in apache-beam-2.2.0/sdks/python: README.md

Presumably we should just purge these files from the rc?


FWIW, the Python tarball looks fine.

On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov
 wrote:
> How can I specify a dependency on the staged RC? E.g. I'm trying to
> validate the quickstart per
> https://beam.apache.org/get-started/quickstart-java/ and specifying version
> 2.2.0 doesn't work I suppose because it's not released yet. Should I pass
> some command-line flag to mvn to make it fetch the version from the staging
> area?
>
> On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik 
> wrote:
>
>> Its open to all, its just that there are binding votes and non-binding
>> votes.
>>
>> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
>> valen...@google.com.invalid> wrote:
>>
>> > I have a process question: is the vote open for committers only or for
>> all
>> > contributors?
>> >
>> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik 
>> > wrote:
>> >
>> > > +1, Approve the release
>> > >
>> > > I have verified the wordcount quickstart on the Apache Beam website
>> using
>> > > Apex, DirectRunner, Flink & Spark on Linux.
>> > >
>> > > The Gearpump runner is yet to have a quickstart listed on our website.
>> > > Adding the quickstart is already represented by this existing issue:
>> > > https://issues.apache.org/jira/browse/BEAM-2692
>> > >
>> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev <
>> > > valen...@google.com.invalid> wrote:
>> > >
>> > > > I have verified: SHA & MD5 signatures of Python artifacts in [2], and
>> > > > checked Python side of the validation checklist on Linux.
>> > > >
>> > > > There is one known issue in UserScore example for Dataflow runner.
>> The
>> > > > issue has been fixed on master branch and does not require a
>> > cherry-pick
>> > > at
>> > > > this point. A workaround is to pass --save_main_session pipeline
>> > option.
>> > > >
>> > > > RC4 looks good to me so far.
>> > > >
>> > > >
>> > > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles
>> > > > > >
>> > > > wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > Following up on past 

Re: [VOTE] Release 2.2.0, release candidate #4

[Eugene Kirpichov]

How can I specify a dependency on the staged RC? E.g. I'm trying to
validate the quickstart per
https://beam.apache.org/get-started/quickstart-java/ and specifying version
2.2.0 doesn't work I suppose because it's not released yet. Should I pass
some command-line flag to mvn to make it fetch the version from the staging
area?

On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik 
wrote:

> Its open to all, its just that there are binding votes and non-binding
> votes.
>
> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
> valen...@google.com.invalid> wrote:
>
> > I have a process question: is the vote open for committers only or for
> all
> > contributors?
> >
> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik 
> > wrote:
> >
> > > +1, Approve the release
> > >
> > > I have verified the wordcount quickstart on the Apache Beam website
> using
> > > Apex, DirectRunner, Flink & Spark on Linux.
> > >
> > > The Gearpump runner is yet to have a quickstart listed on our website.
> > > Adding the quickstart is already represented by this existing issue:
> > > https://issues.apache.org/jira/browse/BEAM-2692
> > >
> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev <
> > > valen...@google.com.invalid> wrote:
> > >
> > > > I have verified: SHA & MD5 signatures of Python artifacts in [2], and
> > > > checked Python side of the validation checklist on Linux.
> > > >
> > > > There is one known issue in UserScore example for Dataflow runner.
> The
> > > > issue has been fixed on master branch and does not require a
> > cherry-pick
> > > at
> > > > this point. A workaround is to pass --save_main_session pipeline
> > option.
> > > >
> > > > RC4 looks good to me so far.
> > > >
> > > >
> > > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles
> >  > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Following up on past discussions and
> > > > > https://issues.apache.org/jira/browse/BEAM-1189 I have prepared a
> > > > > spreadsheet so we can sign up for validation steps that must be
> done
> > > by a
> > > > > human.
> > > > >
> > > > > The spreadsheet for 2.2.0 is at
> > > > > https://s.apache.org/beam-2.2.0-release-validation. Everyone can
> > edit,
> > > > so
> > > > > it is easy to sign up or add new rows.
> > > > >
> > > > > FYI the template is at
> https://s.apache.org/beam-release-validation.
> > > > After
> > > > > this release, I will update the rows in the template to match
> > whatever
> > > we
> > > > > have added to the validation for 2.2.0.
> > > > >
> > > > > Kenn
> > > > >
> > > > >
> > > > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax
> >  > > >
> > > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Please review and vote on the release candidate #4 for the
> version
> > > > 2.2.0,
> > > > > > as follows:
> > > > > >   [ ] +1, Approve the release
> > > > > >   [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > >
> > > > > >
> > > > > > The complete staging area is available for your review, which
> > > includes:
> > > > > >   * JIRA release notes [1],
> > > > > >   * the official Apache source release to be deployed to
> > > > dist.apache.org
> > > > > > [2],
> > > > > > which is signed with the key with fingerprint B98B7708 [3],
> > > > > >   * all artifacts to be deployed to the Maven Central Repository
> > [4],
> > > > > >   * source code tag "v2.2.0-RC4" [5],
> > > > > >   * website pull request listing the release and publishing the
> API
> > > > > > reference manual [6].
> > > > > >   * Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle
> > JDK
> > > > > > 1.8.0_144.
> > > > > >   * Python artifacts are deployed along with the source release
> to
> > > the
> > > > > > dist.apache.org [2].
> > > > > >
> > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > >
> > > > > > Thanks,
> > > > > > Reuven
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
> > > > > > rojectId=12319527=12341044
> > > > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
> > > > > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > > > > [4] https://repository.apache.org/content/repositories/
> > > > > orgapachebeam-1025/
> > > > > > [5] https://github.com/apache/beam/tree/v2.2.0-RC4
> > > > > > 
> > > > > > [6] https://github.com/apache/beam-site/pull/337
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release 2.2.0, release candidate #4

[Lukasz Cwik]

Its open to all, its just that there are binding votes and non-binding
votes.

On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev <
valen...@google.com.invalid> wrote:

> I have a process question: is the vote open for committers only or for all
> contributors?
>
> On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik 
> wrote:
>
> > +1, Approve the release
> >
> > I have verified the wordcount quickstart on the Apache Beam website using
> > Apex, DirectRunner, Flink & Spark on Linux.
> >
> > The Gearpump runner is yet to have a quickstart listed on our website.
> > Adding the quickstart is already represented by this existing issue:
> > https://issues.apache.org/jira/browse/BEAM-2692
> >
> > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev <
> > valen...@google.com.invalid> wrote:
> >
> > > I have verified: SHA & MD5 signatures of Python artifacts in [2], and
> > > checked Python side of the validation checklist on Linux.
> > >
> > > There is one known issue in UserScore example for Dataflow runner. The
> > > issue has been fixed on master branch and does not require a
> cherry-pick
> > at
> > > this point. A workaround is to pass --save_main_session pipeline
> option.
> > >
> > > RC4 looks good to me so far.
> > >
> > >
> > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles
>  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Following up on past discussions and
> > > > https://issues.apache.org/jira/browse/BEAM-1189 I have prepared a
> > > > spreadsheet so we can sign up for validation steps that must be done
> > by a
> > > > human.
> > > >
> > > > The spreadsheet for 2.2.0 is at
> > > > https://s.apache.org/beam-2.2.0-release-validation. Everyone can
> edit,
> > > so
> > > > it is easy to sign up or add new rows.
> > > >
> > > > FYI the template is at https://s.apache.org/beam-release-validation.
> > > After
> > > > this release, I will update the rows in the template to match
> whatever
> > we
> > > > have added to the validation for 2.2.0.
> > > >
> > > > Kenn
> > > >
> > > >
> > > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax
>  > >
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Please review and vote on the release candidate #4 for the version
> > > 2.2.0,
> > > > > as follows:
> > > > >   [ ] +1, Approve the release
> > > > >   [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > >
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > >   * JIRA release notes [1],
> > > > >   * the official Apache source release to be deployed to
> > > dist.apache.org
> > > > > [2],
> > > > > which is signed with the key with fingerprint B98B7708 [3],
> > > > >   * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > >   * source code tag "v2.2.0-RC4" [5],
> > > > >   * website pull request listing the release and publishing the API
> > > > > reference manual [6].
> > > > >   * Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle
> JDK
> > > > > 1.8.0_144.
> > > > >   * Python artifacts are deployed along with the source release to
> > the
> > > > > dist.apache.org [2].
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > Thanks,
> > > > > Reuven
> > > > >
> > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
> > > > > rojectId=12319527=12341044
> > > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
> > > > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > > > [4] https://repository.apache.org/content/repositories/
> > > > orgapachebeam-1025/
> > > > > [5] https://github.com/apache/beam/tree/v2.2.0-RC4
> > > > > 
> > > > > [6] https://github.com/apache/beam-site/pull/337
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release 2.2.0, release candidate #4

[Valentyn Tymofieiev]

I have a process question: is the vote open for committers only or for all
contributors?

On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik 
wrote:

> +1, Approve the release
>
> I have verified the wordcount quickstart on the Apache Beam website using
> Apex, DirectRunner, Flink & Spark on Linux.
>
> The Gearpump runner is yet to have a quickstart listed on our website.
> Adding the quickstart is already represented by this existing issue:
> https://issues.apache.org/jira/browse/BEAM-2692
>
> On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev <
> valen...@google.com.invalid> wrote:
>
> > I have verified: SHA & MD5 signatures of Python artifacts in [2], and
> > checked Python side of the validation checklist on Linux.
> >
> > There is one known issue in UserScore example for Dataflow runner. The
> > issue has been fixed on master branch and does not require a cherry-pick
> at
> > this point. A workaround is to pass --save_main_session pipeline option.
> >
> > RC4 looks good to me so far.
> >
> >
> > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles  >
> > wrote:
> >
> > > Hi all,
> > >
> > > Following up on past discussions and
> > > https://issues.apache.org/jira/browse/BEAM-1189 I have prepared a
> > > spreadsheet so we can sign up for validation steps that must be done
> by a
> > > human.
> > >
> > > The spreadsheet for 2.2.0 is at
> > > https://s.apache.org/beam-2.2.0-release-validation. Everyone can edit,
> > so
> > > it is easy to sign up or add new rows.
> > >
> > > FYI the template is at https://s.apache.org/beam-release-validation.
> > After
> > > this release, I will update the rows in the template to match whatever
> we
> > > have added to the validation for 2.2.0.
> > >
> > > Kenn
> > >
> > >
> > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax  >
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #4 for the version
> > 2.2.0,
> > > > as follows:
> > > >   [ ] +1, Approve the release
> > > >   [ ] -1, Do not approve the release (please provide specific
> comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > >   * JIRA release notes [1],
> > > >   * the official Apache source release to be deployed to
> > dist.apache.org
> > > > [2],
> > > > which is signed with the key with fingerprint B98B7708 [3],
> > > >   * all artifacts to be deployed to the Maven Central Repository [4],
> > > >   * source code tag "v2.2.0-RC4" [5],
> > > >   * website pull request listing the release and publishing the API
> > > > reference manual [6].
> > > >   * Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
> > > > 1.8.0_144.
> > > >   * Python artifacts are deployed along with the source release to
> the
> > > > dist.apache.org [2].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Reuven
> > > >
> > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
> > > > rojectId=12319527=12341044
> > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
> > > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > > [4] https://repository.apache.org/content/repositories/
> > > orgapachebeam-1025/
> > > > [5] https://github.com/apache/beam/tree/v2.2.0-RC4
> > > > 
> > > > [6] https://github.com/apache/beam-site/pull/337
> > > >
> > >
> >
>

Re: [VOTE] Release 2.2.0, release candidate #4

[Lukasz Cwik]

+1, Approve the release

I have verified the wordcount quickstart on the Apache Beam website using
Apex, DirectRunner, Flink & Spark on Linux.

The Gearpump runner is yet to have a quickstart listed on our website.
Adding the quickstart is already represented by this existing issue:
https://issues.apache.org/jira/browse/BEAM-2692

On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev <
valen...@google.com.invalid> wrote:

> I have verified: SHA & MD5 signatures of Python artifacts in [2], and
> checked Python side of the validation checklist on Linux.
>
> There is one known issue in UserScore example for Dataflow runner. The
> issue has been fixed on master branch and does not require a cherry-pick at
> this point. A workaround is to pass --save_main_session pipeline option.
>
> RC4 looks good to me so far.
>
>
> On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles 
> wrote:
>
> > Hi all,
> >
> > Following up on past discussions and
> > https://issues.apache.org/jira/browse/BEAM-1189 I have prepared a
> > spreadsheet so we can sign up for validation steps that must be done by a
> > human.
> >
> > The spreadsheet for 2.2.0 is at
> > https://s.apache.org/beam-2.2.0-release-validation. Everyone can edit,
> so
> > it is easy to sign up or add new rows.
> >
> > FYI the template is at https://s.apache.org/beam-release-validation.
> After
> > this release, I will update the rows in the template to match whatever we
> > have added to the validation for 2.2.0.
> >
> > Kenn
> >
> >
> > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #4 for the version
> 2.2.0,
> > > as follows:
> > >   [ ] +1, Approve the release
> > >   [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > >
> > > The complete staging area is available for your review, which includes:
> > >   * JIRA release notes [1],
> > >   * the official Apache source release to be deployed to
> dist.apache.org
> > > [2],
> > > which is signed with the key with fingerprint B98B7708 [3],
> > >   * all artifacts to be deployed to the Maven Central Repository [4],
> > >   * source code tag "v2.2.0-RC4" [5],
> > >   * website pull request listing the release and publishing the API
> > > reference manual [6].
> > >   * Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
> > > 1.8.0_144.
> > >   * Python artifacts are deployed along with the source release to the
> > > dist.apache.org [2].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Reuven
> > >
> > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
> > > rojectId=12319527=12341044
> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
> > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > [4] https://repository.apache.org/content/repositories/
> > orgapachebeam-1025/
> > > [5] https://github.com/apache/beam/tree/v2.2.0-RC4
> > > 
> > > [6] https://github.com/apache/beam-site/pull/337
> > >
> >
>

Re: [VOTE] Release 2.2.0, release candidate #4

[Valentyn Tymofieiev]

I have verified: SHA & MD5 signatures of Python artifacts in [2], and
checked Python side of the validation checklist on Linux.

There is one known issue in UserScore example for Dataflow runner. The
issue has been fixed on master branch and does not require a cherry-pick at
this point. A workaround is to pass --save_main_session pipeline option.

RC4 looks good to me so far.


On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles 
wrote:

> Hi all,
>
> Following up on past discussions and
> https://issues.apache.org/jira/browse/BEAM-1189 I have prepared a
> spreadsheet so we can sign up for validation steps that must be done by a
> human.
>
> The spreadsheet for 2.2.0 is at
> https://s.apache.org/beam-2.2.0-release-validation. Everyone can edit, so
> it is easy to sign up or add new rows.
>
> FYI the template is at https://s.apache.org/beam-release-validation. After
> this release, I will update the rows in the template to match whatever we
> have added to the validation for 2.2.0.
>
> Kenn
>
>
> On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax 
> wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #4 for the version 2.2.0,
> > as follows:
> >   [ ] +1, Approve the release
> >   [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> >   * JIRA release notes [1],
> >   * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > which is signed with the key with fingerprint B98B7708 [3],
> >   * all artifacts to be deployed to the Maven Central Repository [4],
> >   * source code tag "v2.2.0-RC4" [5],
> >   * website pull request listing the release and publishing the API
> > reference manual [6].
> >   * Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
> > 1.8.0_144.
> >   * Python artifacts are deployed along with the source release to the
> > dist.apache.org [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Reuven
> >
> > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
> > rojectId=12319527=12341044
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4] https://repository.apache.org/content/repositories/
> orgapachebeam-1025/
> > [5] https://github.com/apache/beam/tree/v2.2.0-RC4
> > 
> > [6] https://github.com/apache/beam-site/pull/337
> >
>

Re: [VOTE] Release 2.2.0, release candidate #4

[Kenneth Knowles]

Hi all,

Following up on past discussions and
https://issues.apache.org/jira/browse/BEAM-1189 I have prepared a
spreadsheet so we can sign up for validation steps that must be done by a
human.

The spreadsheet for 2.2.0 is at
https://s.apache.org/beam-2.2.0-release-validation. Everyone can edit, so
it is easy to sign up or add new rows.

FYI the template is at https://s.apache.org/beam-release-validation. After
this release, I will update the rows in the template to match whatever we
have added to the validation for 2.2.0.

Kenn


On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax 
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #4 for the version 2.2.0,
> as follows:
>   [ ] +1, Approve the release
>   [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
>   * JIRA release notes [1],
>   * the official Apache source release to be deployed to dist.apache.org
> [2],
> which is signed with the key with fingerprint B98B7708 [3],
>   * all artifacts to be deployed to the Maven Central Repository [4],
>   * source code tag "v2.2.0-RC4" [5],
>   * website pull request listing the release and publishing the API
> reference manual [6].
>   * Java artifacts were built with Maven 3.5.0 and OpenJDK/Oracle JDK
> 1.8.0_144.
>   * Python artifacts are deployed along with the source release to the
> dist.apache.org [2].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Reuven
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p
> rojectId=12319527=12341044
> [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1025/
> [5] https://github.com/apache/beam/tree/v2.2.0-RC4
> 
> [6] https://github.com/apache/beam-site/pull/337
>