subject:"Beam 2.2.0 release"

Re: Beam 2.2.0 release

2017-09-11 Thread Jean-Baptiste Onofré


Hi,

Any PMC member has the permissions (updating Jira, access to 
reporter.apache.org, etc).


Regards
JB

On 09/11/2017 06:46 PM, Reuven Lax wrote:

Are these permissions that only you have, or does anyone on the PMC have
these permissions? I'm asking so that in the future if you are unavailable,
we know who has these permissions. We should also make sure this is all
documented on the Beam release guide.

Reuven

On Wed, Sep 6, 2017 at 9:25 PM, Jean-Baptiste Onofré 
wrote:


It sounds good to me.

By the way, you will need my help to complete the release process (as you
need some permissions that you don't have).

Regards
JB


On 09/07/2017 01:00 AM, Reuven Lax wrote:


It sounds like SQL is still not in, and there are a couple of other PRs
that people have requested in 2.2.0. I am mostly out next week, so let's
set September 18 as a target date for cutting the first RC. That should
hopefully give plenty of time to get SQL and the remaining PRs merged into
master.

Reuven

On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu  wrote:

Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to

merge DSL_SQL. There may be something wrong in the back-end(maybe
RunnerApi) to handle parametered CustomCoder in TestPipeline.

On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré 
wrote:

Fair enough.


That's fine for me.

Regards
JB

On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz 
wrote:


I'll chime in as a user who would love to see 2.2.0 sooner than later,
specifically for the file IO Eugene mentioned.  We're using the AvroIO
enhancements extensively, but I am hesitant to run from HEAD in master
in
production.

On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

There are a lot of users including very large production customers



who have


been asking specifically for the features that are in 2.2.0 (most of


them


accumulated while 2.1.0 was being iterated on) - mostly I'm referring


to


the vastly improved file IO - and they have been hesitant to use Beam


at


HEAD in production. I think the slight unusualness of having a


release


published soon after the previous release is a small price to pay for
helping those users :)

On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré 
wrote:

As we released 2.1.0 couple of weeks ago, it could sound weird to



the



users to

do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1


If



it's


new
features, why not having a release pace in October (2.2.0) ?

Thoughts ?

Regards
JB

On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:


I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0


for



October. I don't see a reason to delay 2.2.0 until October:



there's a



huge



amount of features worth releasing between when 2.1.0 was cut and


the



current HEAD.


On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré


 wrote:

Any chance to get the RedisIO in this release?

[BEAM-1017] Add RedisIO #1687

Its not my PR but ll be happy to assist if there is anything I


can



do


to


help.


On 30 Aug 2017 22:46, "Daniel Ribeiro"





dependency.



It



is

currently very outdated (v1-rev10-1.22.0, which was released


over a



year



ago


).


On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

Thanks Ismael. I've marked these two issues for fix in



2.2.0.



Definitely



agree that at least the first one must be fixed.


Here's the current burndown list

https://issues.apache.org/jira/projects/BEAM/versions/12341044

-


we


should



clean it up.

On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía






wrote:





The current master has accumulated a good amount of nice



features



since 2.1.0 so a new release is welcomed. I have two



JIRAs/PR



that


I


think are important to check/solve before the cut:


BEAM-2516 (this is a regression on the performance of


Direct



runner


on



Java). We had never really defined if a performance



regression is



critical to be a blocker. I executed WordCount with the



kinglear.txt



(170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT



and I



found that the execution time passed from 5s to

Re: Beam 2.2.0 release

2017-09-11 Thread Mingmin Xu

I don't think there're some particular permissions, #3782 is merged to
master. So from SQL perspective I'm good to the 2.2.0 release, will do a
quick POC job for verification purpose.

Mingmin

On Mon, Sep 11, 2017 at 9:46 AM, Reuven Lax 
wrote:

> Are these permissions that only you have, or does anyone on the PMC have
> these permissions? I'm asking so that in the future if you are unavailable,
> we know who has these permissions. We should also make sure this is all
> documented on the Beam release guide.
>
> Reuven
>
> On Wed, Sep 6, 2017 at 9:25 PM, Jean-Baptiste Onofré 
> wrote:
>
> > It sounds good to me.
> >
> > By the way, you will need my help to complete the release process (as you
> > need some permissions that you don't have).
> >
> > Regards
> > JB
> >
> >
> > On 09/07/2017 01:00 AM, Reuven Lax wrote:
> >
> >> It sounds like SQL is still not in, and there are a couple of other PRs
> >> that people have requested in 2.2.0. I am mostly out next week, so let's
> >> set September 18 as a target date for cutting the first RC. That should
> >> hopefully give plenty of time to get SQL and the remaining PRs merged
> into
> >> master.
> >>
> >> Reuven
> >>
> >> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu  wrote:
> >>
> >> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker
> to
> >>> merge DSL_SQL. There may be something wrong in the back-end(maybe
> >>> RunnerApi) to handle parametered CustomCoder in TestPipeline.
> >>>
> >>> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <
> j...@nanthrax.net>
> >>> wrote:
> >>>
> >>> Fair enough.
> 
>  That's fine for me.
> 
>  Regards
>  JB
> 
>  On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz 
>  wrote:
> 
> > I'll chime in as a user who would love to see 2.2.0 sooner than
> later,
> > specifically for the file IO Eugene mentioned.  We're using the
> AvroIO
> > enhancements extensively, but I am hesitant to run from HEAD in
> master
> > in
> > production.
> >
> > On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> > kirpic...@google.com.invalid> wrote:
> >
> > There are a lot of users including very large production customers
> >>
> > who have
> >
> >> been asking specifically for the features that are in 2.2.0 (most of
> >>
> > them
> >
> >> accumulated while 2.1.0 was being iterated on) - mostly I'm
> referring
> >>
> > to
> >
> >> the vastly improved file IO - and they have been hesitant to use
> Beam
> >>
> > at
> >
> >> HEAD in production. I think the slight unusualness of having a
> >>
> > release
> >
> >> published soon after the previous release is a small price to pay
> for
> >> helping those users :)
> >>
> >> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <
> j...@nanthrax.net>
> >> wrote:
> >>
> >> As we released 2.1.0 couple of weeks ago, it could sound weird to
> >>>
> >> the
> >
> >> users to
> >>> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
> >>>
> >> If
> >
> >> it's
> >>
> >>> new
> >>> features, why not having a release pace in October (2.2.0) ?
> >>>
> >>> Thoughts ?
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> >>>
>  I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
> 
> >>> for
> >
> >> October. I don't see a reason to delay 2.2.0 until October:
> 
> >>> there's a
> >
> >> huge
> >>>
>  amount of features worth releasing between when 2.1.0 was cut and
> 
> >>> the
> >
> >> current HEAD.
> 
>  On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
> 
> >>>  >
> >>
> >>> wrote:
> 
>  With a 2.2.0 in October, I think we can try to move forward on
> >
>  RedisIO.
> >>
> >>>
> > I'm now back from vacation and I will resume the work on this
> >
>  IO.
> >
> >>
> > Regards
> > JB
> >
> > On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> >
> >> RedisIO in 2.2.0 is very unlikely. There's still a lot of
> >>
> > review
> >
> >> remaining
> >
> >> last time I checked on the PR.
> >>
> >> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> >> vonehrenh...@gmail.com> wrote:
> >>
> >> Any chance to get the RedisIO in this release?
> >>> [BEAM-1017] Add RedisIO #1687
> >>>
> >>> Its not my PR but ll be happy to assist if there is anything I
> >>>
> >> can
> >
> >> do
> >>
> >>> to
> >>>
>

Re: Beam 2.2.0 release

2017-09-11 Thread Reuven Lax

Are these permissions that only you have, or does anyone on the PMC have
these permissions? I'm asking so that in the future if you are unavailable,
we know who has these permissions. We should also make sure this is all
documented on the Beam release guide.

Reuven

On Wed, Sep 6, 2017 at 9:25 PM, Jean-Baptiste Onofré 
wrote:

> It sounds good to me.
>
> By the way, you will need my help to complete the release process (as you
> need some permissions that you don't have).
>
> Regards
> JB
>
>
> On 09/07/2017 01:00 AM, Reuven Lax wrote:
>
>> It sounds like SQL is still not in, and there are a couple of other PRs
>> that people have requested in 2.2.0. I am mostly out next week, so let's
>> set September 18 as a target date for cutting the first RC. That should
>> hopefully give plenty of time to get SQL and the remaining PRs merged into
>> master.
>>
>> Reuven
>>
>> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu  wrote:
>>
>> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
>>> merge DSL_SQL. There may be something wrong in the back-end(maybe
>>> RunnerApi) to handle parametered CustomCoder in TestPipeline.
>>>
>>> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré 
>>> wrote:
>>>
>>> Fair enough.

 That's fine for me.

 Regards
 JB

 On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz 
 wrote:

> I'll chime in as a user who would love to see 2.2.0 sooner than later,
> specifically for the file IO Eugene mentioned.  We're using the AvroIO
> enhancements extensively, but I am hesitant to run from HEAD in master
> in
> production.
>
> On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
> There are a lot of users including very large production customers
>>
> who have
>
>> been asking specifically for the features that are in 2.2.0 (most of
>>
> them
>
>> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
>>
> to
>
>> the vastly improved file IO - and they have been hesitant to use Beam
>>
> at
>
>> HEAD in production. I think the slight unusualness of having a
>>
> release
>
>> published soon after the previous release is a small price to pay for
>> helping those users :)
>>
>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré 
>> wrote:
>>
>> As we released 2.1.0 couple of weeks ago, it could sound weird to
>>>
>> the
>
>> users to
>>> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
>>>
>> If
>
>> it's
>>
>>> new
>>> features, why not having a release pace in October (2.2.0) ?
>>>
>>> Thoughts ?
>>>
>>> Regards
>>> JB
>>>
>>> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
>>>
 I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0

>>> for
>
>> October. I don't see a reason to delay 2.2.0 until October:

>>> there's a
>
>> huge
>>>
 amount of features worth releasing between when 2.1.0 was cut and

>>> the
>
>> current HEAD.

 On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré

>>> 
>>
>>> wrote:

 With a 2.2.0 in October, I think we can try to move forward on
>
 RedisIO.
>>
>>>
> I'm now back from vacation and I will resume the work on this
>
 IO.
>
>>
> Regards
> JB
>
> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>
>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
>>
> review
>
>> remaining
>
>> last time I checked on the PR.
>>
>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>> vonehrenh...@gmail.com> wrote:
>>
>> Any chance to get the RedisIO in this release?
>>> [BEAM-1017] Add RedisIO #1687
>>>
>>> Its not my PR but ll be happy to assist if there is anything I
>>>
>> can
>
>> do
>>
>>> to
>>>
 help.
>>>
>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
>>>
>> >
>>>
 wrote:
>>>
>>> It would be great to get a bump on pubsub
 

>>> dependency.
>>>
 It
>
>> is
 currently very outdated (v1-rev10-1.22.0, which was released

>>> over a
>
>> year
>
>> ago

Re: Beam 2.2.0 release

2017-09-07 Thread Mingmin Xu

Thanks @JB, add you and Tyler as reviewer and is waiting for jenkins job.

On Thu, Sep 7, 2017 at 12:25 PM, Jean-Baptiste Onofré 
wrote:

> It sounds good to me.
>
> By the way, you will need my help to complete the release process (as you
> need some permissions that you don't have).
>
> Regards
> JB
>
>
> On 09/07/2017 01:00 AM, Reuven Lax wrote:
>
>> It sounds like SQL is still not in, and there are a couple of other PRs
>> that people have requested in 2.2.0. I am mostly out next week, so let's
>> set September 18 as a target date for cutting the first RC. That should
>> hopefully give plenty of time to get SQL and the remaining PRs merged into
>> master.
>>
>> Reuven
>>
>> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu  wrote:
>>
>> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
>>> merge DSL_SQL. There may be something wrong in the back-end(maybe
>>> RunnerApi) to handle parametered CustomCoder in TestPipeline.
>>>
>>> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré 
>>> wrote:
>>>
>>> Fair enough.

 That's fine for me.

 Regards
 JB

 On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz 
 wrote:

> I'll chime in as a user who would love to see 2.2.0 sooner than later,
> specifically for the file IO Eugene mentioned.  We're using the AvroIO
> enhancements extensively, but I am hesitant to run from HEAD in master
> in
> production.
>
> On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
> There are a lot of users including very large production customers
>>
> who have
>
>> been asking specifically for the features that are in 2.2.0 (most of
>>
> them
>
>> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
>>
> to
>
>> the vastly improved file IO - and they have been hesitant to use Beam
>>
> at
>
>> HEAD in production. I think the slight unusualness of having a
>>
> release
>
>> published soon after the previous release is a small price to pay for
>> helping those users :)
>>
>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré 
>> wrote:
>>
>> As we released 2.1.0 couple of weeks ago, it could sound weird to
>>>
>> the
>
>> users to
>>> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
>>>
>> If
>
>> it's
>>
>>> new
>>> features, why not having a release pace in October (2.2.0) ?
>>>
>>> Thoughts ?
>>>
>>> Regards
>>> JB
>>>
>>> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
>>>
 I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0

>>> for
>
>> October. I don't see a reason to delay 2.2.0 until October:

>>> there's a
>
>> huge
>>>
 amount of features worth releasing between when 2.1.0 was cut and

>>> the
>
>> current HEAD.

 On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré

>>> 
>>
>>> wrote:

 With a 2.2.0 in October, I think we can try to move forward on
>
 RedisIO.
>>
>>>
> I'm now back from vacation and I will resume the work on this
>
 IO.
>
>>
> Regards
> JB
>
> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>
>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
>>
> review
>
>> remaining
>
>> last time I checked on the PR.
>>
>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>> vonehrenh...@gmail.com> wrote:
>>
>> Any chance to get the RedisIO in this release?
>>> [BEAM-1017] Add RedisIO #1687
>>>
>>> Its not my PR but ll be happy to assist if there is anything I
>>>
>> can
>
>> do
>>
>>> to
>>>
 help.
>>>
>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
>>>
>> >
>>>
 wrote:
>>>
>>> It would be great to get a bump on pubsub
 

>>> dependency.
>>>
 It
>
>> is
 currently very outdated (v1-rev10-1.22.0, which was released

>>> over a
>
>> year
>
>> ago
 
 ).


 On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
 kirpic...@google.com.invalid> wrote:

Re: Beam 2.2.0 release

2017-09-06 Thread Jean-Baptiste Onofré


It sounds good to me.

By the way, you will need my help to complete the release process (as you need 
some permissions that you don't have).


Regards
JB

On 09/07/2017 01:00 AM, Reuven Lax wrote:

It sounds like SQL is still not in, and there are a couple of other PRs
that people have requested in 2.2.0. I am mostly out next week, so let's
set September 18 as a target date for cutting the first RC. That should
hopefully give plenty of time to get SQL and the remaining PRs merged into
master.

Reuven

On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu  wrote:


Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
merge DSL_SQL. There may be something wrong in the back-end(maybe
RunnerApi) to handle parametered CustomCoder in TestPipeline.

On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré 
wrote:


Fair enough.

That's fine for me.

Regards
JB

On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz 
wrote:

I'll chime in as a user who would love to see 2.2.0 sooner than later,
specifically for the file IO Eugene mentioned.  We're using the AvroIO
enhancements extensively, but I am hesitant to run from HEAD in master
in
production.

On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:


There are a lot of users including very large production customers

who have

been asking specifically for the features that are in 2.2.0 (most of

them

accumulated while 2.1.0 was being iterated on) - mostly I'm referring

to

the vastly improved file IO - and they have been hesitant to use Beam

at

HEAD in production. I think the slight unusualness of having a

release

published soon after the previous release is a small price to pay for
helping those users :)

On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré 
wrote:


As we released 2.1.0 couple of weeks ago, it could sound weird to

the

users to
do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1

If

it's

new
features, why not having a release pace in October (2.2.0) ?

Thoughts ?

Regards
JB

On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:

I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0

for

October. I don't see a reason to delay 2.2.0 until October:

there's a

huge

amount of features worth releasing between when 2.1.0 was cut and

the

current HEAD.

On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré

 wrote:


Any chance to get the RedisIO in this release?
[BEAM-1017] Add RedisIO #1687

Its not my PR but ll be happy to assist if there is anything I

can

do

to

help.

On 30 Aug 2017 22:46, "Daniel Ribeiro"



dependency.

It

is
currently very outdated (v1-rev10-1.22.0, which was released

over a

year

ago

).


On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:


Thanks Ismael. I've marked these two issues for fix in

2.2.0.

Definitely

agree that at least the first one must be fixed.

Here's the current burndown list


https://issues.apache.org/jira/projects/BEAM/versions/12341044 -

we

should

clean it up.

On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía



wrote:



The current master has accumulated a good amount of nice

features

since 2.1.0 so a new release is welcomed. I have two

JIRAs/PR

that

I

think are important to check/solve before the cut:

BEAM-2516 (this is a regression on the performance of

Direct

runner

on

Java). We had never really defined if a performance

regression is

critical to be a blocker. I executed WordCount with the

kinglear.txt

(170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT

and I

found that the execution time passed from 5s to 126s. So

maybe we

need

to review this one before the release. I can understand if

others

consider this a minor issue because the Direct runner is

not

supposed

to be used for production, but this performance regression

can

cause

a

bad impression for a casual user starting with Beam.

BEAM-2790 (fix reading from Amazon S3 via

HadoopFileSystem). I

think

this one is a nice to have. I am not sure that I can tackle

it

for

the

wednesday cut. I’m OOO until the beginning of next week,

but

maybe

someone else can take a look. In the worst case this is not

a

release

blocker but definitely a really

Re: Beam 2.2.0 release

2017-09-06 Thread Mingmin Xu

Regarding to BEAM-2833, a PR is there for review(
https://github.com/apache/beam/pull/3803). Once it's completed, I'll move
forward for SQL merge.

Mingmin

On Wed, Sep 6, 2017 at 4:00 PM, Reuven Lax  wrote:

> It sounds like SQL is still not in, and there are a couple of other PRs
> that people have requested in 2.2.0. I am mostly out next week, so let's
> set September 18 as a target date for cutting the first RC. That should
> hopefully give plenty of time to get SQL and the remaining PRs merged into
> master.
>
> Reuven
>
> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu  wrote:
>
> > Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker
> to
> > merge DSL_SQL. There may be something wrong in the back-end(maybe
> > RunnerApi) to handle parametered CustomCoder in TestPipeline.
> >
> > On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré 
> > wrote:
> >
> > > Fair enough.
> > >
> > > That's fine for me.
> > >
> > > Regards
> > > JB
> > >
> > > On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz 
> > > wrote:
> > > >I'll chime in as a user who would love to see 2.2.0 sooner than later,
> > > >specifically for the file IO Eugene mentioned.  We're using the AvroIO
> > > >enhancements extensively, but I am hesitant to run from HEAD in master
> > > >in
> > > >production.
> > > >
> > > >On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> > > >kirpic...@google.com.invalid> wrote:
> > > >
> > > >> There are a lot of users including very large production customers
> > > >who have
> > > >> been asking specifically for the features that are in 2.2.0 (most of
> > > >them
> > > >> accumulated while 2.1.0 was being iterated on) - mostly I'm
> referring
> > > >to
> > > >> the vastly improved file IO - and they have been hesitant to use
> Beam
> > > >at
> > > >> HEAD in production. I think the slight unusualness of having a
> > > >release
> > > >> published soon after the previous release is a small price to pay
> for
> > > >> helping those users :)
> > > >>
> > > >> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > > >> wrote:
> > > >>
> > > >> > As we released 2.1.0 couple of weeks ago, it could sound weird to
> > > >the
> > > >> > users to
> > > >> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
> > > >If
> > > >> it's
> > > >> > new
> > > >> > features, why not having a release pace in October (2.2.0) ?
> > > >> >
> > > >> > Thoughts ?
> > > >> >
> > > >> > Regards
> > > >> > JB
> > > >> >
> > > >> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> > > >> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
> > > >for
> > > >> > > October. I don't see a reason to delay 2.2.0 until October:
> > > >there's a
> > > >> > huge
> > > >> > > amount of features worth releasing between when 2.1.0 was cut
> and
> > > >the
> > > >> > > current HEAD.
> > > >> > >
> > > >> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
> > > > > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > >> With a 2.2.0 in October, I think we can try to move forward on
> > > >> RedisIO.
> > > >> > >>
> > > >> > >> I'm now back from vacation and I will resume the work on this
> > > >IO.
> > > >> > >>
> > > >> > >> Regards
> > > >> > >> JB
> > > >> > >>
> > > >> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> > > >> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
> > > >review
> > > >> > >> remaining
> > > >> > >>> last time I checked on the PR.
> > > >> > >>>
> > > >> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> > > >> > >>> vonehrenh...@gmail.com> wrote:
> > > >> > >>>
> > > >> >  Any chance to get the RedisIO in this release?
> > > >> >  [BEAM-1017] Add RedisIO #1687
> > > >> > 
> > > >> >  Its not my PR but ll be happy to assist if there is anything
> I
> > > >can
> > > >> do
> > > >> > to
> > > >> >  help.
> > > >> > 
> > > >> >  On 30 Aug 2017 22:46, "Daniel Ribeiro"
> > > >>  > > >> > >
> > > >> >  wrote:
> > > >> > 
> > > >> > > It would be great to get a bump on pubsub
> > > >> > > 
> > > >> > dependency.
> > > >> > >> It
> > > >> > > is
> > > >> > > currently very outdated (v1-rev10-1.22.0, which was released
> > > >over a
> > > >> > >> year
> > > >> > > ago
> > > >> > >  > > >> > > api-services-pubsub/v1-rev10-1.22.0/>
> > > >> > > ).
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > > >> > > kirpic...@google.com.invalid> wrote:
> > > >> > >
> > > >> > >> Thanks Ismael. I've marked these two issues for fix in
> > > >2.2.0.
> > > >> >  Definitely
> > > >> > >> agree that at least the first one must be fixed.
> > > >> > >>

Re: Beam 2.2.0 release

2017-09-06 Thread Reuven Lax

It sounds like SQL is still not in, and there are a couple of other PRs
that people have requested in 2.2.0. I am mostly out next week, so let's
set September 18 as a target date for cutting the first RC. That should
hopefully give plenty of time to get SQL and the remaining PRs merged into
master.

Reuven

On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu  wrote:

> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
> merge DSL_SQL. There may be something wrong in the back-end(maybe
> RunnerApi) to handle parametered CustomCoder in TestPipeline.
>
> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré 
> wrote:
>
> > Fair enough.
> >
> > That's fine for me.
> >
> > Regards
> > JB
> >
> > On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz 
> > wrote:
> > >I'll chime in as a user who would love to see 2.2.0 sooner than later,
> > >specifically for the file IO Eugene mentioned.  We're using the AvroIO
> > >enhancements extensively, but I am hesitant to run from HEAD in master
> > >in
> > >production.
> > >
> > >On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> > >kirpic...@google.com.invalid> wrote:
> > >
> > >> There are a lot of users including very large production customers
> > >who have
> > >> been asking specifically for the features that are in 2.2.0 (most of
> > >them
> > >> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
> > >to
> > >> the vastly improved file IO - and they have been hesitant to use Beam
> > >at
> > >> HEAD in production. I think the slight unusualness of having a
> > >release
> > >> published soon after the previous release is a small price to pay for
> > >> helping those users :)
> > >>
> > >> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré 
> > >> wrote:
> > >>
> > >> > As we released 2.1.0 couple of weeks ago, it could sound weird to
> > >the
> > >> > users to
> > >> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
> > >If
> > >> it's
> > >> > new
> > >> > features, why not having a release pace in October (2.2.0) ?
> > >> >
> > >> > Thoughts ?
> > >> >
> > >> > Regards
> > >> > JB
> > >> >
> > >> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> > >> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
> > >for
> > >> > > October. I don't see a reason to delay 2.2.0 until October:
> > >there's a
> > >> > huge
> > >> > > amount of features worth releasing between when 2.1.0 was cut and
> > >the
> > >> > > current HEAD.
> > >> > >
> > >> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
> > > > >> >
> > >> > > wrote:
> > >> > >
> > >> > >> With a 2.2.0 in October, I think we can try to move forward on
> > >> RedisIO.
> > >> > >>
> > >> > >> I'm now back from vacation and I will resume the work on this
> > >IO.
> > >> > >>
> > >> > >> Regards
> > >> > >> JB
> > >> > >>
> > >> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> > >> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
> > >review
> > >> > >> remaining
> > >> > >>> last time I checked on the PR.
> > >> > >>>
> > >> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> > >> > >>> vonehrenh...@gmail.com> wrote:
> > >> > >>>
> > >> >  Any chance to get the RedisIO in this release?
> > >> >  [BEAM-1017] Add RedisIO #1687
> > >> > 
> > >> >  Its not my PR but ll be happy to assist if there is anything I
> > >can
> > >> do
> > >> > to
> > >> >  help.
> > >> > 
> > >> >  On 30 Aug 2017 22:46, "Daniel Ribeiro"
> > >>  > >> > >
> > >> >  wrote:
> > >> > 
> > >> > > It would be great to get a bump on pubsub
> > >> > > 
> > >> > dependency.
> > >> > >> It
> > >> > > is
> > >> > > currently very outdated (v1-rev10-1.22.0, which was released
> > >over a
> > >> > >> year
> > >> > > ago
> > >> > >  > >> > > api-services-pubsub/v1-rev10-1.22.0/>
> > >> > > ).
> > >> > >
> > >> > >
> > >> > > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > >> > > kirpic...@google.com.invalid> wrote:
> > >> > >
> > >> > >> Thanks Ismael. I've marked these two issues for fix in
> > >2.2.0.
> > >> >  Definitely
> > >> > >> agree that at least the first one must be fixed.
> > >> > >>
> > >> > >> Here's the current burndown list
> > >> > >>
> > >https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
> > >> we
> > >> > > should
> > >> > >> clean it up.
> > >> > >>
> > >> > >> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
> > >
> > >> >  wrote:
> > >> > >>
> > >> > >>> The current master has accumulated a good amount of nice
> > >features
> > >> > >>> since 2.1.0 so a new release is welcomed. I have two
> > >JIRAs/PR
> > >>

Re: Beam 2.2.0 release

2017-08-31 Thread Mingmin Xu

Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
merge DSL_SQL. There may be something wrong in the back-end(maybe
RunnerApi) to handle parametered CustomCoder in TestPipeline.

On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré 
wrote:

> Fair enough.
>
> That's fine for me.
>
> Regards
> JB
>
> On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz 
> wrote:
> >I'll chime in as a user who would love to see 2.2.0 sooner than later,
> >specifically for the file IO Eugene mentioned.  We're using the AvroIO
> >enhancements extensively, but I am hesitant to run from HEAD in master
> >in
> >production.
> >
> >On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> >kirpic...@google.com.invalid> wrote:
> >
> >> There are a lot of users including very large production customers
> >who have
> >> been asking specifically for the features that are in 2.2.0 (most of
> >them
> >> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
> >to
> >> the vastly improved file IO - and they have been hesitant to use Beam
> >at
> >> HEAD in production. I think the slight unusualness of having a
> >release
> >> published soon after the previous release is a small price to pay for
> >> helping those users :)
> >>
> >> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré 
> >> wrote:
> >>
> >> > As we released 2.1.0 couple of weeks ago, it could sound weird to
> >the
> >> > users to
> >> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
> >If
> >> it's
> >> > new
> >> > features, why not having a release pace in October (2.2.0) ?
> >> >
> >> > Thoughts ?
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> >> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
> >for
> >> > > October. I don't see a reason to delay 2.2.0 until October:
> >there's a
> >> > huge
> >> > > amount of features worth releasing between when 2.1.0 was cut and
> >the
> >> > > current HEAD.
> >> > >
> >> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
> > >> >
> >> > > wrote:
> >> > >
> >> > >> With a 2.2.0 in October, I think we can try to move forward on
> >> RedisIO.
> >> > >>
> >> > >> I'm now back from vacation and I will resume the work on this
> >IO.
> >> > >>
> >> > >> Regards
> >> > >> JB
> >> > >>
> >> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> >> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
> >review
> >> > >> remaining
> >> > >>> last time I checked on the PR.
> >> > >>>
> >> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> >> > >>> vonehrenh...@gmail.com> wrote:
> >> > >>>
> >> >  Any chance to get the RedisIO in this release?
> >> >  [BEAM-1017] Add RedisIO #1687
> >> > 
> >> >  Its not my PR but ll be happy to assist if there is anything I
> >can
> >> do
> >> > to
> >> >  help.
> >> > 
> >> >  On 30 Aug 2017 22:46, "Daniel Ribeiro"
> >>  >> > >
> >> >  wrote:
> >> > 
> >> > > It would be great to get a bump on pubsub
> >> > > 
> >> > dependency.
> >> > >> It
> >> > > is
> >> > > currently very outdated (v1-rev10-1.22.0, which was released
> >over a
> >> > >> year
> >> > > ago
> >> > >  >> > > api-services-pubsub/v1-rev10-1.22.0/>
> >> > > ).
> >> > >
> >> > >
> >> > > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> >> > > kirpic...@google.com.invalid> wrote:
> >> > >
> >> > >> Thanks Ismael. I've marked these two issues for fix in
> >2.2.0.
> >> >  Definitely
> >> > >> agree that at least the first one must be fixed.
> >> > >>
> >> > >> Here's the current burndown list
> >> > >>
> >https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
> >> we
> >> > > should
> >> > >> clean it up.
> >> > >>
> >> > >> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
> >
> >> >  wrote:
> >> > >>
> >> > >>> The current master has accumulated a good amount of nice
> >features
> >> > >>> since 2.1.0 so a new release is welcomed. I have two
> >JIRAs/PR
> >> that
> >> > I
> >> > >>> think are important to check/solve before the cut:
> >> > >>>
> >> > >>> BEAM-2516 (this is a regression on the performance of
> >Direct
> >> runner
> >> >  on
> >> > >>> Java). We had never really defined if a performance
> >regression is
> >> > >>> critical to be a blocker. I executed WordCount with the
> >> > kinglear.txt
> >> > >>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
> >and I
> >> > >>> found that the execution time passed from 5s to 126s. So
> >maybe we
> >> >  need
> >> > >>> to review this one before the release. I can understand if
> >others
> >> > >>>

Re: Beam 2.2.0 release

2017-08-31 Thread Jean-Baptiste Onofré

Fair enough.

That's fine for me.

Regards
JB

On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz  wrote:
>I'll chime in as a user who would love to see 2.2.0 sooner than later,
>specifically for the file IO Eugene mentioned.  We're using the AvroIO
>enhancements extensively, but I am hesitant to run from HEAD in master
>in
>production.
>
>On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
>kirpic...@google.com.invalid> wrote:
>
>> There are a lot of users including very large production customers
>who have
>> been asking specifically for the features that are in 2.2.0 (most of
>them
>> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
>to
>> the vastly improved file IO - and they have been hesitant to use Beam
>at
>> HEAD in production. I think the slight unusualness of having a
>release
>> published soon after the previous release is a small price to pay for
>> helping those users :)
>>
>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré 
>> wrote:
>>
>> > As we released 2.1.0 couple of weeks ago, it could sound weird to
>the
>> > users to
>> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
>If
>> it's
>> > new
>> > features, why not having a release pace in October (2.2.0) ?
>> >
>> > Thoughts ?
>> >
>> > Regards
>> > JB
>> >
>> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
>> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
>for
>> > > October. I don't see a reason to delay 2.2.0 until October:
>there's a
>> > huge
>> > > amount of features worth releasing between when 2.1.0 was cut and
>the
>> > > current HEAD.
>> > >
>> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
>> >
>> > > wrote:
>> > >
>> > >> With a 2.2.0 in October, I think we can try to move forward on
>> RedisIO.
>> > >>
>> > >> I'm now back from vacation and I will resume the work on this
>IO.
>> > >>
>> > >> Regards
>> > >> JB
>> > >>
>> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
>review
>> > >> remaining
>> > >>> last time I checked on the PR.
>> > >>>
>> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>> > >>> vonehrenh...@gmail.com> wrote:
>> > >>>
>> >  Any chance to get the RedisIO in this release?
>> >  [BEAM-1017] Add RedisIO #1687
>> > 
>> >  Its not my PR but ll be happy to assist if there is anything I
>can
>> do
>> > to
>> >  help.
>> > 
>> >  On 30 Aug 2017 22:46, "Daniel Ribeiro"
>> > > >
>> >  wrote:
>> > 
>> > > It would be great to get a bump on pubsub
>> > > 
>> > dependency.
>> > >> It
>> > > is
>> > > currently very outdated (v1-rev10-1.22.0, which was released
>over a
>> > >> year
>> > > ago
>> > > > > > api-services-pubsub/v1-rev10-1.22.0/>
>> > > ).
>> > >
>> > >
>> > > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>> > > kirpic...@google.com.invalid> wrote:
>> > >
>> > >> Thanks Ismael. I've marked these two issues for fix in
>2.2.0.
>> >  Definitely
>> > >> agree that at least the first one must be fixed.
>> > >>
>> > >> Here's the current burndown list
>> > >>
>https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
>> we
>> > > should
>> > >> clean it up.
>> > >>
>> > >> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
>
>> >  wrote:
>> > >>
>> > >>> The current master has accumulated a good amount of nice
>features
>> > >>> since 2.1.0 so a new release is welcomed. I have two
>JIRAs/PR
>> that
>> > I
>> > >>> think are important to check/solve before the cut:
>> > >>>
>> > >>> BEAM-2516 (this is a regression on the performance of
>Direct
>> runner
>> >  on
>> > >>> Java). We had never really defined if a performance
>regression is
>> > >>> critical to be a blocker. I executed WordCount with the
>> > kinglear.txt
>> > >>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
>and I
>> > >>> found that the execution time passed from 5s to 126s. So
>maybe we
>> >  need
>> > >>> to review this one before the release. I can understand if
>others
>> > >>> consider this a minor issue because the Direct runner is
>not
>> > supposed
>> > >>> to be used for production, but this performance regression
>can
>> > cause
>> >  a
>> > >>> bad impression for a casual user starting with Beam.
>> > >>>
>> > >>> BEAM-2790 (fix reading from Amazon S3 via
>HadoopFileSystem). I
>> > think
>> > >>> this one is a nice to have. I am not sure that I can tackle
>it
>> for
>> >  the
>> > >>> wednesday cut. I’m OOO until the beginning of next week,
>but
>> maybe
>> > >>> someone else can take a look. In the worst case

Re: Beam 2.2.0 release

2017-08-31 Thread Steve Niemitz

I'll chime in as a user who would love to see 2.2.0 sooner than later,
specifically for the file IO Eugene mentioned.  We're using the AvroIO
enhancements extensively, but I am hesitant to run from HEAD in master in
production.

On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> There are a lot of users including very large production customers who have
> been asking specifically for the features that are in 2.2.0 (most of them
> accumulated while 2.1.0 was being iterated on) - mostly I'm referring to
> the vastly improved file IO - and they have been hesitant to use Beam at
> HEAD in production. I think the slight unusualness of having a release
> published soon after the previous release is a small price to pay for
> helping those users :)
>
> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré 
> wrote:
>
> > As we released 2.1.0 couple of weeks ago, it could sound weird to the
> > users to
> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1 If
> it's
> > new
> > features, why not having a release pace in October (2.2.0) ?
> >
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0 for
> > > October. I don't see a reason to delay 2.2.0 until October: there's a
> > huge
> > > amount of features worth releasing between when 2.1.0 was cut and the
> > > current HEAD.
> > >
> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré  >
> > > wrote:
> > >
> > >> With a 2.2.0 in October, I think we can try to move forward on
> RedisIO.
> > >>
> > >> I'm now back from vacation and I will resume the work on this IO.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of review
> > >> remaining
> > >>> last time I checked on the PR.
> > >>>
> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> > >>> vonehrenh...@gmail.com> wrote:
> > >>>
> >  Any chance to get the RedisIO in this release?
> >  [BEAM-1017] Add RedisIO #1687
> > 
> >  Its not my PR but ll be happy to assist if there is anything I can
> do
> > to
> >  help.
> > 
> >  On 30 Aug 2017 22:46, "Daniel Ribeiro"
>  > >
> >  wrote:
> > 
> > > It would be great to get a bump on pubsub
> > > 
> > dependency.
> > >> It
> > > is
> > > currently very outdated (v1-rev10-1.22.0, which was released over a
> > >> year
> > > ago
> > >  > > api-services-pubsub/v1-rev10-1.22.0/>
> > > ).
> > >
> > >
> > > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > > kirpic...@google.com.invalid> wrote:
> > >
> > >> Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> >  Definitely
> > >> agree that at least the first one must be fixed.
> > >>
> > >> Here's the current burndown list
> > >> https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
> we
> > > should
> > >> clean it up.
> > >>
> > >> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía 
> >  wrote:
> > >>
> > >>> The current master has accumulated a good amount of nice features
> > >>> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR
> that
> > I
> > >>> think are important to check/solve before the cut:
> > >>>
> > >>> BEAM-2516 (this is a regression on the performance of Direct
> runner
> >  on
> > >>> Java). We had never really defined if a performance regression is
> > >>> critical to be a blocker. I executed WordCount with the
> > kinglear.txt
> > >>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > >>> found that the execution time passed from 5s to 126s. So maybe we
> >  need
> > >>> to review this one before the release. I can understand if others
> > >>> consider this a minor issue because the Direct runner is not
> > supposed
> > >>> to be used for production, but this performance regression can
> > cause
> >  a
> > >>> bad impression for a casual user starting with Beam.
> > >>>
> > >>> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I
> > think
> > >>> this one is a nice to have. I am not sure that I can tackle it
> for
> >  the
> > >>> wednesday cut. I’m OOO until the beginning of next week, but
> maybe
> > >>> someone else can take a look. In the worst case this is not a
> > release
> > >>> blocker but definitely a really nice fix to include.
> > >>>
> > >>>
> > >>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > >>>  wrote:
> >  I'd like to get the following PRs into 2.2.0:
>

Re: Beam 2.2.0 release

2017-08-31 Thread Eugene Kirpichov

There are a lot of users including very large production customers who have
been asking specifically for the features that are in 2.2.0 (most of them
accumulated while 2.1.0 was being iterated on) - mostly I'm referring to
the vastly improved file IO - and they have been hesitant to use Beam at
HEAD in production. I think the slight unusualness of having a release
published soon after the previous release is a small price to pay for
helping those users :)

On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré  wrote:

> As we released 2.1.0 couple of weeks ago, it could sound weird to the
> users to
> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1 If it's
> new
> features, why not having a release pace in October (2.2.0) ?
>
> Thoughts ?
>
> Regards
> JB
>
> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0 for
> > October. I don't see a reason to delay 2.2.0 until October: there's a
> huge
> > amount of features worth releasing between when 2.1.0 was cut and the
> > current HEAD.
> >
> > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré 
> > wrote:
> >
> >> With a 2.2.0 in October, I think we can try to move forward on RedisIO.
> >>
> >> I'm now back from vacation and I will resume the work on this IO.
> >>
> >> Regards
> >> JB
> >>
> >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of review
> >> remaining
> >>> last time I checked on the PR.
> >>>
> >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> >>> vonehrenh...@gmail.com> wrote:
> >>>
>  Any chance to get the RedisIO in this release?
>  [BEAM-1017] Add RedisIO #1687
> 
>  Its not my PR but ll be happy to assist if there is anything I can do
> to
>  help.
> 
>  On 30 Aug 2017 22:46, "Daniel Ribeiro"  >
>  wrote:
> 
> > It would be great to get a bump on pubsub
> > 
> dependency.
> >> It
> > is
> > currently very outdated (v1-rev10-1.22.0, which was released over a
> >> year
> > ago
> >  > api-services-pubsub/v1-rev10-1.22.0/>
> > ).
> >
> >
> > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > kirpic...@google.com.invalid> wrote:
> >
> >> Thanks Ismael. I've marked these two issues for fix in 2.2.0.
>  Definitely
> >> agree that at least the first one must be fixed.
> >>
> >> Here's the current burndown list
> >> https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> > should
> >> clean it up.
> >>
> >> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía 
>  wrote:
> >>
> >>> The current master has accumulated a good amount of nice features
> >>> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that
> I
> >>> think are important to check/solve before the cut:
> >>>
> >>> BEAM-2516 (this is a regression on the performance of Direct runner
>  on
> >>> Java). We had never really defined if a performance regression is
> >>> critical to be a blocker. I executed WordCount with the
> kinglear.txt
> >>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> >>> found that the execution time passed from 5s to 126s. So maybe we
>  need
> >>> to review this one before the release. I can understand if others
> >>> consider this a minor issue because the Direct runner is not
> supposed
> >>> to be used for production, but this performance regression can
> cause
>  a
> >>> bad impression for a casual user starting with Beam.
> >>>
> >>> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I
> think
> >>> this one is a nice to have. I am not sure that I can tackle it for
>  the
> >>> wednesday cut. I’m OOO until the beginning of next week, but maybe
> >>> someone else can take a look. In the worst case this is not a
> release
> >>> blocker but definitely a really nice fix to include.
> >>>
> >>>
> >>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> >>>  wrote:
>  I'd like to get the following PRs into 2.2.0:
> 
>  #3765  [BEAM-2753
>  ] Fixes
>  translation
> >> of
>  WriteFiles side inputs (important bugfix for DynamicDestinations
> in
> >>> files)
>  #3725  [BEAM-2827
>  ] Introduces
>  AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> >> important
>  features)
>  #3759

Re: Beam 2.2.0 release

2017-08-31 Thread Jean-Baptiste Onofré

As we released 2.1.0 couple of weeks ago, it could sound weird to the users to 
do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1 If it's new 
features, why not having a release pace in October (2.2.0) ?


Thoughts ?

Regards
JB

On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:

I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0 for
October. I don't see a reason to delay 2.2.0 until October: there's a huge
amount of features worth releasing between when 2.1.0 was cut and the
current HEAD.

On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré 
wrote:


With a 2.2.0 in October, I think we can try to move forward on RedisIO.

I'm now back from vacation and I will resume the work on this IO.

Regards
JB

On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:

RedisIO in 2.2.0 is very unlikely. There's still a lot of review

remaining

last time I checked on the PR.

On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
vonehrenh...@gmail.com> wrote:


Any chance to get the RedisIO in this release?
[BEAM-1017] Add RedisIO #1687

Its not my PR but ll be happy to assist if there is anything I can do to
help.

On 30 Aug 2017 22:46, "Daniel Ribeiro" 
wrote:


It would be great to get a bump on pubsub
 dependency.

It

is
currently very outdated (v1-rev10-1.22.0, which was released over a

year

ago

).


On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:


Thanks Ismael. I've marked these two issues for fix in 2.2.0.

Definitely

agree that at least the first one must be fixed.

Here's the current burndown list
https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we

should

clean it up.

On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía 

wrote:



The current master has accumulated a good amount of nice features
since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
think are important to check/solve before the cut:

BEAM-2516 (this is a regression on the performance of Direct runner

on

Java). We had never really defined if a performance regression is
critical to be a blocker. I executed WordCount with the kinglear.txt
(170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
found that the execution time passed from 5s to 126s. So maybe we

need

to review this one before the release. I can understand if others
consider this a minor issue because the Direct runner is not supposed
to be used for production, but this performance regression can cause

a

bad impression for a casual user starting with Beam.

BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
this one is a nice to have. I am not sure that I can tackle it for

the

wednesday cut. I’m OOO until the beginning of next week, but maybe
someone else can take a look. In the worst case this is not a release
blocker but definitely a really nice fix to include.


On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
 wrote:

I'd like to get the following PRs into 2.2.0:

#3765  [BEAM-2753
] Fixes

translation

of

WriteFiles side inputs (important bugfix for DynamicDestinations in

files)

#3725  [BEAM-2827
] Introduces
AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few

important

features)
#3759  [BEAM-2828
] Moves Match

into

FileIO.match()/matchAll() (to prevent releasing current
Match.filepatterns() into 2.2.0 and then having to keep it under

that

name)


On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu 

wrote:



Glad to see that 2.2.0 is coming. Can we include SQL feature in

next

release? We're in the final stage and expect to merge back to

master

this

week.

On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax

Re: Beam 2.2.0 release

2017-08-31 Thread Eugene Kirpichov

I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0 for
October. I don't see a reason to delay 2.2.0 until October: there's a huge
amount of features worth releasing between when 2.1.0 was cut and the
current HEAD.

On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré 
wrote:

> With a 2.2.0 in October, I think we can try to move forward on RedisIO.
>
> I'm now back from vacation and I will resume the work on this IO.
>
> Regards
> JB
>
> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> > RedisIO in 2.2.0 is very unlikely. There's still a lot of review
> remaining
> > last time I checked on the PR.
> >
> > On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> > vonehrenh...@gmail.com> wrote:
> >
> >> Any chance to get the RedisIO in this release?
> >> [BEAM-1017] Add RedisIO #1687
> >>
> >> Its not my PR but ll be happy to assist if there is anything I can do to
> >> help.
> >>
> >> On 30 Aug 2017 22:46, "Daniel Ribeiro" 
> >> wrote:
> >>
> >>> It would be great to get a bump on pubsub
> >>>  dependency.
> It
> >>> is
> >>> currently very outdated (v1-rev10-1.22.0, which was released over a
> year
> >>> ago
> >>>  >>> api-services-pubsub/v1-rev10-1.22.0/>
> >>> ).
> >>>
> >>>
> >>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> >>> kirpic...@google.com.invalid> wrote:
> >>>
>  Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> >> Definitely
>  agree that at least the first one must be fixed.
> 
>  Here's the current burndown list
>  https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> >>> should
>  clean it up.
> 
>  On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía 
> >> wrote:
> 
> > The current master has accumulated a good amount of nice features
> > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> > think are important to check/solve before the cut:
> >
> > BEAM-2516 (this is a regression on the performance of Direct runner
> >> on
> > Java). We had never really defined if a performance regression is
> > critical to be a blocker. I executed WordCount with the kinglear.txt
> > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > found that the execution time passed from 5s to 126s. So maybe we
> >> need
> > to review this one before the release. I can understand if others
> > consider this a minor issue because the Direct runner is not supposed
> > to be used for production, but this performance regression can cause
> >> a
> > bad impression for a casual user starting with Beam.
> >
> > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> > this one is a nice to have. I am not sure that I can tackle it for
> >> the
> > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > someone else can take a look. In the worst case this is not a release
> > blocker but definitely a really nice fix to include.
> >
> >
> > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> >  wrote:
> >> I'd like to get the following PRs into 2.2.0:
> >>
> >> #3765  [BEAM-2753
> >> ] Fixes
> >> translation
>  of
> >> WriteFiles side inputs (important bugfix for DynamicDestinations in
> > files)
> >> #3725  [BEAM-2827
> >> ] Introduces
> >> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
>  important
> >> features)
> >> #3759  [BEAM-2828
> >> ] Moves Match
> >> into
> >> FileIO.match()/matchAll() (to prevent releasing current
> >> Match.filepatterns() into 2.2.0 and then having to keep it under
> >> that
> > name)
> >>
> >> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu 
>  wrote:
> >>
> >>> Glad to see that 2.2.0 is coming. Can we include SQL feature in
> >> next
> >>> release? We're in the final stage and expect to merge back to
> >> master
> > this
> >>> week.
> >>>
> >>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> >>>  >
> >>> wrote:
> >>>
>  Now that Beam 2.1.0 has finally completed, I think we should cut
>  Beam
> >>> 2.2.0
>  soon. I volunteer to coordinate this release.
> 
>  Are there any pending pull requests that people think should be
>  merged
>  before we cut 2.2.0? If so, please let me know soon, as I would
> >>> like
> > to
> >>> cut
>  by

Re: Beam 2.2.0 release

2017-08-30 Thread Jean-Baptiste Onofré


With a 2.2.0 in October, I think we can try to move forward on RedisIO.

I'm now back from vacation and I will resume the work on this IO.

Regards
JB

On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:

RedisIO in 2.2.0 is very unlikely. There's still a lot of review remaining
last time I checked on the PR.

On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
vonehrenh...@gmail.com> wrote:


Any chance to get the RedisIO in this release?
[BEAM-1017] Add RedisIO #1687

Its not my PR but ll be happy to assist if there is anything I can do to
help.

On 30 Aug 2017 22:46, "Daniel Ribeiro" 
wrote:


It would be great to get a bump on pubsub
 dependency. It
is
currently very outdated (v1-rev10-1.22.0, which was released over a year
ago

).


On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:


Thanks Ismael. I've marked these two issues for fix in 2.2.0.

Definitely

agree that at least the first one must be fixed.

Here's the current burndown list
https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we

should

clean it up.

On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía 

wrote:



The current master has accumulated a good amount of nice features
since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
think are important to check/solve before the cut:

BEAM-2516 (this is a regression on the performance of Direct runner

on

Java). We had never really defined if a performance regression is
critical to be a blocker. I executed WordCount with the kinglear.txt
(170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
found that the execution time passed from 5s to 126s. So maybe we

need

to review this one before the release. I can understand if others
consider this a minor issue because the Direct runner is not supposed
to be used for production, but this performance regression can cause

a

bad impression for a casual user starting with Beam.

BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
this one is a nice to have. I am not sure that I can tackle it for

the

wednesday cut. I’m OOO until the beginning of next week, but maybe
someone else can take a look. In the worst case this is not a release
blocker but definitely a really nice fix to include.


On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
 wrote:

I'd like to get the following PRs into 2.2.0:

#3765  [BEAM-2753
] Fixes

translation

of

WriteFiles side inputs (important bugfix for DynamicDestinations in

files)

#3725  [BEAM-2827
] Introduces
AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few

important

features)
#3759  [BEAM-2828
] Moves Match

into

FileIO.match()/matchAll() (to prevent releasing current
Match.filepatterns() into 2.2.0 and then having to keep it under

that

name)


On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu 

wrote:



Glad to see that 2.2.0 is coming. Can we include SQL feature in

next

release? We're in the final stage and expect to merge back to

master

this

week.

On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax

Re: Beam 2.2.0 release

2017-08-30 Thread Jean-Baptiste Onofré


Hi Reuven,

sorry I missed the initial message in the thread.

Thanks to be volunteer for this release !

For the target deadline, I think October make sense. We have some new feature in 
progress (M/R runner, Spark 2.x runner,  etc). We can finalize it during 
September and so, October is a good timing for 2.2.0 release IMHO.


Regards
JB

On 08/30/2017 08:27 PM, Reuven Lax wrote:

Now that Beam 2.1.0 has finally completed, I think we should cut Beam 2.2.0
soon. I volunteer to coordinate this release.

Are there any pending pull requests that people think should be merged
before we cut 2.2.0? If so, please let me know soon, as I would like to cut
by Wednesday of next week.

Thanks,

Reuven



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Beam 2.2.0 release

2017-08-30 Thread Jean-Baptiste Onofré


Hi Kenn,

thanks !

I think we can target 2.2.0 release for October. Thoughts ?

I'm also volunteer to manage this next release.

Regards
JB

On 08/31/2017 05:57 AM, Kenneth Knowles wrote:

I went ahead and set up https://s.apache.org/beam-2.2.0-burndown

On Wed, Aug 30, 2017 at 2:27 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:


RedisIO in 2.2.0 is very unlikely. There's still a lot of review remaining
last time I checked on the PR.

On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
vonehrenh...@gmail.com> wrote:


Any chance to get the RedisIO in this release?
[BEAM-1017] Add RedisIO #1687

Its not my PR but ll be happy to assist if there is anything I can do to
help.

On 30 Aug 2017 22:46, "Daniel Ribeiro" 
wrote:


It would be great to get a bump on pubsub
 dependency.

It

is
currently very outdated (v1-rev10-1.22.0, which was released over a

year

ago

).


On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:


Thanks Ismael. I've marked these two issues for fix in 2.2.0.

Definitely

agree that at least the first one must be fixed.

Here's the current burndown list
https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we

should

clean it up.

On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía 

wrote:



The current master has accumulated a good amount of nice features
since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that

I

think are important to check/solve before the cut:

BEAM-2516 (this is a regression on the performance of Direct runner

on

Java). We had never really defined if a performance regression is
critical to be a blocker. I executed WordCount with the

kinglear.txt

(170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
found that the execution time passed from 5s to 126s. So maybe we

need

to review this one before the release. I can understand if others
consider this a minor issue because the Direct runner is not

supposed

to be used for production, but this performance regression can

cause

a

bad impression for a casual user starting with Beam.

BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I

think

this one is a nice to have. I am not sure that I can tackle it for

the

wednesday cut. I’m OOO until the beginning of next week, but maybe
someone else can take a look. In the worst case this is not a

release

blocker but definitely a really nice fix to include.


On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
 wrote:

I'd like to get the following PRs into 2.2.0:

#3765  [BEAM-2753
] Fixes

translation

of

WriteFiles side inputs (important bugfix for DynamicDestinations

in

files)

#3725  [BEAM-2827
] Introduces
AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few

important

features)
#3759  [BEAM-2828
] Moves Match

into

FileIO.match()/matchAll() (to prevent releasing current
Match.filepatterns() into 2.2.0 and then having to keep it under

that

name)


On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu 

wrote:



Glad to see that 2.2.0 is coming. Can we include SQL feature in

next

release? We're in the final stage and expect to merge back to

master

this

week.

On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax

Re: Beam 2.2.0 release

2017-08-30 Thread Kenneth Knowles

I went ahead and set up https://s.apache.org/beam-2.2.0-burndown

On Wed, Aug 30, 2017 at 2:27 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> RedisIO in 2.2.0 is very unlikely. There's still a lot of review remaining
> last time I checked on the PR.
>
> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> vonehrenh...@gmail.com> wrote:
>
> > Any chance to get the RedisIO in this release?
> > [BEAM-1017] Add RedisIO #1687
> >
> > Its not my PR but ll be happy to assist if there is anything I can do to
> > help.
> >
> > On 30 Aug 2017 22:46, "Daniel Ribeiro" 
> > wrote:
> >
> > > It would be great to get a bump on pubsub
> > >  dependency.
> It
> > > is
> > > currently very outdated (v1-rev10-1.22.0, which was released over a
> year
> > > ago
> > >  > > api-services-pubsub/v1-rev10-1.22.0/>
> > > ).
> > >
> > >
> > > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > > kirpic...@google.com.invalid> wrote:
> > >
> > > > Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> > Definitely
> > > > agree that at least the first one must be fixed.
> > > >
> > > > Here's the current burndown list
> > > > https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> > > should
> > > > clean it up.
> > > >
> > > > On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía 
> > wrote:
> > > >
> > > > > The current master has accumulated a good amount of nice features
> > > > > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that
> I
> > > > > think are important to check/solve before the cut:
> > > > >
> > > > > BEAM-2516 (this is a regression on the performance of Direct runner
> > on
> > > > > Java). We had never really defined if a performance regression is
> > > > > critical to be a blocker. I executed WordCount with the
> kinglear.txt
> > > > > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > > > > found that the execution time passed from 5s to 126s. So maybe we
> > need
> > > > > to review this one before the release. I can understand if others
> > > > > consider this a minor issue because the Direct runner is not
> supposed
> > > > > to be used for production, but this performance regression can
> cause
> > a
> > > > > bad impression for a casual user starting with Beam.
> > > > >
> > > > > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I
> think
> > > > > this one is a nice to have. I am not sure that I can tackle it for
> > the
> > > > > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > > > > someone else can take a look. In the worst case this is not a
> release
> > > > > blocker but definitely a really nice fix to include.
> > > > >
> > > > >
> > > > > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > > > >  wrote:
> > > > > > I'd like to get the following PRs into 2.2.0:
> > > > > >
> > > > > > #3765  [BEAM-2753
> > > > > > ] Fixes
> > translation
> > > > of
> > > > > > WriteFiles side inputs (important bugfix for DynamicDestinations
> in
> > > > > files)
> > > > > > #3725  [BEAM-2827
> > > > > > ] Introduces
> > > > > > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> > > > important
> > > > > > features)
> > > > > > #3759  [BEAM-2828
> > > > > > ] Moves Match
> > into
> > > > > > FileIO.match()/matchAll() (to prevent releasing current
> > > > > > Match.filepatterns() into 2.2.0 and then having to keep it under
> > that
> > > > > name)
> > > > > >
> > > > > > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu 
> > > > wrote:
> > > > > >
> > > > > >> Glad to see that 2.2.0 is coming. Can we include SQL feature in
> > next
> > > > > >> release? We're in the final stage and expect to merge back to
> > master
> > > > > this
> > > > > >> week.
> > > > > >>
> > > > > >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> > >  > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Now that Beam 2.1.0 has finally completed, I think we should
> cut
> > > > Beam
> > > > > >> 2.2.0
> > > > > >> > soon. I volunteer to coordinate this release.
> > > > > >> >
> > > > > >> > Are there any pending pull requests that people think should
> be
> > > > merged
> > > > > >> > before we cut 2.2.0? If so, please let me know soon, as I
> would
> > > like
> > > > > to
> > > > > >> cut
> > > > > >> > by Wednesday of next week.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > Reuven
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> 
> > > > > >> Mingmin
> > > > > >>

Re: Beam 2.2.0 release

2017-08-30 Thread Eugene Kirpichov

RedisIO in 2.2.0 is very unlikely. There's still a lot of review remaining
last time I checked on the PR.

On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
vonehrenh...@gmail.com> wrote:

> Any chance to get the RedisIO in this release?
> [BEAM-1017] Add RedisIO #1687
>
> Its not my PR but ll be happy to assist if there is anything I can do to
> help.
>
> On 30 Aug 2017 22:46, "Daniel Ribeiro" 
> wrote:
>
> > It would be great to get a bump on pubsub
> >  dependency. It
> > is
> > currently very outdated (v1-rev10-1.22.0, which was released over a year
> > ago
> >  > api-services-pubsub/v1-rev10-1.22.0/>
> > ).
> >
> >
> > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > kirpic...@google.com.invalid> wrote:
> >
> > > Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> Definitely
> > > agree that at least the first one must be fixed.
> > >
> > > Here's the current burndown list
> > > https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> > should
> > > clean it up.
> > >
> > > On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía 
> wrote:
> > >
> > > > The current master has accumulated a good amount of nice features
> > > > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> > > > think are important to check/solve before the cut:
> > > >
> > > > BEAM-2516 (this is a regression on the performance of Direct runner
> on
> > > > Java). We had never really defined if a performance regression is
> > > > critical to be a blocker. I executed WordCount with the kinglear.txt
> > > > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > > > found that the execution time passed from 5s to 126s. So maybe we
> need
> > > > to review this one before the release. I can understand if others
> > > > consider this a minor issue because the Direct runner is not supposed
> > > > to be used for production, but this performance regression can cause
> a
> > > > bad impression for a casual user starting with Beam.
> > > >
> > > > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> > > > this one is a nice to have. I am not sure that I can tackle it for
> the
> > > > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > > > someone else can take a look. In the worst case this is not a release
> > > > blocker but definitely a really nice fix to include.
> > > >
> > > >
> > > > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > > >  wrote:
> > > > > I'd like to get the following PRs into 2.2.0:
> > > > >
> > > > > #3765  [BEAM-2753
> > > > > ] Fixes
> translation
> > > of
> > > > > WriteFiles side inputs (important bugfix for DynamicDestinations in
> > > > files)
> > > > > #3725  [BEAM-2827
> > > > > ] Introduces
> > > > > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> > > important
> > > > > features)
> > > > > #3759  [BEAM-2828
> > > > > ] Moves Match
> into
> > > > > FileIO.match()/matchAll() (to prevent releasing current
> > > > > Match.filepatterns() into 2.2.0 and then having to keep it under
> that
> > > > name)
> > > > >
> > > > > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu 
> > > wrote:
> > > > >
> > > > >> Glad to see that 2.2.0 is coming. Can we include SQL feature in
> next
> > > > >> release? We're in the final stage and expect to merge back to
> master
> > > > this
> > > > >> week.
> > > > >>
> > > > >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> >  > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Now that Beam 2.1.0 has finally completed, I think we should cut
> > > Beam
> > > > >> 2.2.0
> > > > >> > soon. I volunteer to coordinate this release.
> > > > >> >
> > > > >> > Are there any pending pull requests that people think should be
> > > merged
> > > > >> > before we cut 2.2.0? If so, please let me know soon, as I would
> > like
> > > > to
> > > > >> cut
> > > > >> > by Wednesday of next week.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Reuven
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> 
> > > > >> Mingmin
> > > > >>
> > > >
> > >
> >
>

Re: Beam 2.2.0 release

2017-08-30 Thread Vilhelm von Ehrenheim

Any chance to get the RedisIO in this release?
[BEAM-1017] Add RedisIO #1687

Its not my PR but ll be happy to assist if there is anything I can do to
help.

On 30 Aug 2017 22:46, "Daniel Ribeiro" 
wrote:

> It would be great to get a bump on pubsub
>  dependency. It
> is
> currently very outdated (v1-rev10-1.22.0, which was released over a year
> ago
>  api-services-pubsub/v1-rev10-1.22.0/>
> ).
>
>
> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> kirpic...@google.com.invalid> wrote:
>
> > Thanks Ismael. I've marked these two issues for fix in 2.2.0. Definitely
> > agree that at least the first one must be fixed.
> >
> > Here's the current burndown list
> > https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> should
> > clean it up.
> >
> > On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía  wrote:
> >
> > > The current master has accumulated a good amount of nice features
> > > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> > > think are important to check/solve before the cut:
> > >
> > > BEAM-2516 (this is a regression on the performance of Direct runner on
> > > Java). We had never really defined if a performance regression is
> > > critical to be a blocker. I executed WordCount with the kinglear.txt
> > > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > > found that the execution time passed from 5s to 126s. So maybe we need
> > > to review this one before the release. I can understand if others
> > > consider this a minor issue because the Direct runner is not supposed
> > > to be used for production, but this performance regression can cause a
> > > bad impression for a casual user starting with Beam.
> > >
> > > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> > > this one is a nice to have. I am not sure that I can tackle it for the
> > > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > > someone else can take a look. In the worst case this is not a release
> > > blocker but definitely a really nice fix to include.
> > >
> > >
> > > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > >  wrote:
> > > > I'd like to get the following PRs into 2.2.0:
> > > >
> > > > #3765  [BEAM-2753
> > > > ] Fixes translation
> > of
> > > > WriteFiles side inputs (important bugfix for DynamicDestinations in
> > > files)
> > > > #3725  [BEAM-2827
> > > > ] Introduces
> > > > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> > important
> > > > features)
> > > > #3759  [BEAM-2828
> > > > ] Moves Match into
> > > > FileIO.match()/matchAll() (to prevent releasing current
> > > > Match.filepatterns() into 2.2.0 and then having to keep it under that
> > > name)
> > > >
> > > > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu 
> > wrote:
> > > >
> > > >> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
> > > >> release? We're in the final stage and expect to merge back to master
> > > this
> > > >> week.
> > > >>
> > > >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>  > >
> > > >> wrote:
> > > >>
> > > >> > Now that Beam 2.1.0 has finally completed, I think we should cut
> > Beam
> > > >> 2.2.0
> > > >> > soon. I volunteer to coordinate this release.
> > > >> >
> > > >> > Are there any pending pull requests that people think should be
> > merged
> > > >> > before we cut 2.2.0? If so, please let me know soon, as I would
> like
> > > to
> > > >> cut
> > > >> > by Wednesday of next week.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Reuven
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> 
> > > >> Mingmin
> > > >>
> > >
> >
>

Re: Beam 2.2.0 release

2017-08-30 Thread Daniel Ribeiro

It would be great to get a bump on pubsub
 dependency. It is
currently very outdated (v1-rev10-1.22.0, which was released over a year ago

).


On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> Thanks Ismael. I've marked these two issues for fix in 2.2.0. Definitely
> agree that at least the first one must be fixed.
>
> Here's the current burndown list
> https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we should
> clean it up.
>
> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía  wrote:
>
> > The current master has accumulated a good amount of nice features
> > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> > think are important to check/solve before the cut:
> >
> > BEAM-2516 (this is a regression on the performance of Direct runner on
> > Java). We had never really defined if a performance regression is
> > critical to be a blocker. I executed WordCount with the kinglear.txt
> > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > found that the execution time passed from 5s to 126s. So maybe we need
> > to review this one before the release. I can understand if others
> > consider this a minor issue because the Direct runner is not supposed
> > to be used for production, but this performance regression can cause a
> > bad impression for a casual user starting with Beam.
> >
> > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> > this one is a nice to have. I am not sure that I can tackle it for the
> > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > someone else can take a look. In the worst case this is not a release
> > blocker but definitely a really nice fix to include.
> >
> >
> > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> >  wrote:
> > > I'd like to get the following PRs into 2.2.0:
> > >
> > > #3765  [BEAM-2753
> > > ] Fixes translation
> of
> > > WriteFiles side inputs (important bugfix for DynamicDestinations in
> > files)
> > > #3725  [BEAM-2827
> > > ] Introduces
> > > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> important
> > > features)
> > > #3759  [BEAM-2828
> > > ] Moves Match into
> > > FileIO.match()/matchAll() (to prevent releasing current
> > > Match.filepatterns() into 2.2.0 and then having to keep it under that
> > name)
> > >
> > > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu 
> wrote:
> > >
> > >> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
> > >> release? We're in the final stage and expect to merge back to master
> > this
> > >> week.
> > >>
> > >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax  >
> > >> wrote:
> > >>
> > >> > Now that Beam 2.1.0 has finally completed, I think we should cut
> Beam
> > >> 2.2.0
> > >> > soon. I volunteer to coordinate this release.
> > >> >
> > >> > Are there any pending pull requests that people think should be
> merged
> > >> > before we cut 2.2.0? If so, please let me know soon, as I would like
> > to
> > >> cut
> > >> > by Wednesday of next week.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Reuven
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> 
> > >> Mingmin
> > >>
> >
>

Re: Beam 2.2.0 release

2017-08-30 Thread Eugene Kirpichov

Thanks Ismael. I've marked these two issues for fix in 2.2.0. Definitely
agree that at least the first one must be fixed.

Here's the current burndown list
https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we should
clean it up.

On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía  wrote:

> The current master has accumulated a good amount of nice features
> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> think are important to check/solve before the cut:
>
> BEAM-2516 (this is a regression on the performance of Direct runner on
> Java). We had never really defined if a performance regression is
> critical to be a blocker. I executed WordCount with the kinglear.txt
> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> found that the execution time passed from 5s to 126s. So maybe we need
> to review this one before the release. I can understand if others
> consider this a minor issue because the Direct runner is not supposed
> to be used for production, but this performance regression can cause a
> bad impression for a casual user starting with Beam.
>
> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> this one is a nice to have. I am not sure that I can tackle it for the
> wednesday cut. I’m OOO until the beginning of next week, but maybe
> someone else can take a look. In the worst case this is not a release
> blocker but definitely a really nice fix to include.
>
>
> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>  wrote:
> > I'd like to get the following PRs into 2.2.0:
> >
> > #3765  [BEAM-2753
> > ] Fixes translation of
> > WriteFiles side inputs (important bugfix for DynamicDestinations in
> files)
> > #3725  [BEAM-2827
> > ] Introduces
> > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few important
> > features)
> > #3759  [BEAM-2828
> > ] Moves Match into
> > FileIO.match()/matchAll() (to prevent releasing current
> > Match.filepatterns() into 2.2.0 and then having to keep it under that
> name)
> >
> > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu  wrote:
> >
> >> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
> >> release? We're in the final stage and expect to merge back to master
> this
> >> week.
> >>
> >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax 
> >> wrote:
> >>
> >> > Now that Beam 2.1.0 has finally completed, I think we should cut Beam
> >> 2.2.0
> >> > soon. I volunteer to coordinate this release.
> >> >
> >> > Are there any pending pull requests that people think should be merged
> >> > before we cut 2.2.0? If so, please let me know soon, as I would like
> to
> >> cut
> >> > by Wednesday of next week.
> >> >
> >> > Thanks,
> >> >
> >> > Reuven
> >> >
> >>
> >>
> >>
> >> --
> >> 
> >> Mingmin
> >>
>

Re: Beam 2.2.0 release

2017-08-30 Thread Ismaël Mejía

The current master has accumulated a good amount of nice features
since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
think are important to check/solve before the cut:

BEAM-2516 (this is a regression on the performance of Direct runner on
Java). We had never really defined if a performance regression is
critical to be a blocker. I executed WordCount with the kinglear.txt
(170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
found that the execution time passed from 5s to 126s. So maybe we need
to review this one before the release. I can understand if others
consider this a minor issue because the Direct runner is not supposed
to be used for production, but this performance regression can cause a
bad impression for a casual user starting with Beam.

BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
this one is a nice to have. I am not sure that I can tackle it for the
wednesday cut. I’m OOO until the beginning of next week, but maybe
someone else can take a look. In the worst case this is not a release
blocker but definitely a really nice fix to include.

On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
 wrote:
> I'd like to get the following PRs into 2.2.0:
>
> #3765  [BEAM-2753
> ] Fixes translation of
> WriteFiles side inputs (important bugfix for DynamicDestinations in files)
> #3725  [BEAM-2827
> ] Introduces
> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few important
> features)
> #3759  [BEAM-2828
> ] Moves Match into
> FileIO.match()/matchAll() (to prevent releasing current
> Match.filepatterns() into 2.2.0 and then having to keep it under that name)
>
> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu  wrote:
>
>> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
>> release? We're in the final stage and expect to merge back to master this
>> week.
>>
>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax 
>> wrote:
>>
>> > Now that Beam 2.1.0 has finally completed, I think we should cut Beam
>> 2.2.0
>> > soon. I volunteer to coordinate this release.
>> >
>> > Are there any pending pull requests that people think should be merged
>> > before we cut 2.2.0? If so, please let me know soon, as I would like to
>> cut
>> > by Wednesday of next week.
>> >
>> > Thanks,
>> >
>> > Reuven
>> >
>>
>>
>>
>> --
>> 
>> Mingmin
>>

Re: Beam 2.2.0 release

2017-08-30 Thread Eugene Kirpichov

I'd like to get the following PRs into 2.2.0:

#3765  [BEAM-2753
] Fixes translation of
WriteFiles side inputs (important bugfix for DynamicDestinations in files)
#3725  [BEAM-2827
] Introduces
AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few important
features)
#3759  [BEAM-2828
] Moves Match into
FileIO.match()/matchAll() (to prevent releasing current
Match.filepatterns() into 2.2.0 and then having to keep it under that name)

On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu  wrote:

> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
> release? We're in the final stage and expect to merge back to master this
> week.
>
> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax 
> wrote:
>
> > Now that Beam 2.1.0 has finally completed, I think we should cut Beam
> 2.2.0
> > soon. I volunteer to coordinate this release.
> >
> > Are there any pending pull requests that people think should be merged
> > before we cut 2.2.0? If so, please let me know soon, as I would like to
> cut
> > by Wednesday of next week.
> >
> > Thanks,
> >
> > Reuven
> >
>
>
>
> --
> 
> Mingmin
>

Re: Beam 2.2.0 release

2017-08-30 Thread Mingmin Xu

Glad to see that 2.2.0 is coming. Can we include SQL feature in next
release? We're in the final stage and expect to merge back to master this
week.

On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax 
wrote:

> Now that Beam 2.1.0 has finally completed, I think we should cut Beam 2.2.0
> soon. I volunteer to coordinate this release.
>
> Are there any pending pull requests that people think should be merged
> before we cut 2.2.0? If so, please let me know soon, as I would like to cut
> by Wednesday of next week.
>
> Thanks,
>
> Reuven
>

-- 

Mingmin

Beam 2.2.0 release

2017-08-30 Thread Reuven Lax

Now that Beam 2.1.0 has finally completed, I think we should cut Beam 2.2.0
soon. I volunteer to coordinate this release.

Are there any pending pull requests that people think should be merged
before we cut 2.2.0? If so, please let me know soon, as I would like to cut
by Wednesday of next week.

Thanks,

Reuven

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Re: Beam 2.2.0 release

Beam 2.2.0 release

25 matches

Site Navigation

Mail list logo

Footer information