Re: [CANCEL][VOTE] Release 2.3.0, release candidate #2

[Jean-Baptiste Onofré]

Hi guys

Quick update for RC3: BEAM-3668 & BEAM-3519 have a PR in review (shading netty
in GCP IO).
Hopefully, the PR will be merge later today, then it will be cherry-pick on
release-2.3.0 branch.

As soon as it's done, I will cut a RC3 (hopefully tonight or tomorrow).

Regards
JB

On 02/10/2018 02:11 AM, Chamikara Jayalath wrote:
> https://github.com/apache/beam/pull/4651 is in review.
> 
> On Fri, Feb 9, 2018 at 1:38 PM Reuven Lax  > wrote:
> 
> Late to respond, but +1 to this decision.
> 
> On Fri, Feb 9, 2018 at 2:26 AM, Jean-Baptiste Onofré  > wrote:
> 
> Hi guys,
> 
> due to BEAM-3519 [1], I cancel RC2.
> 
> We have to shade netty version in the IOs to avoid issue with netty
> version in
> runners (Spark in that case).
> 
> We gonna work on a fix (I should be pretty fast), and I will cut a 
> RC3 when
> cherry-pick on release-2.3.0 branch.
> 
> Regards
> JB
> 
> [1] https://issues.apache.org/jira/browse/BEAM-3519
> 
> On 02/08/2018 06:37 AM, Jean-Baptiste Onofré wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #2 for the version
> 2.3.0, as
> > follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific 
> comments)
> >
> >
> > The complete staging area is available for your review, which 
> includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to 
> dist.apache.org
>  [2],
> > which is signed with the key with fingerprint C8282E76 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.3.0-RC2" [5],
> > * website pull request listing the release and publishing the API
> reference
> > manual [6].
> > * Java artifacts were built with Maven 3.3.9 and Oracle JDK 
> 1.8.0_111.
> > * Python artifacts are deployed along with the source release to the
> > dist.apache.org  [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by 
> majority
> approval,
> > with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > JB
> >
> > [1]
> >
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4] 
> https://repository.apache.org/content/repositories/orgapachebeam-1027/
> > [5] https://github.com/apache/beam/tree/v2.3.0-RC2
> > [6] https://github.com/apache/beam-site/pull/381
> >
> 
> --
> Jean-Baptiste Onofré
> jbono...@apache.org 
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [SQL] Reject unsupported inputs to Joins

[Robert Bradshaw]

Huge +1 to only allowing single-firing triggers through for sane
semantics in this case. Commented on the doc.

On Fri, Feb 9, 2018 at 5:08 PM, Anton Kedin  wrote:
> Hi,
>
> If you're not using Beam SQL JOINs, you're not affected.
>
> In Short
>
> Beam SQL JOIN operation does not work well with multiple trigger firings.
>
> More Detail
>
> Beam SQL JOIN is a CoGBK under the hood. It joins available elements
> per-pane. This means that:
>
> in discarding mode we're joining only new elements which arrived since last
> trigger firing, forgetting about any past elements;
> in accumulating mode in addition to joining new elements we're also emitting
> join results we have already emitted when trigger fired last time;
>
> This behavior cannot be configured or handled in pure SQL.
>
> Even More Detail
>
> Here
>
> Proposal
>
> Short term, allow only trigger configuration which we can reason about,
> rejecting anything else.
>
> For example we know that non-global windows with default trigger with zero
> allowed lateness will fire once per window. In this case we will join all
> elements in the window once after watermark is past end of window.
>
> Long term, retractions will allow correct handling of this situation.
>
> Jira, Pull Request
>
> Regards,
> Anton
>

Re: [CANCEL][VOTE] Release 2.3.0, release candidate #2

[Chamikara Jayalath]

https://github.com/apache/beam/pull/4651 is in review.

On Fri, Feb 9, 2018 at 1:38 PM Reuven Lax  wrote:

> Late to respond, but +1 to this decision.
>
> On Fri, Feb 9, 2018 at 2:26 AM, Jean-Baptiste Onofré 
> wrote:
>
>> Hi guys,
>>
>> due to BEAM-3519 [1], I cancel RC2.
>>
>> We have to shade netty version in the IOs to avoid issue with netty
>> version in
>> runners (Spark in that case).
>>
>> We gonna work on a fix (I should be pretty fast), and I will cut a RC3
>> when
>> cherry-pick on release-2.3.0 branch.
>>
>> Regards
>> JB
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-3519
>>
>> On 02/08/2018 06:37 AM, Jean-Baptiste Onofré wrote:
>> > Hi everyone,
>> >
>> > Please review and vote on the release candidate #2 for the version
>> 2.3.0, as
>> > follows:
>> >
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific comments)
>> >
>> >
>> > The complete staging area is available for your review, which includes:
>> > * JIRA release notes [1],
>> > * the official Apache source release to be deployed to dist.apache.org
>> [2],
>> > which is signed with the key with fingerprint C8282E76 [3],
>> > * all artifacts to be deployed to the Maven Central Repository [4],
>> > * source code tag "v2.3.0-RC2" [5],
>> > * website pull request listing the release and publishing the API
>> reference
>> > manual [6].
>> > * Java artifacts were built with Maven 3.3.9 and Oracle JDK 1.8.0_111.
>> > * Python artifacts are deployed along with the source release to the
>> > dist.apache.org [2].
>> >
>> > The vote will be open for at least 72 hours. It is adopted by majority
>> approval,
>> > with at least 3 PMC affirmative votes.
>> >
>> > Thanks,
>> > JB
>> >
>> > [1]
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608
>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/
>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> > [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1027/
>> > [5] https://github.com/apache/beam/tree/v2.3.0-RC2
>> > [6] https://github.com/apache/beam-site/pull/381
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>

[SQL] Reject unsupported inputs to Joins

[Anton Kedin]

Hi,

If you're not using Beam SQL JOINs, you're not affected.

In Short

Beam SQL JOIN operation does not work well with multiple trigger firings.

More Detail

Beam SQL JOIN is a CoGBK under the hood. It joins available elements
per-pane. This means that:

   - in discarding mode we're joining only new elements which arrived since
   last trigger firing, forgetting about any past elements;
   - in accumulating mode in addition to joining new elements we're also
   emitting join results we have already emitted when trigger fired last time;

This behavior cannot be configured or handled in pure SQL.

Even More Detail

Here


Proposal

Short term, allow only trigger configuration which we can reason about,
rejecting anything else.

For example we know that non-global windows with default trigger with zero
allowed lateness will fire once per window. In this case we will join all
elements in the window once after watermark is past end of window.

Long term, retractions will allow correct handling of this situation.

Jira , Pull Request


Regards,
Anton

Re: [CANCEL][VOTE] Release 2.3.0, release candidate #2

[Reuven Lax]

Late to respond, but +1 to this decision.

On Fri, Feb 9, 2018 at 2:26 AM, Jean-Baptiste Onofré 
wrote:

> Hi guys,
>
> due to BEAM-3519 [1], I cancel RC2.
>
> We have to shade netty version in the IOs to avoid issue with netty
> version in
> runners (Spark in that case).
>
> We gonna work on a fix (I should be pretty fast), and I will cut a RC3 when
> cherry-pick on release-2.3.0 branch.
>
> Regards
> JB
>
> [1] https://issues.apache.org/jira/browse/BEAM-3519
>
> On 02/08/2018 06:37 AM, Jean-Baptiste Onofré wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #2 for the version
> 2.3.0, as
> > follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> [2],
> > which is signed with the key with fingerprint C8282E76 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.3.0-RC2" [5],
> > * website pull request listing the release and publishing the API
> reference
> > manual [6].
> > * Java artifacts were built with Maven 3.3.9 and Oracle JDK 1.8.0_111.
> > * Python artifacts are deployed along with the source release to the
> > dist.apache.org [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> approval,
> > with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > JB
> >
> > [1]
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12341608
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4] https://repository.apache.org/content/repositories/
> orgapachebeam-1027/
> > [5] https://github.com/apache/beam/tree/v2.3.0-RC2
> > [6] https://github.com/apache/beam-site/pull/381
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [jira] [Commented] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Kenneth Knowles]

I think we should take it case-by-case. Guava universal shading is already
quite a mess - build super slow, copies of Guava in jars that don't even
depend on Guava, things shaded that break API surfaces. In this case
there's no mystery what needs to happen, is there?

Kenn

On Fri, Feb 9, 2018 at 11:47 AM, Lukasz Cwik  wrote:

> Shading netty everywhere would prevent conflicts with user supplied
> versions of netty.
>
> On Fri, Feb 9, 2018 at 1:03 AM, Ismaël Mejía (JIRA) 
> wrote:
>
>>
>> [ https://issues.apache.org/jira/browse/BEAM-3519?page=com.
>> atlassian.jira.plugin.system.issuetabpanels:comment-tabpane
>> l&focusedCommentId=16358132#comment-16358132 ]
>>
>> Ismaël Mejía commented on BEAM-3519:
>> 
>>
>> The explicit issue probably existed before (GCPIO leaking netty), but it
>> was not detected because the previous version of the Spark runner (based on
>> Spark 1.x) used a version of netty that didn't complain.
>> In any case shading correctly GCPIO would be great to avoid this issue in
>> the future in other runners too.
>>
>> > GCP IO exposes netty on its API surface, causing conflicts with runners
>> > ---
>> >
>> > Key: BEAM-3519
>> > URL: https://issues.apache.org/jira/browse/BEAM-3519
>> > Project: Beam
>> >  Issue Type: Bug
>> >  Components: sdk-java-gcp
>> >Reporter: Ismaël Mejía
>> >Assignee: Chamikara Jayalath
>> >Priority: Blocker
>> > Fix For: 2.3.0
>> >
>> >
>> > Google Cloud Platform IOs module leaks netty this causes conflicts in
>> particular with execution systems that use conflicting versions of such
>> modules.
>> >  For the case there is a dependency conflict with the Spark Runner
>> version of netty, see: BEAM-3492
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v7.6.3#76005)
>>
>
>

Re: [jira] [Commented] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Lukasz Cwik]

Shading netty everywhere would prevent conflicts with user supplied
versions of netty.

On Fri, Feb 9, 2018 at 1:03 AM, Ismaël Mejía (JIRA)  wrote:

>
> [ https://issues.apache.org/jira/browse/BEAM-3519?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=16358132#comment-16358132 ]
>
> Ismaël Mejía commented on BEAM-3519:
> 
>
> The explicit issue probably existed before (GCPIO leaking netty), but it
> was not detected because the previous version of the Spark runner (based on
> Spark 1.x) used a version of netty that didn't complain.
> In any case shading correctly GCPIO would be great to avoid this issue in
> the future in other runners too.
>
> > GCP IO exposes netty on its API surface, causing conflicts with runners
> > ---
> >
> > Key: BEAM-3519
> > URL: https://issues.apache.org/jira/browse/BEAM-3519
> > Project: Beam
> >  Issue Type: Bug
> >  Components: sdk-java-gcp
> >Reporter: Ismaël Mejía
> >Assignee: Chamikara Jayalath
> >Priority: Blocker
> > Fix For: 2.3.0
> >
> >
> > Google Cloud Platform IOs module leaks netty this causes conflicts in
> particular with execution systems that use conflicting versions of such
> modules.
> >  For the case there is a dependency conflict with the Spark Runner
> version of netty, see: BEAM-3492
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>

Re: [jira] [Commented] (BEAM-2591) Python shim for submitting to FlinkRunner

[Kenneth Knowles]

Just seeing this. I'm not sure if a Jira integration has regressed, but
these emailed replies didn't make it to the ticket.

Kenn

On Thu, Feb 8, 2018 at 4:31 PM, Robert Bradshaw  wrote:

> From the Python point of view, I think it makes sense to configure the
> "shim" runner with either (1) a command to run to launch and tear down
> the service or (2) an already existing endpoint. We could make (1)
> more convenient by providing some ready-made commands in some
> namespace that use the existing containers and/or set up good
> defaults.
>
> On Thu, Feb 8, 2018 at 4:23 PM, Kenneth Knowles (JIRA) 
> wrote:
> >
> > [ https://issues.apache.org/jira/browse/BEAM-2591?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=16357748#comment-16357748 ]
> >
> > Kenneth Knowles commented on BEAM-2591:
> > ---
> >
> > I think probably best to get feedback on idiomatic uses from
> [~robertwb], [~altay], [~jgrier], etc.
> >
> > Basically I imagine a user will
> > 1. test bounded smoke test with the Python direct runner
> > 2. try the same on a local flink (or whatever is a one-liner to launch
> and teardown) and also some unbounded data at that point
> > 3. go big by:
> > a. point the pipeline at a production endpoint where they don't know
> or care what tech is behind it
> > b. or launch their pipeline main alongside flink as a complete app
> via k8s or a bespoke setup
> > c. ???
> >
> > This ticket had been filed for 2 with the idea that it would be reusable
> for 3 but actually I think it should just be split into separate tasks and
> this focus on 2.
> >
> >> Python shim for submitting to FlinkRunner
> >> -
> >>
> >> Key: BEAM-2591
> >> URL: https://issues.apache.org/jira/browse/BEAM-2591
> >> Project: Beam
> >>  Issue Type: Sub-task
> >>  Components: runner-flink, sdk-py-core
> >>Reporter: Kenneth Knowles
> >>Priority: Major
> >>  Labels: portability
> >>
> >> Whatever the result of https://s.apache.org/beam-job-api, Python users
> will need to be able to pass --runner=FlinkRunner and have it work.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v7.6.3#76005)
>

Re: [INFO] Gradle build is flaky on Jenkins

[Kenneth Knowles]

Separately, we have many tests that seem to have been passing by luck and
Gradle exposes some of their weaknesses by its increased parallelism. They
are filed under the "flake" and later the "sickbay" tags. I've put together
a saved search: https://issues.apache.org/jira/issues/?filter=12343195

These may be good starter tasks that don't require deep Beam knowledge.

Kenn

On Fri, Feb 9, 2018 at 7:04 AM, Kenneth Knowles  wrote:

> I have been hitting these issues and investigating them.
>
>  - HCatalog pentaho dependency: https://issues.apache.org/
> jira/browse/BEAM-3621
>  - We rerun way way more than we need to: https://issues.apache.org/
> jira/browse/BEAM-3253 but dependency stuff occurs during config, not
> tasks, I think
>
> The biggest source of flakes with Maven is dependency download. We only
> got any green with maven once we saved .m2 across builds. It seems like
> gradle is in that state, and is not caching them. I'm not sure why - by
> default it should cache to $HOME/.gradle. I think some versions of the
> Jenkins plugin caches to $WORKSPACE/.gradle but that should also work just
> fine unless we are wiping out the workspace, which we shouldn't do as we
> have isolated the build to the $WORKSPACE/src directory. When I look in a
> workspace I don't see a cache anyhow.
>
> There's an issue with the Jenkins workers so I can't ssh to them at the
> moment to investigate. I am hoping that https://github.com/apache/
> beam/pull/4644 will also make it easier to see what is happening.
>
> Kenn
>
> On Fri, Feb 9, 2018 at 5:29 AM, Jean-Baptiste Onofré 
> wrote:
>
>> Hi Romain,
>>
>> that's actually part of my investigations ;)
>>
>> Regards
>> JB
>>
>> On 02/09/2018 02:26 PM, Romain Manni-Bucau wrote:
>> > did you check the concurrency? For me it is plain wrong since it is
>> hardcoded in
>> > the build file and doesnt let me or the tool customize it. This means
>> it just
>> > blocks my computer in general and makes some timeout related tests fail
>> easily.
>> > (this kind of config should always be customized on the CI and never
>> hardcoded IMHO)
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau  |  Blog
>> >  | Old Blog
>> >  | Github <
>> https://github.com/rmannibucau> |
>> > LinkedIn  | Book
>> > > high-performance>
>> >
>> > 2018-02-09 12:00 GMT+01:00 Aljoscha Krettek > > >:
>> >
>> > Yes, I was just about to write about this as well. In my recent PRs
>> this
>> > always failed for different reasons.
>> >
>> > Thanks for looking into this!
>> >
>> > > On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré > > > wrote:
>> > >
>> > > Hi guys,
>> > >
>> > > I noticed that the Gradle build on Jenkins is flaky: it almost
>> always
>> > fails for
>> > > different reason (I can't download pentaho artifact sometime,
>> interrupted
>> > other
>> > > times).
>> > >
>> > > Jenkins is doing:
>> > >
>> > > Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
>> > >
>> > > I gonna investigate how to improve the situation.
>> > >
>> > > Regards
>> > > JB
>> > > --
>> > > Jean-Baptiste Onofré
>> > > jbono...@apache.org 
>> > > http://blog.nanthrax.net
>> > > Talend - http://www.talend.com
>> >
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>

Re: [INFO] Gradle build is flaky on Jenkins

[Kenneth Knowles]

I have been hitting these issues and investigating them.

 - HCatalog pentaho dependency:
https://issues.apache.org/jira/browse/BEAM-3621
 - We rerun way way more than we need to:
https://issues.apache.org/jira/browse/BEAM-3253 but dependency stuff occurs
during config, not tasks, I think

The biggest source of flakes with Maven is dependency download. We only got
any green with maven once we saved .m2 across builds. It seems like gradle
is in that state, and is not caching them. I'm not sure why - by default it
should cache to $HOME/.gradle. I think some versions of the Jenkins plugin
caches to $WORKSPACE/.gradle but that should also work just fine unless we
are wiping out the workspace, which we shouldn't do as we have isolated the
build to the $WORKSPACE/src directory. When I look in a workspace I don't
see a cache anyhow.

There's an issue with the Jenkins workers so I can't ssh to them at the
moment to investigate. I am hoping that
https://github.com/apache/beam/pull/4644 will also make it easier to see
what is happening.

Kenn

On Fri, Feb 9, 2018 at 5:29 AM, Jean-Baptiste Onofré 
wrote:

> Hi Romain,
>
> that's actually part of my investigations ;)
>
> Regards
> JB
>
> On 02/09/2018 02:26 PM, Romain Manni-Bucau wrote:
> > did you check the concurrency? For me it is plain wrong since it is
> hardcoded in
> > the build file and doesnt let me or the tool customize it. This means it
> just
> > blocks my computer in general and makes some timeout related tests fail
> easily.
> > (this kind of config should always be customized on the CI and never
> hardcoded IMHO)
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau  |  Blog
> >  | Old Blog
> >  | Github  rmannibucau> |
> > LinkedIn  | Book
> >  ee-8-high-performance>
> >
> > 2018-02-09 12:00 GMT+01:00 Aljoscha Krettek  > >:
> >
> > Yes, I was just about to write about this as well. In my recent PRs
> this
> > always failed for different reasons.
> >
> > Thanks for looking into this!
> >
> > > On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré  > > wrote:
> > >
> > > Hi guys,
> > >
> > > I noticed that the Gradle build on Jenkins is flaky: it almost
> always
> > fails for
> > > different reason (I can't download pentaho artifact sometime,
> interrupted
> > other
> > > times).
> > >
> > > Jenkins is doing:
> > >
> > > Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
> > >
> > > I gonna investigate how to improve the situation.
> > >
> > > Regards
> > > JB
> > > --
> > > Jean-Baptiste Onofré
> > > jbono...@apache.org 
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [INFO] Gradle build is flaky on Jenkins

[Jean-Baptiste Onofré]

Hi Romain,

that's actually part of my investigations ;)

Regards
JB

On 02/09/2018 02:26 PM, Romain Manni-Bucau wrote:
> did you check the concurrency? For me it is plain wrong since it is hardcoded 
> in
> the build file and doesnt let me or the tool customize it. This means it just
> blocks my computer in general and makes some timeout related tests fail 
> easily.
> (this kind of config should always be customized on the CI and never 
> hardcoded IMHO)
> 
> 
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github  |
> LinkedIn  | Book
> 
> 
> 2018-02-09 12:00 GMT+01:00 Aljoscha Krettek  >:
> 
> Yes, I was just about to write about this as well. In my recent PRs this
> always failed for different reasons.
> 
> Thanks for looking into this!
> 
> > On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré  > wrote:
> >
> > Hi guys,
> >
> > I noticed that the Gradle build on Jenkins is flaky: it almost always
> fails for
> > different reason (I can't download pentaho artifact sometime, 
> interrupted
> other
> > times).
> >
> > Jenkins is doing:
> >
> > Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
> >
> > I gonna investigate how to improve the situation.
> >
> > Regards
> > JB
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org 
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [INFO] Gradle build is flaky on Jenkins

[Romain Manni-Bucau]

did you check the concurrency? For me it is plain wrong since it is
hardcoded in the build file and doesnt let me or the tool customize it.
This means it just blocks my computer in general and makes some timeout
related tests fail easily. (this kind of config should always be customized
on the CI and never hardcoded IMHO)


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-02-09 12:00 GMT+01:00 Aljoscha Krettek :

> Yes, I was just about to write about this as well. In my recent PRs this
> always failed for different reasons.
>
> Thanks for looking into this!
>
> > On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré  wrote:
> >
> > Hi guys,
> >
> > I noticed that the Gradle build on Jenkins is flaky: it almost always
> fails for
> > different reason (I can't download pentaho artifact sometime,
> interrupted other
> > times).
> >
> > Jenkins is doing:
> >
> > Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
> >
> > I gonna investigate how to improve the situation.
> >
> > Regards
> > JB
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
>

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #24

[Apache Jenkins Server]

See 


Changes:

[robertwb] [BEAM-3625] Enable DoFn params in Map, Filter, etc.

[apilloud] [BEAM-410] Sort PriorityQueue with explicit 
comparator

[robertwb] Disable verbose typecoder warnings.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8c5ebe4a4d158fa8c46640a0401376216a1d11c2 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8c5ebe4a4d158fa8c46640a0401376216a1d11c2
Commit message: "Merge pull request #4641 [BEAM-3644] Disable verbose typecoder 
warnings."
 > git rev-list 6792846069763770bbfd92a5eb61e64968522967 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostRelease_NightlySnapshot] $ /bin/bash -xe 
/tmp/jenkins3588425326827574657.sh
+ cd src/release
+ groovy quickstart-java-direct.groovy
/tmp/jenkins3588425326827574657.sh: line 2: groovy: command not found
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user apill...@google.com

Re: [INFO] Gradle build is flaky on Jenkins

[Aljoscha Krettek]

Yes, I was just about to write about this as well. In my recent PRs this always 
failed for different reasons.

Thanks for looking into this!

> On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré  wrote:
> 
> Hi guys,
> 
> I noticed that the Gradle build on Jenkins is flaky: it almost always fails 
> for
> different reason (I can't download pentaho artifact sometime, interrupted 
> other
> times).
> 
> Jenkins is doing:
> 
> Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
> 
> I gonna investigate how to improve the situation.
> 
> Regards
> JB
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com

[INFO] Gradle build is flaky on Jenkins

[Jean-Baptiste Onofré]

Hi guys,

I noticed that the Gradle build on Jenkins is flaky: it almost always fails for
different reason (I can't download pentaho artifact sometime, interrupted other
times).

Jenkins is doing:

Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit

I gonna investigate how to improve the situation.

Regards
JB
-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

[CANCEL][VOTE] Release 2.3.0, release candidate #2

[Jean-Baptiste Onofré]

Hi guys,

due to BEAM-3519 [1], I cancel RC2.

We have to shade netty version in the IOs to avoid issue with netty version in
runners (Spark in that case).

We gonna work on a fix (I should be pretty fast), and I will cut a RC3 when
cherry-pick on release-2.3.0 branch.

Regards
JB

[1] https://issues.apache.org/jira/browse/BEAM-3519

On 02/08/2018 06:37 AM, Jean-Baptiste Onofré wrote:
> Hi everyone,
> 
> Please review and vote on the release candidate #2 for the version 2.3.0, as
> follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org [2],
> which is signed with the key with fingerprint C8282E76 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.3.0-RC2" [5],
> * website pull request listing the release and publishing the API reference
> manual [6].
> * Java artifacts were built with Maven 3.3.9 and Oracle JDK 1.8.0_111.
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2].
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval,
> with at least 3 PMC affirmative votes.
> 
> Thanks,
> JB
> 
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608
> [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1027/
> [5] https://github.com/apache/beam/tree/v2.3.0-RC2
> [6] https://github.com/apache/beam-site/pull/381
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.3.0, release candidate #2

[Ismaël Mejía]

+1 for option 1 (fix and create a new +RC3). This not only breaks
examples, this means that users of the spark runner won't have any
guarantee about the fact that the Google IOs work or not because of
possible missing methods or conflicts between netty versions.

On Fri, Feb 9, 2018 at 7:31 AM, Romain Manni-Bucau
 wrote:
> since it breaks only examples not sure it does worth yet another reroll
> (which means already a 2 weeks delay on the plan). Users will be affected
> the same anyway - and in an expected way until beam handles classloaders per
> transform. A note in the side is fine probably.
>
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
> 2018-02-09 7:23 GMT+01:00 Chamikara Jayalath :
>>
>>
>>
>> On Thu, Feb 8, 2018 at 10:18 PM Jean-Baptiste Onofré 
>> wrote:
>>>
>>> It means a RC3 then.
>>>
>>> Basically, we have two options:
>>>
>>> 1. I cancel RC2, to include PR 4645 and cut a RC3. It can be done super
>>> fast
>>> (today).
>>
>>
>> +1 for option 1 since IMO we should not release with quickstart broken for
>> Spark.
>>
>>>
>>> 2. We continue RC2 vote and we add a note about shading (as I did for the
>>> TextIO
>>> issue with Flink runner).
>>>
>>> I'm more in favor of 1 as the fix is already there and cutting a release
>>> is
>>> super fast for me.
>>>
>>> Thoughts ?
>>>
>>> Regards
>>> JB
>>>
>>> On 02/09/2018 07:06 AM, Chamikara Jayalath wrote:
>>> > +1 for continuing the release after the immediate fix
>>> > (https://github.com/apache/beam/pull/4645). I don't
>>> > think https://issues.apache.org/jira/browse/BEAM-3519 is due to a
>>> > recent update
>>> > to google-cloud-platform module so the issue likely existed in some
>>> > form in
>>> > previous releases as well.
>>> >
>>> > - Cham
>>> >
>>> > On Thu, Feb 8, 2018 at 9:47 PM Romain Manni-Bucau
>>> > >> > > wrote:
>>> >
>>> > IMHO it is not a blocker but an incompatibility between spark and
>>> > some IO
>>> > stack. Trivial workaround is to shade the io before importing it in
>>> > its
>>> > project. Amternative is to wrap IO in custom classloaders.
>>> >
>>> > Didnt check for this one but it is a common beam issue to have
>>> > conflicts
>>> > between runners/io or even 2 ios so it shouldnt block a release by
>>> > itself
>>> > until beam aims to solve properly conflicts - which means without
>>> > shading
>>> > which breaks the io ecosystem on the user side.
>>> >
>>> > Just my 2cts
>>> >
>>> > Le 9 févr. 2018 06:07, "Jean-Baptiste Onofré" >> > > a écrit :
>>> >
>>> > Is it specific to this release ? I think it was like this
>>> > before no ?
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 02/09/2018 12:48 AM, Kenneth Knowles wrote:
>>> > > Since root cause is
>>> > https://issues.apache.org/jira/browse/BEAM-3519 I
>>> > marked it
>>> > > a blocker so we can discuss fixes or workarounds there.
>>> > >
>>> > > On Thu, Feb 8, 2018 at 1:24 PM, Lukasz Cwik >> > 
>>> > > >> wrote:
>>> > >
>>> > > I validated several of the quickstarts and updated the
>>> > spreadsheet and
>>> > > currently am voting -1 for this release due to Spark
>>> > runner
>>> > failing. Filed
>>> > > https://issues.apache.org/jira/browse/BEAM-3668
>>> > >  with
>>> > the full
>>> > details.
>>> > >
>>> > >
>>> > > On Thu, Feb 8, 2018 at 10:32 AM, Valentyn Tymofieiev
>>> > mailto:valen...@google.com>
>>> > > >> > >> wrote:
>>> > >
>>> > > Yes (thanks
>>> > Kenn!): https://s.apache.org/beam-2.3.0-release-validation
>>> > > 
>>> > >
>>> > > On Thu, Feb 8, 2018 at 10:14 AM, Eugene Kirpichov
>>> > mailto:kirpic...@google.com>
>>> > > >> > >>
>>> > wrote:
>>> > >
>>> > > Do we have a release validation spreadsheet for
>>> > this one?
>>> > >
>>> > > On Thu, Feb 8, 2018 at 9:30 AM Ahmet Altay
>>> > mailto:al...@google.com>
>>> > > >> > >> wrote:
>>> > >
>>> > > +1
>>> > >
>>> > > I verified python quick start, mobile gaming
>>> > examples,
>>> > streaming
>>> > > on Direct and Dataflow runners. Thank you JB!
>>> > >
>>> > > On Thu, Feb 8, 2018 at 2:27 AM, Romain
>>> > Manni-Bucau
>>> > >