Re: [VOTE] Release 1.6.0, release candidate #4

[Yaz Sh]

Thanks for the fix!

/Yaz

On Thu, Aug 9, 2018 at 9:10 AM Till Rohrmann  wrote:

> Thanks for reporting this problem Yaz. I just pushed a commit which should
> update the links accordingly once the Flink documentation gets rebuilt
> (over night). Tomorrow it should be fixed.
>
> Cheers,
> Till
>
> On Thu, Aug 9, 2018 at 2:53 PM Yaz Sh  wrote:
>
> > Great Job on Release 1.6!
> >
> > I just checked it out and still I can see v.1.6-SNAPSHOT on the title of
> > https://ci.apache.org/projects/flink/flink-docs-release-1.6/ <
> > https://ci.apache.org/projects/flink/flink-docs-release-1.6/>
> >
> > and when I click on any options, it redirects me to master docs
> > 1.7-SNAPSHOT.
> >
> > I opened this ticket https://issues.apache.org/jira/browse/FLINK-10112 <
> > https://issues.apache.org/jira/browse/FLINK-10112>
> >
> > Also I don’t see v1.6 on “Pick Docs Version" drop down
> >
> > Cheers,
> > Yaz
> >
> > > On Aug 8, 2018, at 3:24 PM, Timo Walther  wrote:
> > >
> > > +1
> > >
> > > - successfully run `mvn clean verify` locally
> > > - successfully run end-to-end tests locally (except for SQL Client
> > end-to-end test)
> > >
> > > Found a bug in the class loading of SQL JAR files. This is not a
> blocker
> > but a bug that we should fix soon. As an easy workaround user should not
> > use different Kafka versions as SQL Client dependencies.
> > >
> > > Regards,
> > > Timo
> > >
> > > Am 08.08.18 um 18:10 schrieb Dawid Wysakowicz:
> > >> +1
> > >>
> > >> - verified compilation, tests
> > >> - verified checksum and gpg files
> > >> - verified sbt templates (g8, quickstart) - run assemblies on local
> > cluster
> > >>
> > >> - I could not execute the nightly-tests.sh though. The tests that were
> > >> failing most often are:
> > >> - test_streaming_file_sink.sh
> > >> - test_streaming_elasticsearch.sh
> > >>
> > >> Those are connectors though and it might be only tests flakiness so I
> > >> think it should not block the release.
> > >>
> > >> On 08/08/18 16:36, Chesnay Schepler wrote:
> > >>> I did not use the tools/list_deps.py script as I wasn't aware that it
> > >>> existed.
> > >>>
> > >>> Even if I were I wouldn't have used it and in fact would advocate for
> > >>> removing it.
> > >>> It manually parses and constructs dependency information which is
> > >>> utterly unnecessary as maven already provides this functionality,
> with
> > >>> the added bonus of also accounting for dependencyManagement and
> > >>> transitive dependencies which we obviously have to take into account.
> > >>>
> > >>> I used this one-liner instead:
> > >>> |mvn dependency:list | ||grep| |":.*:.*:.*"| || ||grep| |-||v| |-e
> > >>> ||"Finished at"| |-e ||"Some problems"| || ||cut| |-d] -f2- | ||sed|
> > >>> |'s/:[a-z]*$//g'| || ||sort| |-u
> > >>>
> > >>> |which I have documented here:
> > >>> https://cwiki.apache.org/confluence/display/FLINK/Dependencies
> > >>>
> > >>> On 08.08.2018 15:06, Aljoscha Krettek wrote:
> > >>>> +1
> > >>>>
> > >>>> - verified checksum and gpg files
> > >>>> - verified LICENSE and NOTICE: NOTICE didn't change from 1.5,
> LICENSE
> > >>>> had one unnecessary part removed
> > >>>>
> > >>>> Side comment: I'm not sure whether the "Verify that the LICENSE and
> > >>>> NOTICE file is correct for the binary and source releases" part is
> > >>>> valid anymore because we only have one LICENSE and NOTICE file. also
> > >>>> "The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer
> > >>>> to the binary distribution and mention all of Flink's Maven
> > >>>> dependencies as well" can be dropped because we don't have them
> > anymore.
> > >>>>
> > >>>> I came to the same conclusion on dependencies. I used
> > >>>> tools/list_deps.py and diff'ed the output for 1.5 and 1.6, that's
> > >>>> probably what Chesnay also did ... :-)
> > >>>>
> > >>>>> On 8. Aug 2018, at 14:43, Chesnay Sc

Re: [VOTE] Release 1.6.0, release candidate #4

[Yaz Sh]

Great Job on Release 1.6!

I just checked it out and still I can see v.1.6-SNAPSHOT on the title of 
https://ci.apache.org/projects/flink/flink-docs-release-1.6/ 


and when I click on any options, it redirects me to master docs 1.7-SNAPSHOT.

I opened this ticket https://issues.apache.org/jira/browse/FLINK-10112 


Also I don’t see v1.6 on “Pick Docs Version" drop down

Cheers,
Yaz

> On Aug 8, 2018, at 3:24 PM, Timo Walther  wrote:
> 
> +1
> 
> - successfully run `mvn clean verify` locally
> - successfully run end-to-end tests locally (except for SQL Client end-to-end 
> test)
> 
> Found a bug in the class loading of SQL JAR files. This is not a blocker but 
> a bug that we should fix soon. As an easy workaround user should not use 
> different Kafka versions as SQL Client dependencies.
> 
> Regards,
> Timo
> 
> Am 08.08.18 um 18:10 schrieb Dawid Wysakowicz:
>> +1
>> 
>> - verified compilation, tests
>> - verified checksum and gpg files
>> - verified sbt templates (g8, quickstart) - run assemblies on local cluster
>> 
>> - I could not execute the nightly-tests.sh though. The tests that were
>> failing most often are:
>> - test_streaming_file_sink.sh
>> - test_streaming_elasticsearch.sh
>> 
>> Those are connectors though and it might be only tests flakiness so I
>> think it should not block the release.
>> 
>> On 08/08/18 16:36, Chesnay Schepler wrote:
>>> I did not use the tools/list_deps.py script as I wasn't aware that it
>>> existed.
>>> 
>>> Even if I were I wouldn't have used it and in fact would advocate for
>>> removing it.
>>> It manually parses and constructs dependency information which is
>>> utterly unnecessary as maven already provides this functionality, with
>>> the added bonus of also accounting for dependencyManagement and
>>> transitive dependencies which we obviously have to take into account.
>>> 
>>> I used this one-liner instead:
>>> |mvn dependency:list | ||grep| |":.*:.*:.*"| || ||grep| |-||v| |-e
>>> ||"Finished at"| |-e ||"Some problems"| || ||cut| |-d] -f2- | ||sed|
>>> |'s/:[a-z]*$//g'| || ||sort| |-u
>>> 
>>> |which I have documented here:
>>> https://cwiki.apache.org/confluence/display/FLINK/Dependencies
>>> 
>>> On 08.08.2018 15:06, Aljoscha Krettek wrote:
 +1
 
 - verified checksum and gpg files
 - verified LICENSE and NOTICE: NOTICE didn't change from 1.5, LICENSE
 had one unnecessary part removed
 
 Side comment: I'm not sure whether the "Verify that the LICENSE and
 NOTICE file is correct for the binary and source releases" part is
 valid anymore because we only have one LICENSE and NOTICE file. also
 "The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer
 to the binary distribution and mention all of Flink's Maven
 dependencies as well" can be dropped because we don't have them anymore.
 
 I came to the same conclusion on dependencies. I used
 tools/list_deps.py and diff'ed the output for 1.5 and 1.6, that's
 probably what Chesnay also did ... :-)
 
> On 8. Aug 2018, at 14:43, Chesnay Schepler  wrote:
> 
> +1
> 
> - verified source release contains no binaries
> - verified correct versions in source release
> - verified compilation, tests and E2E-tests pass (on travis)
> - verified checksum and gpg files
> 
> New dependencies (excluding dependencies where we simply depend on a
> different version now):
> Apache licensed:
> io.confluent:common-utils:jar:3.3.1
> io.confluent:kafka-schema-registry-client:jar:3.3.1
> io.prometheus:simpleclient_pushgateway:jar:0.3.0
> various Apache Nifi dependencies
> various Apache Parquet dependencies
> various ElasticSearch dependencies
> CDDL:
> javax.ws.rs:javax.ws.rs-api:jar:2.1
> Bouncycastle (MIT-like):
> org.bouncycastle:bcpkix-jdk15on:jar:1.59
> org.bouncycastle:bcprov-jdk15on:jar:1.59
> MIT:
> org.projectlombok:lombok:jar:1.16.20
> 
> On 08.08.2018 13:28, Till Rohrmann wrote:
>> Thanks for reporting these problems Chesnay. The usage string in
>> `standalone-job.sh` is out dated and should be updated. The same
>> applies to
>> the typo.
>> 
>> When calling `standalone-job.sh start --job-classname foobar.Job`
>> please
>> make sure that the user code jar is contained in the classpath (e.g.
>> putting the jar in the lib directory). Documenting this behaviour
>> is part
>> of the pending issue FLINK-10001.
>> 
>> We should fix all of these issues. They are, however, no release
>> blockers.
>> 
>> Cheers,
>> Till
>> 
>> On Wed, Aug 8, 2018 at 11:31 AM Chesnay Schepler
>>  wrote:
>> 
>>> I found some issues with the standalone-job.sh script.

SSL Error on Flink-Kafka

[Yaz Sh]

Hi Devs,

I am trying to connect to Kaka via SSL and when I run the code on cluster I get 
bellow SSL error. 
(The code is running on IntelliJ without any issue )

Caused by: org.apache.kafka.common.errors.SerializationException: Error 
deserializing Avro message for id 161
Caused by: javax.net.ssl.SSLHandshakeException: 
sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target
...
Caused by: sun.security.validator.ValidatorException: PKIX path building 
failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to 
find valid certification path to requested target

I had this on my code:
System.setProperty(
  "javax.net.ssl.trustStore", “/Path/to/TrusStore")
System.setProperty(
  "javax.net.ssl.trustStorePassword”, “TrustStore Password")

System.setProperty(
  "javax.net.ssl.keyStore”, “path/to/keyStore")
System.setProperty(
  "javax.net.ssl.keyStorePassword”, “keyStore Password")

also I added this on flink-conf.yml

javax.net.ssl.trustStore: “path to trustStore"
javax.net.ssl.keyStore: “pathToKeyStore"
javax.net.ssl.keyStorePassword: "KeyStorepassword"
javax.net.ssl.trustStorePassword: “TrusStorepassword”

I also ran the code via CLI by passing Java params

./bin/flink run 
/Users/hnw511/GitHub/DataPipeLine/Enki/harmony-event-pipeline/core/target/ReactiveDecline.jar
 -Djavax.net.ssl.trustStore=/path/to/truststore.jks

For all of them I get the 
sun.security.provider.certpath.SunCertPathBuilderException error. 
I am not sure what I am missing and would appreciate any help.

Cheers,
Yaz

Re: [RESULT][VOTE] Release 1.5.2, release candidate #2

[Yaz Sh]

Great job and Thanks Chesnay
Cheers,
Yaz

On Tue, Jul 31, 2018 at 2:31 PM Thomas Weise  wrote:

> Thanks to Chesnay for running the release and getting it completed quickly!
>
>
> On Tue, Jul 31, 2018 at 2:02 AM Fabian Hueske  wrote:
>
> > Thanks Chesnay!
> >
> > 2018-07-31 10:59 GMT+02:00 vino yang :
> >
> > > Thanks for releasing Flink 1.5.2, Chesnay!
> > >
> > > Thanks.
> > > Vino.
> > >
> > > 2018-07-31 16:49 GMT+08:00 Till Rohrmann :
> > >
> > > > Thanks Chesnay for being our release manager and thanks to the
> > community
> > > > for all the work!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Tue, Jul 31, 2018 at 10:17 AM Timo Walther 
> > > wrote:
> > > >
> > > > > Thanks for managing the release process Chesnay!
> > > > >
> > > > > Timo
> > > > >
> > > > > Am 31.07.18 um 10:05 schrieb Chesnay Schepler:
> > > > > > I'm happy to announce that we have approved this release.
> > > > > >
> > > > > > There are 5 approving votes, 3 of which are binding:
> > > > > >
> > > > > > * Yaz (non-binding)
> > > > > > * Thomas (non-binding)
> > > > > > * Till (binding)
> > > > > > * Timo (binding)
> > > > > > * Chesnay (binding)
> > > > > >
> > > > > > There is 1 disapproving vote:
> > > > > >
> > > > > > * Piotr (non-binding)
> > > > > >
> > > > > > Thanks everyone!
> > > > > >
> > > > > >
> > > > > > On 26.07.2018 10:39, Chesnay Schepler wrote:
> > > > > >> Hi everyone,
> > > > > >> Please review and vote on the release candidate #2 for the
> version
> > > > > >> 1.5.2, as follows:
> > > > > >> [ ] +1, Approve the release
> > > > > >> [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > >>
> > > > > >>
> > > > > >> The complete staging area is available for your review, which
> > > > includes:
> > > > > >> * JIRA release notes [1],
> > > > > >> * the official Apache source release and binary convenience
> > releases
> > > > > >> to be deployed to dist.apache.org [2], which are signed with
> the
> > > key
> > > > > >> with fingerprint 11D464BA [3],
> > > > > >> * all artifacts to be deployed to the Maven Central Repository
> > [4],
> > > > > >> * source code tag "release-1.5.1-rc2" [5],
> > > > > >> * website pull request listing the new release and adding
> > > > > >> announcement blog post [6].
> > > > > >>
> > > > > >> The vote will be open for at least 72 hours. It is adopted by
> > > > > >> majority approval, with at least 3 PMC affirmative votes.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Chesnay
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > projectId=12315522=12343588
> > > > > >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.5.2/
> > > > > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > >> [4]
> > > > > >> https://repository.apache.org/content/repositories/
> > > > orgapacheflink-1174
> > > > > >> [5]
> > > > > >>
> > > > > https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=
> > > > refs/tags/release-1.5.2-rc2
> > > > > >> [6] https://github.com/apache/flink-web/pull/114
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release 1.5.2, release candidate #2

[Yaz Sh]

+1

- Checked “signatures" for all packages
- Checked “checksum" for all packages
- Started cluster with no exceptions on log files
- Stoped cluster with no exceptions on log file
- Added more taskManager by running “taskmanager.sh start”
- Added more slots via “flink-conf.yml"
- Checked for all examples to be available on the packages
- Tested several batch and stream examples via WebUI
- Tested several batch and stream examples via CLI
- Tested Kafka010Example
- Checked the website PR
 

Cheers,
Yaz

> On Jul 26, 2018, at 9:31 AM, Timo Walther  wrote:
> 
> I agree with Chesnay. Only regressions in the release candidate should cancel 
> minor releases.
> 
> Timo
> 
> 
> Am 26.07.18 um 15:02 schrieb Chesnay Schepler:
>> Since the regression already existed in 1.5.0 I will not cancel the vote,
>> as there's no benefit to canceling the bugfix release.
>> 
>> If a fix is found within the voting period we should consider it so we
>> don't have to create a separate release shortly afterwards for this issue.
>> 
>> On 26.07.2018 14:29, Piotr Nowojski wrote:
>>> Voting -1
>>> 
>>> Flip6 in 1.5.0 seems like introduced regression in resource management in 
>>> batch: https://issues.apache.org/jira/browse/FLINK-9969 
>>> 
>>> 
>>> It’s not a super new bug, yet still I would block the release until we 
>>> either fix it or confirm that’s a some misconfiguration.
>>> 
>>> Piotrek
>>> 
 On 26 Jul 2018, at 10:39, Chesnay Schepler  wrote:
 
 Hi everyone,
 Please review and vote on the release candidate #2 for the version 1.5.2, 
 as follows:
 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific comments)
 
 
 The complete staging area is available for your review, which includes:
 * JIRA release notes [1],
 * the official Apache source release and binary convenience releases to be 
 deployed to dist.apache.org [2], which are signed with the key with 
 fingerprint 11D464BA [3],
 * all artifacts to be deployed to the Maven Central Repository [4],
 * source code tag "release-1.5.1-rc2" [5],
 * website pull request listing the new release and adding announcement 
 blog post [6].
 
 The vote will be open for at least 72 hours. It is adopted by majority 
 approval, with at least 3 PMC affirmative votes.
 
 Thanks,
 Chesnay
 
 [1] 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12343588
 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.5.2/
 [3] https://dist.apache.org/repos/dist/release/flink/KEYS
 [4] https://repository.apache.org/content/repositories/orgapacheflink-1174
 [5] 
 https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=refs/tags/release-1.5.2-rc2
 [6] https://github.com/apache/flink-web/pull/114
 
 
 
 
 
>>> 
> 

Re: [ANNOUNCE] Apache Flink 1.5.1 released

[Yaz Sh]

Excellent!
Thanks to Chesnay, great work!

Yaz

> On Jul 13, 2018, at 8:37 AM, Dawid Wysakowicz  wrote:
> 
> Good job everyone and Chesnay for being the release manager!
> 
> 
> On 13/07/18 14:34, Hequn Cheng wrote:
>> Cool, thanks to Chesnay!
>> 
>> Best, Hequn
>> 
>> On Fri, Jul 13, 2018 at 8:25 PM, vino yang > > wrote:
>> 
>>Thanks Chesnay, great job!
>> 
>>Thanks,
>>Vino
>> 
>>2018-07-13 20:20 GMT+08:00 Till Rohrmann >>:
>> 
>>Great to hear. Big thank you to the community for the hard
>>work and to Chesnay for being our release manager.
>> 
>>Cheers,
>>Till
>> 
>>On Fri, Jul 13, 2018 at 12:05 PM Chesnay Schepler
>>mailto:ches...@apache.org>> wrote:
>> 
>>The Apache Flink community is very happy to announce the
>>release of Apache Flink 1.5.1, which is the first bugfix
>>release for the Apache Flink 1.5 series.
>> 
>>Apache Flink® is an open-source stream processing
>>framework for distributed, high-performing,
>>always-available, and accurate data streaming applications.
>> 
>>The release is available for download at:
>>https://flink.apache.org/downloads.html
>> 
>> 
>>Please check out the release blog post for an overview of
>>the improvements for this bugfix release:
>>http://flink.apache.org/news/2018/07/12/release-1.5.1.html
>>
>> 
>>The full release notes are available in Jira:
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12343053
>>
>> 
>> 
>>We would like to thank all contributors of the Apache
>>Flink community who made this release possible!
>> 
>>PS: A regression was identified where the CLI job
>>submission fails when SSL is enabled. (FLINK-9842)
>> 
>>Regards,
>>Chesnay
>> 
>> 
>> 
> 
> 

Re: [VOTE] Release 1.5.1, release candidate #3

[Yaz Sh]

Hi Till,

Thanks for reply!

For version 1.4.x when Parallelism > Available task Slots has been selected, 
Flink throw bellow error immediately as you said

NoResourceAvailableException: Not enough free slots available to run the job
 
but for version 1.5.x there are two different behaviors: Sometimes job ran 
successfully and sometimes it throw error after 5 minutes with this message:

org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: 
Could not allocate all requires slots within timeout of 30 ms. Slots 
required: 5, slots allocated: 2


I think it might be a bug and created this ticket to keep track of it
https://issues.apache.org/jira/browse/FLINK-9848 
<https://issues.apache.org/jira/browse/FLINK-9848>

Please let me know if you think otherwise.


Cheers,
Yazdan 

> On Jul 12, 2018, at 3:54 AM, Till Rohrmann  wrote:
> 
> Hi Yazdan,
> 
> that is the expected behaviour. If the system cannot allocate enough slots
> it will fail with a NoResourceAvailableException after
> the slot.request.timeout has been exceeded.
> 
> Cheers,
> Till
> 
> On Wed, Jul 11, 2018 at 7:44 PM Yaz Sh  <mailto:yazda...@gmail.com>> wrote:
> 
>> +1
>> 
>> - Verified the signatures for all binary artifacts
>> - Verifies Checksum for all binary packages
>> - Ran local cluster with no error on logs and empty *.out
>> - Stop local cluster  with no error on logs
>> - Ran multiple batch and streaming example via WebUI
>> - Rab multiple batch and streaming examples via CLI
>> - Increase number of task managers and ran examples with Parallelism > 1
>> - Ran WebUI on multiple browsers
>> - Check Example folder for all binary packages
>> 
>> Just an observation:
>> When ran a job with Parallelism > Available task slots intermediately job
>> stay in “Running" status for a very long time and neither finish nor throw
>> any errors.
>> 
>> Please check if someone else can reproduce it.
>> 
>> Cheers,
>> Yazdan
>> 
>> 
>>> On Jul 11, 2018, at 11:21 AM, Till Rohrmann 
>> wrote:
>>> 
>>> +1 (binding)
>>> 
>>> - Verified the signatures of all binary artifacts
>>> - Verified that no new dependencies were added for which the LICENSE and
>> NOTICE files need to be adapted.
>>> - Build 1.5.1 from the source artifact
>>> - Run flink-end-to-end tests for 12 hours for the 1.5.1 Hadoop 2.7
>> binary artifact
>>> - Run Jepsen tests for 12 hours for the 1.5.1 Hadoop 2.8 binary artifact
>>> 
>>> Cheers,
>>> Till
>>> 
>>> 
>>> On Wed, Jul 11, 2018 at 9:49 AM Chesnay Schepler > <mailto:ches...@apache.org <mailto:ches...@apache.org>>> wrote:
>>> Correction on my part, it does affect all packages.
>>> 
>>> I've also found the cause. To speed up the process I only built modules
>>> that flink-dist depends on (see FLINK-9768). However flink-dist depends
>>> on neither flink-examples-batch nor flink-examples-streaming, yet
>>> happily accesses their target directory. The existing build process only
>>> worked since _by chance_ these 2 modules are built before flink-dist
>>> when doing a complete build.
>>> 
>>> I will rebuild the binaries (I don't think we have to cancel the RC for
>>> this) and open a JIRA to fix the dependencies.
>>> 
>>> On 11.07.2018 09:27, Chesnay Schepler wrote:
>>>> oh, the packages that include hadoop are really missing it...
>>>> 
>>>> On 11.07.2018 09:25, Chesnay Schepler wrote:
>>>>> @Yaz which binary package did you check? I looked into the
>>>>> hadoop-free package and the folder is there.
>>>>> 
>>>>> Did you maybe encounter an error when extracting the package?
>>>>> 
>>>>> On 11.07.2018 05:44, Yaz Sh wrote:
>>>>>> -1
>>>>>> 
>>>>>> ./examples/streaming folder is missing in binary packages
>>>>>> 
>>>>>> 
>>>>>> Cheers,
>>>>>> Yazdan
>>>>>> 
>>>>>>> On Jul 10, 2018, at 9:57 PM, vino yang >>>>>> <mailto:yanghua1...@gmail.com>
>> <mailto:yanghua1...@gmail.com <mailto:yanghua1...@gmail.com>>> wrote:
>>>>>>> 
>>>>>>> +1
>>>>>>> reviewed [1], [4] and [6]
>>>>>>> 
>>>>>>> 2018-07-11 3:10 GMT+08:00 Chesnay Schepler >>>>>> <mailto:ches...@apache.org>
&

Re: [VOTE] Release 1.5.1, release candidate #3

[Yaz Sh]

+1

- Verified the signatures for all binary artifacts
- Verifies Checksum for all binary packages
- Ran local cluster with no error on logs and empty *.out
- Stop local cluster  with no error on logs
- Ran multiple batch and streaming example via WebUI
- Rab multiple batch and streaming examples via CLI
- Increase number of task managers and ran examples with Parallelism > 1
- Ran WebUI on multiple browsers
- Check Example folder for all binary packages

Just an observation:
When ran a job with Parallelism > Available task slots intermediately job stay 
in “Running" status for a very long time and neither finish nor throw any 
errors.

Please check if someone else can reproduce it.

Cheers,
Yazdan


> On Jul 11, 2018, at 11:21 AM, Till Rohrmann  wrote:
> 
> +1 (binding)
> 
> - Verified the signatures of all binary artifacts
> - Verified that no new dependencies were added for which the LICENSE and 
> NOTICE files need to be adapted.
> - Build 1.5.1 from the source artifact
> - Run flink-end-to-end tests for 12 hours for the 1.5.1 Hadoop 2.7 binary 
> artifact
> - Run Jepsen tests for 12 hours for the 1.5.1 Hadoop 2.8 binary artifact
> 
> Cheers,
> Till
> 
> 
> On Wed, Jul 11, 2018 at 9:49 AM Chesnay Schepler  <mailto:ches...@apache.org>> wrote:
> Correction on my part, it does affect all packages.
> 
> I've also found the cause. To speed up the process I only built modules 
> that flink-dist depends on (see FLINK-9768). However flink-dist depends 
> on neither flink-examples-batch nor flink-examples-streaming, yet 
> happily accesses their target directory. The existing build process only 
> worked since _by chance_ these 2 modules are built before flink-dist 
> when doing a complete build.
> 
> I will rebuild the binaries (I don't think we have to cancel the RC for 
> this) and open a JIRA to fix the dependencies.
> 
> On 11.07.2018 09:27, Chesnay Schepler wrote:
> > oh, the packages that include hadoop are really missing it...
> >
> > On 11.07.2018 09:25, Chesnay Schepler wrote:
> >> @Yaz which binary package did you check? I looked into the 
> >> hadoop-free package and the folder is there.
> >>
> >> Did you maybe encounter an error when extracting the package?
> >>
> >> On 11.07.2018 05:44, Yaz Sh wrote:
> >>> -1
> >>>
> >>> ./examples/streaming folder is missing in binary packages
> >>>
> >>>
> >>> Cheers,
> >>> Yazdan
> >>>
> >>>> On Jul 10, 2018, at 9:57 PM, vino yang  >>>> <mailto:yanghua1...@gmail.com>> wrote:
> >>>>
> >>>> +1
> >>>> reviewed [1], [4] and [6]
> >>>>
> >>>> 2018-07-11 3:10 GMT+08:00 Chesnay Schepler  >>>> <mailto:ches...@apache.org>>:
> >>>>
> >>>>> Hi everyone,
> >>>>> Please review and vote on the release candidate #3 for the version 
> >>>>> 1.5.1,
> >>>>> as follows:
> >>>>> [ ] +1, Approve the release
> >>>>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>>>
> >>>>>
> >>>>> The complete staging area is available for your review, which 
> >>>>> includes:
> >>>>> * JIRA release notes [1],
> >>>>> * the official Apache source release and binary convenience 
> >>>>> releases to be
> >>>>> deployed to dist.apache.org <http://dist.apache.org/> [2], which are 
> >>>>> signed with the key with
> >>>>> fingerprint 11D464BA [3],
> >>>>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>>>> * source code tag "release-1.5.1-rc3" [5],
> >>>>> * website pull request listing the new release and adding 
> >>>>> announcement
> >>>>> blog post [6].
> >>>>>
> >>>>> This RC is a slightly modified version of the previous RC, with most
> >>>>> release testing being applicable to both release candidates. The 
> >>>>> minimum
> >>>>> voting duration will hence be reduced to 24 hours. It is adopted by
> >>>>> majority approval, with at least 3 PMC affirmative votes.
> >>>>>
> >>>>> Thanks,
> >>>>> Chesnay
> >>>>>
> >>>>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje 
> >>>>> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje>
> >>>>> ctId=12315522=12343053
> >>>>> [2] https://dist.apache.org/repos/dist/dev/flink/1.5.1/ 
> >>>>> <https://dist.apache.org/repos/dist/dev/flink/1.5.1/>
> >>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS 
> >>>>> <https://dist.apache.org/repos/dist/release/flink/KEYS>
> >>>>> [4] 
> >>>>> https://repository.apache.org/content/repositories/orgapacheflink-1171 
> >>>>> <https://repository.apache.org/content/repositories/orgapacheflink-1171>
> >>>>>  
> >>>>>
> >>>>> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h= 
> >>>>> <https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=>
> >>>>> refs/tags/release-1.5.1-rc3
> >>>>> [6] https://github.com/apache/flink-web/pull/112 
> >>>>> <https://github.com/apache/flink-web/pull/112>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>
> >>
> >
> >
> 

Re: [VOTE] Release 1.5.1, release candidate #3

[Yaz Sh]

-1

./examples/streaming folder is missing in binary packages


Cheers,
Yazdan

> On Jul 10, 2018, at 9:57 PM, vino yang  wrote:
> 
> +1
> reviewed [1], [4] and [6]
> 
> 2018-07-11 3:10 GMT+08:00 Chesnay Schepler :
> 
>> Hi everyone,
>> Please review and vote on the release candidate #3 for the version 1.5.1,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> 
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to be
>> deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint 11D464BA [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "release-1.5.1-rc3" [5],
>> * website pull request listing the new release and adding announcement
>> blog post [6].
>> 
>> This RC is a slightly modified version of the previous RC, with most
>> release testing being applicable to both release candidates. The minimum
>> voting duration will hence be reduced to 24 hours. It is adopted by
>> majority approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Chesnay
>> 
>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>> ctId=12315522=12343053
>> [2] https://dist.apache.org/repos/dist/dev/flink/1.5.1/
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapacheflink-1171
>> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=
>> refs/tags/release-1.5.1-rc3
>> [6] https://github.com/apache/flink-web/pull/112
>> 
>> 
>> 
>> 
>> 

Re: [VOTE] Release 1.5.1, release candidate #2

[Yaz Sh]

+1

Tests have been done on OS X 

- Ran in cluster mode ./bin/start-cluster.sh
- Checked that *.out files are empty
- Stopped cluster ./bin/stop-cluster.sh
- Checked for No Exceptions on log output

- Test Examples via WebUI
- Test Example via CLI with different flags (-d, -c, -q, -p)

- Added more task Managers via flink-config.yml and re-ran the Examples 
- Added more task Manager via ./bin/taskmanager.sh and re-ran the Examples 

- Checked “checksum” for all packages 
- Checked GPG signature for all packages
- Checked the README.md


Cheers,
Yazdan

 
> On Jul 6, 2018, at 11:09 AM, Chesnay Schepler  wrote:
> 
> Hi everyone,
> Please review and vote on the release candidate #2 for the version 1.5.1, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be 
> deployed to dist.apache.org [2], which are signed with the key with 
> fingerprint 11D464BA [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.5.1-rc2" [5],
> * website pull request listing the new release and adding announcement blog 
> post [6].
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Chesnay
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12343053
> [2] https://dist.apache.org/repos/dist/dev/flink/1.5.1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1170
> [5] 
> https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=refs/tags/release-1.5.1-rc2
> [6] https://github.com/apache/flink-web/pull/112
> 
> 
> 

Re: [VOTE] Release 1.5.1, release candidate #1

[Yaz Sh]

[-1]

I did follow validation release candidate guide line and when I tested the 
examples via web UI, I got some errors. here are the steps to reproduce the 
error.

- Ran the cluster mode (./bin/start-cluster.sh)
- Upload the WordCount.jar Example via web UI (I retry this with other examples 
also)

this has been done for bellow packages on MAC iOS

- flink-1.5.1-bin-hadoop26-scala_2.11.tgz
- flink-1.5.1-bin-scala_2.11.tgz

and I got these error, web UI also shows no task managers after this error 
occure. I ran the example via command line and it did work fine.


2018-07-05 14:32:19,589 ERROR 
org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler  - Request 
processing failed.
java.nio.file.NoSuchFileException: 
/var/folders/ns/31hcc1gx7t94m8yfvx8gdr20q4c61j/T/flink-web-927a2e20-426c-49b5-aaa1-908bff8f77ee/flink-web-upload/1a208f84-b51d-4fa4-86b2-c473b8ba8a2a
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at 
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
at java.nio.file.Files.walkFileTree(Files.java:2662)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at 
org.apache.flink.runtime.rest.handler.FileUploads.getUploadedFiles(FileUploads.java:68)
at 
org.apache.flink.runtime.rest.AbstractHandler.respondAsLeader(AbstractHandler.java:107)
at 
org.apache.flink.runtime.rest.handler.RedirectHandler.lambda$null$0(RedirectHandler.java:139)
at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at 
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:748)



Cheers,
Yazdan


> On Jul 5, 2018, at 2:00 PM, Chesnay Schepler  wrote:
> 
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 1.5.1, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be 
> deployed to dist.apache.org [2], which are signed with the key with 
> fingerprint 11D464BA [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.5.1-rc1" [5],
> * website pull request listing the new release and adding announcement blog 
> post [6].
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Chesnay
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12343053
> [2] https://dist.apache.org/repos/dist/dev/flink/1.5.1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1169
> [5] 
> https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=refs/tags/release-1.5.1-rc1
> [6] https://github.com/apache/flink-web/pull/112
> 
> 

[Proposal] Code Reviewer List

[Yaz Sh]

Hi Flink Community,

I am working with Flink for last couple of months and still learning it. 
Recently, I have started as a contributor in Flink to perform some minor 
changes and I am very excited to start working on bigger tasks. 
Questions I always have in mind, when I make a PR, are “Who should I tag in my 
PR for code review?” and “How long does it take to get a first response from 
the reviewer?” 

Usually if there is a Jira ticket reported on the task, I will tag reporter as 
well as people who commented on it. Also, I look at other PRs with similar 
topics and try to find out who is doing code review for those PRs.

Here I would like to share a proposal to create a "Code Reviewer List" as part 
of Flink contribution guideline. This list would provide name of code 
reviewers, their availabilities, topics of interest as well as turnaround time 
for initial code review. This will provide good starting point for new 
contributors like myself to find a code reviewer based on a topic he/she is 
contributing. 

I would appreciate if Flink community shares thoughts on this proposal!

Cheers,
Yazdan

Re: [VOTE] Release flink-shaded 4.0, release candidate #2

[Yaz Sh]

+1

I am new in this but I did following tests:

- mvn clean package -Dshade-sources passed will all modules SUCCESS
- checked all the licenses available
- netty-route license has been removed
- netty version 4.1.24.Final-4.0
- jackson version 2.7.9-4.0
- jackson-jsonSchema versio 2.7.9-4.0
- guava version 18.0-4.0
- asm version 5.0.4-4.0
- flink force shading version 4.0
- flink-shaded-netty-4 
/packaged_licenses
 
/LICENSE.jauter.txt
 removed
- flink-shaded-netty-4 
/packaged_licenses
 
/LICENSE.netty_router.txt
 removed


Cheers,
Yaz

> On May 30, 2018, at 10:29 AM, Piotr Nowojski  wrote:
> 
> I’m carrying over my +1
> 
>> On 30 May 2018, at 15:57, Chesnay Schepler  wrote:
>> 
>> +1
>> 
>> The checks i did for the previous RC are still valid.
>> Additionally i verified that the netty jar no longer bundles the licenses 
>> for netty-router.
>> 
>> On 30.05.2018 15:51, Chesnay Schepler wrote:
>>> Hi everyone,
>>> Please review and vote on the release candidate #2 for the version 4.0, as 
>>> follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>> 
>>> The complete staging area is available for your review, which includes:
>>> * GitHub release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org 
>>>  [2], 
>>> which are signed with the key with fingerprint 11D464BA [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "release-4.0-rc2" [5].
>>> 
>>> The vote will be open for at least 72 hours. It is adopted by majority 
>>> approval, with at least 3 PMC affirmative votes.
>>> 
>>> Thanks,
>>> Chesnay
>>> 
>>> [1] https://github.com/apache/flink-shaded/milestone/4?closed=1
>>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-4.0-rc2/
>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>>> [4] https://repository.apache.org/content/repositories/orgapacheflink-1167/
>>> [5] 
>>> https://gitbox.apache.org/repos/asf?p=flink-shaded.git;a=commit;h=33e86998fb8fe3d5b0a20260110d4ea636cc2528
>>> 
>>> 
>> 
> 

Re: Closing (automatically?) inactive pull requests

[Yaz Sh]

I have questions in this regard (you guys might have addressed it in this email 
chain):
 how PRs get assigned to a reviewer apart of a contributor tag someone? what if 
PR never gets a reviewer attention and it became in-active due to long review 
respond? should Bot assign a reviewer to a PR based on reviewers interest 
(i.e., defined via tags) and notify he/she if PR is waiting for review?


Cheers,  
/Yazdan


> On May 15, 2018, at 12:06 PM, Thomas Weise  wrote:
> 
> I like Till's proposal to notify the participants on the PR to PTAL. But I
> would also suggest to auto-close when no action is taken, with a friendly
> reminder that PRs can be reopened anytime.
> 
> The current situation with 350 open PRs may send a signal to contributors
> that it may actually be too much hassle to get a change committed in Flink.
> Since the count keeps on rising, this is also not a past problem. Pruning
> inactive PRs may help, that will also give a more accurate picture of the
> lack of review bandwidth, if that is the root cause.
> 
> Thanks,
> Thomas
> 
> 
> 
> 
> 
> On Tue, May 15, 2018 at 8:24 AM, Ted Yu  wrote:
> 
>> How does the bot decide whether the PR is waiting for reviews or is being
>> abandoned by contributor ?
>> 
>> How about letting the bot count the number of times contributor pings
>> committer(s) for review ?
>> When unanswered ping count crosses some threshold, say 3, the bot publishes
>> the JIRA and PR somewhere.
>> 
>> Cheers
>> 
>> On Tue, May 15, 2018 at 8:19 AM, Till Rohrmann 
>> wrote:
>> 
>>> I'm a bit torn here because I can see the pros and cons for both sides.
>>> 
>>> Maybe a compromise could be to not have a closing but a monitoring bot
>>> which notifies us about inactive PRs. This could then trigger an
>>> investigation of the underlying problem and ultimately lead to a
>> conscious
>>> decision to close or keep the PR open. As such it is not strictly
>> necessary
>>> to have such a bot but it would at least remind us to make a decision
>> about
>>> older PRs with no activity.
>>> 
>>> Cheers,
>>> Till
>>> 
>>> On Tue, May 15, 2018 at 3:26 PM, Chesnay Schepler 
>>> wrote:
>>> 
 /So far I did it twice for older PRs. In both cases I didn’t get any
 response and I even forgot in which PRs I had asked this question, so
>>> now I
 can not even close them :S/
 
 To be honest this sounds more like an issue with how your organize your
 work. No amount of closing PRs can fix that.
 With GitBox we can assign reviewers to a PR, but I'm not sure whether
>> it
 only allows committers to assign people.
 Bookmarks or text files might help as well./
 /
 
 /Regarding only 30% blocked on contributor. I wonder what would be this
 number if we tried to ask in the rest of old PRs “Hey, are you still
 interested in reviewing/merging this?”.  If old PR is waiting for a
 reviewer for X months, it doesn’t mean that’s not abandoned. Even if it
 doesn’t, reducing our overhead by those 30% of the PRs is something./
 
 No doubt the number would be higher if we were to go back, but as i
 explained earlier that is not a reason to close it. If a PR is
>> abandoned
 because we messed up we should still /try /to get it in.
 
 /This is kind of whole point of what I was proposing. If the PR author
>> is
 still there, and can respond/bump/interrupt the closing timeout, that’s
 great. Gives us even more sense of urgency to review it./
 
 Unfortunately knowing that it is more urgent is irrelevant, as we
 currently don't have the manpower to review them. Reviving them now
>> would
 serve no purpose. The alternative is to wait until more people become
 active reviewers.
 
 /As a last resort, closing PR after timeout is not the end of the
>> world.
 It always can be reopened./
 
 Let's be realistic here, it will not be reopened.
 
 
 On 15.05.2018 14:21, Piotr Nowojski wrote:
 
> I agree that we have other, even more important, problems with
>> reviewing
> PR and community, but that shouldn’t block us from trying to clean up
> things a little bit and minimise the effort needed for reviewing PRs.
>>> Now
> before reviewing/picking older PRs I had to ask this “Hey, are you
>> still
> interested in merging this?” manually and wait for the response. If it
> doesn’t come, I have to remember to go back and close PR, which I of
>>> course
> forget to do. Bah, so far I did it twice for older PRs. In both cases
>> I
> didn’t get any response and I even forgot in which PRs I had asked
>> this
> question, so now I can not even close them :S Wasted effort and wasted
>>> time
> on context switching for me and for everyone else that will have to
>>> scroll
> pass or look on those PR to check their status.
> 
> Keep in mind that I am not proposing to close 

Re: request for JIRA contributor permissions

[Yaz Sh]

Thanks! I have it now.

Cheers,
Yaz

On Sun, May 13, 2018 at 12:39 PM Chesnay Schepler <ches...@apache.org>
wrote:

> I've given you contributor permissions.
>
> On 13.05.2018 18:25, Yaz Sh wrote:
> > I would appreciate if someone grant me the JIRA contributor permissions
> to be able to assign tickets to myself that I am working on them.
> > FLINK-9348 <https://issues.apache.org/jira/browse/FLINK-9348>
> > FLINK-9343 <https://issues.apache.org/jira/browse/FLINK-9343>
> > FLINK-9317 <https://issues.apache.org/jira/browse/FLINK-9317>
> >
> > Here is my jira id:
> > Username: yazdanjs
> > Cheers,
> > Yazdan
>
>
>

request for JIRA contributor permissions

[Yaz Sh]

I would appreciate if someone grant me the JIRA contributor permissions to be 
able to assign tickets to myself that I am working on them. 
FLINK-9348 
FLINK-9343 
FLINK-9317 

Here is my jira id: 
Username:   yazdanjs
Cheers,
Yazdan