[jira] [Created] (FLINK-8969) Move TimerService into state backend
Bowen Li created FLINK-8969: --- Summary: Move TimerService into state backend Key: FLINK-8969 URL: https://issues.apache.org/jira/browse/FLINK-8969 Project: Flink Issue Type: Improvement Components: State Backends, Checkpointing Reporter: Bowen Li Assignee: Bowen Li upon discussion with [~aljoscha]. More details need to be added here -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[ANNOUNCE] Apache Flink 1.3.3 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.3.3, which is the third bugfix release for the Apache Flink 1.3 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/news/2018/03/15/release-1.3.3.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12341142 We would like to thank all contributors of the Apache Flink community who made this release possible! Cheers, Gordon
Re: Flip 6 mesos support
Hi Till, This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR to resolve the issue, your help to take a look will be great. Please let me know what I can help to test the Kerberos authentication, I am decently familiar with the Kerberos and YARN security part in Flink. As a starting point, I'd suggest to add an integration test similar to YARNSessionFIFOSecuredITCase for flip6. Shuyi On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann wrote: > Hi Renjie, > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. You're right > that we should fix this test. There is FLINK-8562 which seems to address > the problem. Will take a look. > > Additionally, we want to test Kerberos authentication explicitly as part of > the release testing for Flink 1.5. I will shortly send around a mail where > I will lay out the ongoing testing efforts and where more is needed. > > Cheers, > Till > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu > wrote: > > > Thanks for the clarification > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 wrote: > > > > > Hi Renjie, > > > if I am not misunderstand, you just need to start the cluster as normal > > as > > > before. The dispatcher and resourcemanager are spawned by > > ClusterEntryPoint > > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > > YarnSessionClusterEntrypoint), and the TM are spawned by > ResourceManager > > > lazily (ResourceManager will setup TM according to the submitted job) > or > > > spawned by the setup script (you can have a look at start-cluster.sh). > > > > > > > > > Best Regards, > > > Sihua Zhou > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > On 03/15/2018 10:14，Renjie Liu wrote： > > > Hi, Till: > > > In fact I'm asking how to deploy other components such as dispatcher, > > etc. > > > > > > Till Rohrmann 于 2018年3月15日周四 上午12:17写道： > > > > > > Hi Renjie, > > > > > > in the current master and release-1.5 branch flip-6 is activated by > > > default. If you want to turn it off you have to add `mode: old` to your > > > flink-conf.yaml. I'm really happy that you want to test it out :-) > > > > > > Cheers, > > > Till > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu > > > wrote: > > > > > > Hi Till: > > > Is there any doc on deploying flink in flip6 mode? We want to help > > > testing > > > it. > > > > > > Till Rohrmann 于 2018年3月14日周三 下午7:08写道： > > > > > > Hi Renjie, > > > > > > in order to make Mesos work, we only needed to implement a Mesos > > > specific > > > ResourceManager. Look at MesosResourceManager for more details. As > > > dispatcher, we use the StandaloneDispatcher which is spawned by > > > the MesosSessionClusterEntrypoint. > > > > > > Cheers, > > > Till > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu > > > wrote: > > > > > > Hi all: > > > I'm reading the source code and it seems that flip6 does not support > > > mesos? > > > According to the design, client send job graph to dispatcher and > > > dispatcher > > > spawn job mananger and resource manager for job execution. But I > > > can't > > > find > > > dispatcher implementation for mesos. > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > -- "So you have to trust that the dots will somehow connect in your future."
Re: [RESULT] [VOTE] Release 1.3.3, release candidate #2
The voting time has passed and I'm happy to announce that we've collected enough votes to release this RC as Flink 1.3.3. +1 votes: - Ted (non-binding) - Timo (binding) - Chesnay (binding) - Fabian (binding) - Gordon (binding) - Henry (binding) That's 6 votes, 5 binding. No 0 or -1 votes. I'll modify the JIRA release notes / blog post to include the extra commit. Once the blog post is published and artifacts are pushed, a separate announcement will be sent. Cheers, Gordon -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: [VOTE] Release 1.3.3, release candidate #2
Signature file looks good Hash files look good LICENSE file exists NOTICE file exists Source artifact build and local tests passed +1 On Wed, Mar 14, 2018 at 3:26 AM, Tzu-Li (Gordon) Tai wrote: > Hi everyone, > > Please review and vote on release candidate #2 for Flink 1.3.3, as > follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release and binary convenience releases to be > deployed to dist.apache.org [2], which are signed with the key with > fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code branch “release-1.3.3-rc2” [5], > * website pull request listing the new release [6]. > * A complete list of all new commits in release-1.3.3-rc2, since > release-1.3.2 [7] > > This release candidate contains fixes for only the following issues: > [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase > [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint. > checkpointsMatch() > [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in > infinite loop > [FLINK-7783] Don’t always remove checkpoints in > ZooKeeperCompletedCheckpointStore#recover() > > Since the last candidate was cancelled only due to incorrect binaries in > the source artifacts, I think we can also have a shorter voting period for > RC2. > > Please test the release and vote for the release candidate before Thursday > (March 15th), 7pm CET. > It is adopted by majority approval, with at least 3 PMC affirmative votes. > > Please let me know if you disagree with the shortened voting time. > > Thanks, > Gordon > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12315522&version=12341142 > [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc2/ > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1151 > [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a= > shortlog;h=refs/heads/release-1.3.3-rc2 > [6] https://github.com/apache/flink-web/pull/104 > [7] > - 90559b5413455d9d0f2b61c389a60e26e5c87800 [hotfix] Properly delete temp > flink dir in create_source_release.sh > - 99c0353a34c09af5bedb73f525f691dd7e78fcdd [hotfix] Ensure pristine > release in tools/releasing/create_source_release.sh > - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify ZooKeeper > checkpoint store behaviour with ITCase > - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare > checkpoints with order in CompletedCheckpoint.checkpointsMatch() > - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix > ZookeeperCompleted checkpoint store can get stuck in infinite loop > - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always > remove checkpoints in ZooKeeperCompletedCheckpointStore#recover() > >
Re: [VOTE] Release 1.3.3, release candidate #2
+1 - checked hashes and signatures - built from staged source artifact - ran state machine job chaos monkey test on AWS EMR - ran quickstarts I also agree that we do not need to cancel the RC because of the unintended PythonPlanBinderTest commit. Cheers, Gordon On 15 March 2018 at 11:08:30 PM, Fabian Hueske (fhue...@gmail.com) wrote: +1 - checked hashes and signatures - build from src.tgz with unit tests (mvn clean package) - checked commits for dependency changes and didn't find any Thanks, Fabian 2018-03-15 14:06 GMT+01:00 Chesnay Schepler : > Given that the commit only affects a test I'd be in favor of continuing > the release process and updating the release notes. > > > On 15.03.2018 14:01, Timo Walther wrote: > >> +1 >> >> - built and run tests locally >> - run some training exercises locally >> - looked over the commits >> >> I found one minor thing that should be mentioned. It seems that there is >> one unintended commit in the release: >> >> 2455df962c76d20f1c07a57d6ed0118d1d1a067c [FLINK-7283][python] fix >> PythonPlanBinderTest issues with python paths >> >> It only affects a test so I think we should not cancel the release >> because of that but update the release notes and the Jira issue. >> >> What do you think? >> >> Regards, >> Timo >> >> Am 15.03.18 um 10:10 schrieb Chesnay Schepler: >> >>> +1 >>> >>> * verified that compilation works >>> * ran a few test examples against a local cluster >>> * src artifact contains no binaries >>> * verified signatures >>> >>> On 14.03.2018 21:49, Ted Yu wrote: >>> +1 Ran the following command - passed: mvn clean package -Pjdk8 On Wed, Mar 14, 2018 at 3:26 AM, Tzu-Li (Gordon) Tai < tzuli...@apache.org> wrote: Hi everyone, > > Please review and vote on release candidate #2 for Flink 1.3.3, as > follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release and binary convenience releases > to be > deployed to dist.apache.org [2], which are signed with the key with > fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code branch “release-1.3.3-rc2” [5], > * website pull request listing the new release [6]. > * A complete list of all new commits in release-1.3.3-rc2, since > release-1.3.2 [7] > > This release candidate contains fixes for only the following issues: > [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase > [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint. > checkpointsMatch() > [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in > infinite loop > [FLINK-7783] Don’t always remove checkpoints in > ZooKeeperCompletedCheckpointStore#recover() > > Since the last candidate was cancelled only due to incorrect binaries > in > the source artifacts, I think we can also have a shorter voting period > for > RC2. > > Please test the release and vote for the release candidate before > Thursday > (March 15th), 7pm CET. > It is adopted by majority approval, with at least 3 PMC affirmative > votes. > > Please let me know if you disagree with the shortened voting time. > > Thanks, > Gordon > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12315522&version=12341142 > [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc2/ > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapache > flink-1151 > [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a= > shortlog;h=refs/heads/release-1.3.3-rc2 > [6] https://github.com/apache/flink-web/pull/104 > [7] > - 90559b5413455d9d0f2b61c389a60e26e5c87800 [hotfix] Properly delete > temp > flink dir in create_source_release.sh > - 99c0353a34c09af5bedb73f525f691dd7e78fcdd [hotfix] Ensure pristine > release in tools/releasing/create_source_release.sh > - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify > ZooKeeper > checkpoint store behaviour with ITCase > - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare > checkpoints with order in CompletedCheckpoint.checkpointsMatch() > - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix > ZookeeperCompleted checkpoint store can get stuck in infinite loop > - f69bdf207b92ca47a5ce3
Re: Correct way to reference Hadoop dependencies in Flink 1.4.0
Hi, I believe for FileSystems to be correctly be picked up they have to be in the lib/ folder of Flink. Stephan (cc'ed), please correct me if I'm wrong here, you probably know that one best. Aljoscha > On 14. Mar 2018, at 18:26, l...@lyft.com wrote: > > Hi, > > I am running this on a Hadoop-free cluster (i.e. no YARN etc.). I have the > following dependencies packaged in my user application JAR: > > aws-java-sdk 1.7.4 > flink-hadoop-fs 1.4.0 > flink-shaded-hadoop2 1.4.0 > flink-connector-filesystem_2.11 1.4.0 > hadoop-common 2.7.4 > hadoop-aws 2.7.4 > > I have also tried the following conf: > classloader.resolve-order: parent-first > fs.hdfs.hadoopconf: /srv/hadoop/hadoop-2.7.5/etc/hadoop > > But no luck. Anything else I could be missing? > > On 2018/03/14 18:57:47, Francesco Ciuci wrote: >> Hi, >> >> You do not just need the hadoop dependencies in the jar but you need to >> have the hadoop file system running in your machine/cluster. >> >> Regards >> >> On 14 March 2018 at 18:38, l...@lyft.com wrote: >> >>> I'm trying to use a BucketingSink to write files to S3 in my Flink job. >>> >>> I have the Hadoop dependencies I need packaged in my user application jar. >>> However, on running the job I get the following error (from the >>> taskmanager): >>> >>> java.lang.RuntimeException: Error while creating FileSystem when >>> initializing the state of the BucketingSink. >>>at org.apache.flink.streaming.connectors.fs.bucketing. >>> BucketingSink.initializeState(BucketingSink.java:358) >>>at org.apache.flink.streaming.util.functions. >>> StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) >>>at org.apache.flink.streaming.util.functions. >>> StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java: >>> 160) >>>at org.apache.flink.streaming.api.operators. >>> AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator. >>> java:96) >>>at org.apache.flink.streaming.api.operators. >>> AbstractStreamOperator.initializeState(AbstractStreamOperator.java:259) >>>at org.apache.flink.streaming.runtime.tasks.StreamTask. >>> initializeOperators(StreamTask.java:694) >>>at org.apache.flink.streaming.runtime.tasks.StreamTask. >>> initializeState(StreamTask.java:682) >>>at org.apache.flink.streaming.runtime.tasks.StreamTask. >>> invoke(StreamTask.java:253) >>>at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) >>>at java.lang.Thread.run(Thread.java:748) >>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: >>> Could not find a file system implementation for scheme 's3a'. The scheme is >>> not directly supported by Flink and no Hadoop file system to support this >>> scheme could be loaded. >>>at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( >>> FileSystem.java:405) >>>at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320) >>>at org.apache.flink.streaming.connectors.fs.bucketing. >>> BucketingSink.createHadoopFileSystem(BucketingSink.java:1125) >>>at org.apache.flink.streaming.connectors.fs.bucketing. >>> BucketingSink.initFileSystem(BucketingSink.java:411) >>>at org.apache.flink.streaming.connectors.fs.bucketing. >>> BucketingSink.initializeState(BucketingSink.java:355) >>>... 9 common frames omitted >>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: >>> Hadoop is not in the classpath/dependencies. >>>at org.apache.flink.core.fs.UnsupportedSchemeFactory.create( >>> UnsupportedSchemeFactory.java:64) >>>at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( >>> FileSystem.java:401) >>>... 13 common frames omitted >>> >>> What's the right way to do this? >>> >>
Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0
Nice, it seems we're in agreement. I'll batch-edit the existing issues to remove the 1.6.0 fixVersion. Aljoscha > On 15. Mar 2018, at 08:34, Thomas Weise wrote: > > +1 > > Since it is guaranteed that all pre 1.5.0 release patches also make it to > master, it would be confusing to mark them resolved for 1.6.x. > > > On Thu, Mar 15, 2018 at 8:25 AM, Stephan Ewen wrote: > >> +1 >> >> Important: Please still write in the notes (when closing the issue) the >> commit hashes for both 1.5 (release-1.5) and 1.6 (master). >> Otherwise it makes hind-sight digging into the fixed more complicated. >> >> >> >> On Thu, Mar 15, 2018 at 3:28 PM, Felix Cheung >> wrote: >> >>> +1 >>> >>> >>> From: Till Rohrmann >>> Sent: Thursday, March 15, 2018 5:45:14 AM >>> To: dev@flink.apache.org >>> Subject: Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as >>> resolved in 1.6.0 >>> >>> +1 for marking bugs as fixed 1.5.0 only >>> >>> On Thu, Mar 15, 2018 at 11:09 AM, Ted Yu wrote: >>> +1 on marking bugs as fixed for 1.5.0 only. Original message From: Piotr Nowojski < pi...@data-artisans.com> Date: 3/15/18 12:48 AM (GMT-08:00) To: dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0 Same as Chesnay +1 for marking bugs as fixed 1.5.0 only > On 15 Mar 2018, at 07:57, Chesnay Schepler >> wrote: > > +1 to mark bugs as fixed in 1.5.0 only. > > On 15.03.2018 01:40, Aljoscha Krettek wrote: >> Hi, >> >> We currently have some issues that are marked as resolved for both 1.5.0 and 1.6.0 [1]. The reason is that we have the release-1.5 branch >>> and the master branch, which will eventually become the branch for 1.6.0. >> >> I think this can lead to confusion because the release notes are created based on that data. Say, we fix a bug "foo" after we created >> the release-1.5 branch. Now we will have "[FLINK-] Fixed foo" in the release notes for 1.5.0 and 1.6.0. We basically start our Flink 1.6.0 release notes with around 50 issues that were never bugs in 1.6.0 >> because they were fixed in 1.5.0. Plus, having "[FLINK-] Fixed foo" in the 1.6.0 release notes indicates that "foo" was actually a bug in 1.5.0 (because we now had to fix it), but it wasn't. >> >> I would propose to remove fixVersion 1.6.0 from all issues that have 1.5.0 as fixVersion. What do you think? >> >> On a side note: a bug that is fixed in 1.5.1 should be marked as >> fixed for 1.6.0 separately, because 1.6.0 is not a direct successor to 1.5.1. >> >> Best, >> Aljoscha >> >> [1] https://issues.apache.org/jira/issues/?jql=project%20% 3D%20FLINK%20and%20fixVersion%20%3D%201.6.0%20and%20resolution%20!%3D% 20unresolved > > >>> >>
[jira] [Created] (FLINK-8968) Fix native resource leak caused by ReadOptions
Sihua Zhou created FLINK-8968:
-
Summary: Fix native resource leak caused by ReadOptions
Key: FLINK-8968
URL: https://issues.apache.org/jira/browse/FLINK-8968
Project: Flink
Issue Type: Bug
Components: State Backends, Checkpointing
Affects Versions: 1.5.0
Reporter: Sihua Zhou
Assignee: Sihua Zhou
Fix For: 1.5.0
We should pull the creation of ReadOptions out of the loop in
{{RocksDBFullSnapshotOperation.writeKVStateMetaData()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0
+1 Since it is guaranteed that all pre 1.5.0 release patches also make it to master, it would be confusing to mark them resolved for 1.6.x. On Thu, Mar 15, 2018 at 8:25 AM, Stephan Ewen wrote: > +1 > > Important: Please still write in the notes (when closing the issue) the > commit hashes for both 1.5 (release-1.5) and 1.6 (master). > Otherwise it makes hind-sight digging into the fixed more complicated. > > > > On Thu, Mar 15, 2018 at 3:28 PM, Felix Cheung > wrote: > > > +1 > > > > > > From: Till Rohrmann > > Sent: Thursday, March 15, 2018 5:45:14 AM > > To: dev@flink.apache.org > > Subject: Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as > > resolved in 1.6.0 > > > > +1 for marking bugs as fixed 1.5.0 only > > > > On Thu, Mar 15, 2018 at 11:09 AM, Ted Yu wrote: > > > > > +1 on marking bugs as fixed for 1.5.0 only. > > > Original message From: Piotr Nowojski < > > > pi...@data-artisans.com> Date: 3/15/18 12:48 AM (GMT-08:00) To: > > > dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as > > > resolved in 1.5.0 as resolved in 1.6.0 > > > Same as Chesnay > > > > > > +1 for marking bugs as fixed 1.5.0 only > > > > > > > On 15 Mar 2018, at 07:57, Chesnay Schepler > wrote: > > > > > > > > +1 to mark bugs as fixed in 1.5.0 only. > > > > > > > > On 15.03.2018 01:40, Aljoscha Krettek wrote: > > > >> Hi, > > > >> > > > >> We currently have some issues that are marked as resolved for both > > > 1.5.0 and 1.6.0 [1]. The reason is that we have the release-1.5 branch > > and > > > the master branch, which will eventually become the branch for 1.6.0. > > > >> > > > >> I think this can lead to confusion because the release notes are > > > created based on that data. Say, we fix a bug "foo" after we created > the > > > release-1.5 branch. Now we will have "[FLINK-] Fixed foo" in the > > > release notes for 1.5.0 and 1.6.0. We basically start our Flink 1.6.0 > > > release notes with around 50 issues that were never bugs in 1.6.0 > because > > > they were fixed in 1.5.0. Plus, having "[FLINK-] Fixed foo" in the > > > 1.6.0 release notes indicates that "foo" was actually a bug in 1.5.0 > > > (because we now had to fix it), but it wasn't. > > > >> > > > >> I would propose to remove fixVersion 1.6.0 from all issues that have > > > 1.5.0 as fixVersion. What do you think? > > > >> > > > >> On a side note: a bug that is fixed in 1.5.1 should be marked as > fixed > > > for 1.6.0 separately, because 1.6.0 is not a direct successor to 1.5.1. > > > >> > > > >> Best, > > > >> Aljoscha > > > >> > > > >> [1] https://issues.apache.org/jira/issues/?jql=project%20% > > > 3D%20FLINK%20and%20fixVersion%20%3D%201.6.0%20and%20resolution%20!%3D% > > > 20unresolved > > > > > > > > > > > > > > > > >
Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0
+1 Important: Please still write in the notes (when closing the issue) the commit hashes for both 1.5 (release-1.5) and 1.6 (master). Otherwise it makes hind-sight digging into the fixed more complicated. On Thu, Mar 15, 2018 at 3:28 PM, Felix Cheung wrote: > +1 > > > From: Till Rohrmann > Sent: Thursday, March 15, 2018 5:45:14 AM > To: dev@flink.apache.org > Subject: Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as > resolved in 1.6.0 > > +1 for marking bugs as fixed 1.5.0 only > > On Thu, Mar 15, 2018 at 11:09 AM, Ted Yu wrote: > > > +1 on marking bugs as fixed for 1.5.0 only. > > Original message From: Piotr Nowojski < > > pi...@data-artisans.com> Date: 3/15/18 12:48 AM (GMT-08:00) To: > > dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as > > resolved in 1.5.0 as resolved in 1.6.0 > > Same as Chesnay > > > > +1 for marking bugs as fixed 1.5.0 only > > > > > On 15 Mar 2018, at 07:57, Chesnay Schepler wrote: > > > > > > +1 to mark bugs as fixed in 1.5.0 only. > > > > > > On 15.03.2018 01:40, Aljoscha Krettek wrote: > > >> Hi, > > >> > > >> We currently have some issues that are marked as resolved for both > > 1.5.0 and 1.6.0 [1]. The reason is that we have the release-1.5 branch > and > > the master branch, which will eventually become the branch for 1.6.0. > > >> > > >> I think this can lead to confusion because the release notes are > > created based on that data. Say, we fix a bug "foo" after we created the > > release-1.5 branch. Now we will have "[FLINK-] Fixed foo" in the > > release notes for 1.5.0 and 1.6.0. We basically start our Flink 1.6.0 > > release notes with around 50 issues that were never bugs in 1.6.0 because > > they were fixed in 1.5.0. Plus, having "[FLINK-] Fixed foo" in the > > 1.6.0 release notes indicates that "foo" was actually a bug in 1.5.0 > > (because we now had to fix it), but it wasn't. > > >> > > >> I would propose to remove fixVersion 1.6.0 from all issues that have > > 1.5.0 as fixVersion. What do you think? > > >> > > >> On a side note: a bug that is fixed in 1.5.1 should be marked as fixed > > for 1.6.0 separately, because 1.6.0 is not a direct successor to 1.5.1. > > >> > > >> Best, > > >> Aljoscha > > >> > > >> [1] https://issues.apache.org/jira/issues/?jql=project%20% > > 3D%20FLINK%20and%20fixVersion%20%3D%201.6.0%20and%20resolution%20!%3D% > > 20unresolved > > > > > > > > > > >
Re: [VOTE] Release 1.3.3, release candidate #2
+1 - checked hashes and signatures - build from src.tgz with unit tests (mvn clean package) - checked commits for dependency changes and didn't find any Thanks, Fabian 2018-03-15 14:06 GMT+01:00 Chesnay Schepler : > Given that the commit only affects a test I'd be in favor of continuing > the release process and updating the release notes. > > > On 15.03.2018 14:01, Timo Walther wrote: > >> +1 >> >> - built and run tests locally >> - run some training exercises locally >> - looked over the commits >> >> I found one minor thing that should be mentioned. It seems that there is >> one unintended commit in the release: >> >> 2455df962c76d20f1c07a57d6ed0118d1d1a067c [FLINK-7283][python] fix >> PythonPlanBinderTest issues with python paths >> >> It only affects a test so I think we should not cancel the release >> because of that but update the release notes and the Jira issue. >> >> What do you think? >> >> Regards, >> Timo >> >> Am 15.03.18 um 10:10 schrieb Chesnay Schepler: >> >>> +1 >>> >>> * verified that compilation works >>> * ran a few test examples against a local cluster >>> * src artifact contains no binaries >>> * verified signatures >>> >>> On 14.03.2018 21:49, Ted Yu wrote: >>> +1 Ran the following command - passed: mvn clean package -Pjdk8 On Wed, Mar 14, 2018 at 3:26 AM, Tzu-Li (Gordon) Tai < tzuli...@apache.org> wrote: Hi everyone, > > Please review and vote on release candidate #2 for Flink 1.3.3, as > follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release and binary convenience releases > to be > deployed to dist.apache.org [2], which are signed with the key with > fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code branch “release-1.3.3-rc2” [5], > * website pull request listing the new release [6]. > * A complete list of all new commits in release-1.3.3-rc2, since > release-1.3.2 [7] > > This release candidate contains fixes for only the following issues: > [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase > [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint. > checkpointsMatch() > [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in > infinite loop > [FLINK-7783] Don’t always remove checkpoints in > ZooKeeperCompletedCheckpointStore#recover() > > Since the last candidate was cancelled only due to incorrect binaries > in > the source artifacts, I think we can also have a shorter voting period > for > RC2. > > Please test the release and vote for the release candidate before > Thursday > (March 15th), 7pm CET. > It is adopted by majority approval, with at least 3 PMC affirmative > votes. > > Please let me know if you disagree with the shortened voting time. > > Thanks, > Gordon > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12315522&version=12341142 > [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc2/ > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapache > flink-1151 > [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a= > shortlog;h=refs/heads/release-1.3.3-rc2 > [6] https://github.com/apache/flink-web/pull/104 > [7] > - 90559b5413455d9d0f2b61c389a60e26e5c87800 [hotfix] Properly delete > temp > flink dir in create_source_release.sh > - 99c0353a34c09af5bedb73f525f691dd7e78fcdd [hotfix] Ensure pristine > release in tools/releasing/create_source_release.sh > - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify > ZooKeeper > checkpoint store behaviour with ITCase > - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare > checkpoints with order in CompletedCheckpoint.checkpointsMatch() > - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix > ZookeeperCompleted checkpoint store can get stuck in infinite loop > - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always > remove checkpoints in ZooKeeperCompletedCheckpointStore#recover() > > > >> >> >
[jira] [Created] (FLINK-8967) Port NetworkStackThroughputITCase to flip6
Chesnay Schepler created FLINK-8967: --- Summary: Port NetworkStackThroughputITCase to flip6 Key: FLINK-8967 URL: https://issues.apache.org/jira/browse/FLINK-8967 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8965) Port TimestampITCase to flip6
Chesnay Schepler created FLINK-8965: --- Summary: Port TimestampITCase to flip6 Key: FLINK-8965 URL: https://issues.apache.org/jira/browse/FLINK-8965 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8966) Port AvroExternalJarProgramITCase to flip6
Chesnay Schepler created FLINK-8966: --- Summary: Port AvroExternalJarProgramITCase to flip6 Key: FLINK-8966 URL: https://issues.apache.org/jira/browse/FLINK-8966 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8959) Port AccumulatorLiveITCase to flip6
Chesnay Schepler created FLINK-8959: --- Summary: Port AccumulatorLiveITCase to flip6 Key: FLINK-8959 URL: https://issues.apache.org/jira/browse/FLINK-8959 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8963) Port BigUserProgramJobSubmitITCase to flip6
Chesnay Schepler created FLINK-8963: --- Summary: Port BigUserProgramJobSubmitITCase to flip6 Key: FLINK-8963 URL: https://issues.apache.org/jira/browse/FLINK-8963 Project: Flink Issue Type: Improvement Components: Job-Submission, Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8964) Port JobSubmissionFailsITCase to flip6
Chesnay Schepler created FLINK-8964: --- Summary: Port JobSubmissionFailsITCase to flip6 Key: FLINK-8964 URL: https://issues.apache.org/jira/browse/FLINK-8964 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8962) Port AccumulatorErrorITCase to flip6
Chesnay Schepler created FLINK-8962: --- Summary: Port AccumulatorErrorITCase to flip6 Key: FLINK-8962 URL: https://issues.apache.org/jira/browse/FLINK-8962 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8961) Port JobRetrievalITCase to flip6
Chesnay Schepler created FLINK-8961: --- Summary: Port JobRetrievalITCase to flip6 Key: FLINK-8961 URL: https://issues.apache.org/jira/browse/FLINK-8961 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8960) Port SavepointITCase to flip6
Chesnay Schepler created FLINK-8960: --- Summary: Port SavepointITCase to flip6 Key: FLINK-8960 URL: https://issues.apache.org/jira/browse/FLINK-8960 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8958) Port TaskCancelAsyncProducerConsumerITCase to flip6
Chesnay Schepler created FLINK-8958: --- Summary: Port TaskCancelAsyncProducerConsumerITCase to flip6 Key: FLINK-8958 URL: https://issues.apache.org/jira/browse/FLINK-8958 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8957) Port JMXJobManagerMetricTest to flip6
Chesnay Schepler created FLINK-8957: --- Summary: Port JMXJobManagerMetricTest to flip6 Key: FLINK-8957 URL: https://issues.apache.org/jira/browse/FLINK-8957 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8956) Port RescalingITCase to flip6
Chesnay Schepler created FLINK-8956: --- Summary: Port RescalingITCase to flip6 Key: FLINK-8956 URL: https://issues.apache.org/jira/browse/FLINK-8956 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8955) Port ClassLoaderITCase to flip6
Chesnay Schepler created FLINK-8955: --- Summary: Port ClassLoaderITCase to flip6 Key: FLINK-8955 URL: https://issues.apache.org/jira/browse/FLINK-8955 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0
+1 From: Till Rohrmann Sent: Thursday, March 15, 2018 5:45:14 AM To: dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0 +1 for marking bugs as fixed 1.5.0 only On Thu, Mar 15, 2018 at 11:09 AM, Ted Yu wrote: > +1 on marking bugs as fixed for 1.5.0 only. > Original message From: Piotr Nowojski < > pi...@data-artisans.com> Date: 3/15/18 12:48 AM (GMT-08:00) To: > dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as > resolved in 1.5.0 as resolved in 1.6.0 > Same as Chesnay > > +1 for marking bugs as fixed 1.5.0 only > > > On 15 Mar 2018, at 07:57, Chesnay Schepler wrote: > > > > +1 to mark bugs as fixed in 1.5.0 only. > > > > On 15.03.2018 01:40, Aljoscha Krettek wrote: > >> Hi, > >> > >> We currently have some issues that are marked as resolved for both > 1.5.0 and 1.6.0 [1]. The reason is that we have the release-1.5 branch and > the master branch, which will eventually become the branch for 1.6.0. > >> > >> I think this can lead to confusion because the release notes are > created based on that data. Say, we fix a bug "foo" after we created the > release-1.5 branch. Now we will have "[FLINK-] Fixed foo" in the > release notes for 1.5.0 and 1.6.0. We basically start our Flink 1.6.0 > release notes with around 50 issues that were never bugs in 1.6.0 because > they were fixed in 1.5.0. Plus, having "[FLINK-] Fixed foo" in the > 1.6.0 release notes indicates that "foo" was actually a bug in 1.5.0 > (because we now had to fix it), but it wasn't. > >> > >> I would propose to remove fixVersion 1.6.0 from all issues that have > 1.5.0 as fixVersion. What do you think? > >> > >> On a side note: a bug that is fixed in 1.5.1 should be marked as fixed > for 1.6.0 separately, because 1.6.0 is not a direct successor to 1.5.1. > >> > >> Best, > >> Aljoscha > >> > >> [1] https://issues.apache.org/jira/issues/?jql=project%20% > 3D%20FLINK%20and%20fixVersion%20%3D%201.6.0%20and%20resolution%20!%3D% > 20unresolved > > > > > >
[jira] [Created] (FLINK-8954) Escape control characters when outputting on SQL Client CLI
Timo Walther created FLINK-8954: --- Summary: Escape control characters when outputting on SQL Client CLI Key: FLINK-8954 URL: https://issues.apache.org/jira/browse/FLINK-8954 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: Timo Walther Control characters in the result output of a SQL query influence the behavior of the CLI. We should escape everything that could cause side effects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Release 1.3.3, release candidate #2
Given that the commit only affects a test I'd be in favor of continuing the release process and updating the release notes. On 15.03.2018 14:01, Timo Walther wrote: +1 - built and run tests locally - run some training exercises locally - looked over the commits I found one minor thing that should be mentioned. It seems that there is one unintended commit in the release: 2455df962c76d20f1c07a57d6ed0118d1d1a067c [FLINK-7283][python] fix PythonPlanBinderTest issues with python paths It only affects a test so I think we should not cancel the release because of that but update the release notes and the Jira issue. What do you think? Regards, Timo Am 15.03.18 um 10:10 schrieb Chesnay Schepler: +1 * verified that compilation works * ran a few test examples against a local cluster * src artifact contains no binaries * verified signatures On 14.03.2018 21:49, Ted Yu wrote: +1 Ran the following command - passed: mvn clean package -Pjdk8 On Wed, Mar 14, 2018 at 3:26 AM, Tzu-Li (Gordon) Tai wrote: Hi everyone, Please review and vote on release candidate #2 for Flink 1.3.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code branch “release-1.3.3-rc2” [5], * website pull request listing the new release [6]. * A complete list of all new commits in release-1.3.3-rc2, since release-1.3.2 [7] This release candidate contains fixes for only the following issues: [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint. checkpointsMatch() [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in infinite loop [FLINK-7783] Don’t always remove checkpoints in ZooKeeperCompletedCheckpointStore#recover() Since the last candidate was cancelled only due to incorrect binaries in the source artifacts, I think we can also have a shorter voting period for RC2. Please test the release and vote for the release candidate before Thursday (March 15th), 7pm CET. It is adopted by majority approval, with at least 3 PMC affirmative votes. Please let me know if you disagree with the shortened voting time. Thanks, Gordon [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? projectId=12315522&version=12341142 [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc2/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1151 [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a= shortlog;h=refs/heads/release-1.3.3-rc2 [6] https://github.com/apache/flink-web/pull/104 [7] - 90559b5413455d9d0f2b61c389a60e26e5c87800 [hotfix] Properly delete temp flink dir in create_source_release.sh - 99c0353a34c09af5bedb73f525f691dd7e78fcdd [hotfix] Ensure pristine release in tools/releasing/create_source_release.sh - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint.checkpointsMatch() - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in infinite loop - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always remove checkpoints in ZooKeeperCompletedCheckpointStore#recover()
Re: [VOTE] Release 1.3.3, release candidate #2
+1 - built and run tests locally - run some training exercises locally - looked over the commits I found one minor thing that should be mentioned. It seems that there is one unintended commit in the release: 2455df962c76d20f1c07a57d6ed0118d1d1a067c [FLINK-7283][python] fix PythonPlanBinderTest issues with python paths It only affects a test so I think we should not cancel the release because of that but update the release notes and the Jira issue. What do you think? Regards, Timo Am 15.03.18 um 10:10 schrieb Chesnay Schepler: +1 * verified that compilation works * ran a few test examples against a local cluster * src artifact contains no binaries * verified signatures On 14.03.2018 21:49, Ted Yu wrote: +1 Ran the following command - passed: mvn clean package -Pjdk8 On Wed, Mar 14, 2018 at 3:26 AM, Tzu-Li (Gordon) Tai wrote: Hi everyone, Please review and vote on release candidate #2 for Flink 1.3.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code branch “release-1.3.3-rc2” [5], * website pull request listing the new release [6]. * A complete list of all new commits in release-1.3.3-rc2, since release-1.3.2 [7] This release candidate contains fixes for only the following issues: [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint. checkpointsMatch() [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in infinite loop [FLINK-7783] Don’t always remove checkpoints in ZooKeeperCompletedCheckpointStore#recover() Since the last candidate was cancelled only due to incorrect binaries in the source artifacts, I think we can also have a shorter voting period for RC2. Please test the release and vote for the release candidate before Thursday (March 15th), 7pm CET. It is adopted by majority approval, with at least 3 PMC affirmative votes. Please let me know if you disagree with the shortened voting time. Thanks, Gordon [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? projectId=12315522&version=12341142 [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc2/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1151 [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a= shortlog;h=refs/heads/release-1.3.3-rc2 [6] https://github.com/apache/flink-web/pull/104 [7] - 90559b5413455d9d0f2b61c389a60e26e5c87800 [hotfix] Properly delete temp flink dir in create_source_release.sh - 99c0353a34c09af5bedb73f525f691dd7e78fcdd [hotfix] Ensure pristine release in tools/releasing/create_source_release.sh - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint.checkpointsMatch() - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in infinite loop - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always remove checkpoints in ZooKeeperCompletedCheckpointStore#recover()
Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0
+1 for marking bugs as fixed 1.5.0 only On Thu, Mar 15, 2018 at 11:09 AM, Ted Yu wrote: > +1 on marking bugs as fixed for 1.5.0 only. > Original message From: Piotr Nowojski < > pi...@data-artisans.com> Date: 3/15/18 12:48 AM (GMT-08:00) To: > dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as > resolved in 1.5.0 as resolved in 1.6.0 > Same as Chesnay > > +1 for marking bugs as fixed 1.5.0 only > > > On 15 Mar 2018, at 07:57, Chesnay Schepler wrote: > > > > +1 to mark bugs as fixed in 1.5.0 only. > > > > On 15.03.2018 01:40, Aljoscha Krettek wrote: > >> Hi, > >> > >> We currently have some issues that are marked as resolved for both > 1.5.0 and 1.6.0 [1]. The reason is that we have the release-1.5 branch and > the master branch, which will eventually become the branch for 1.6.0. > >> > >> I think this can lead to confusion because the release notes are > created based on that data. Say, we fix a bug "foo" after we created the > release-1.5 branch. Now we will have "[FLINK-] Fixed foo" in the > release notes for 1.5.0 and 1.6.0. We basically start our Flink 1.6.0 > release notes with around 50 issues that were never bugs in 1.6.0 because > they were fixed in 1.5.0. Plus, having "[FLINK-] Fixed foo" in the > 1.6.0 release notes indicates that "foo" was actually a bug in 1.5.0 > (because we now had to fix it), but it wasn't. > >> > >> I would propose to remove fixVersion 1.6.0 from all issues that have > 1.5.0 as fixVersion. What do you think? > >> > >> On a side note: a bug that is fixed in 1.5.1 should be marked as fixed > for 1.6.0 separately, because 1.6.0 is not a direct successor to 1.5.1. > >> > >> Best, > >> Aljoscha > >> > >> [1] https://issues.apache.org/jira/issues/?jql=project%20% > 3D%20FLINK%20and%20fixVersion%20%3D%201.6.0%20and%20resolution%20!%3D% > 20unresolved > > > > > >
Re: Flip 6 mesos support
Hi Renjie, thanks for the pointer with the YARNSessionFIFOSecuredITCase. You're right that we should fix this test. There is FLINK-8562 which seems to address the problem. Will take a look. Additionally, we want to test Kerberos authentication explicitly as part of the release testing for Flink 1.5. I will shortly send around a mail where I will lay out the ongoing testing efforts and where more is needed. Cheers, Till On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu wrote: > Thanks for the clarification > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 wrote: > > > Hi Renjie, > > if I am not misunderstand, you just need to start the cluster as normal > as > > before. The dispatcher and resourcemanager are spawned by > ClusterEntryPoint > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > YarnSessionClusterEntrypoint), and the TM are spawned by ResourceManager > > lazily (ResourceManager will setup TM according to the submitted job) or > > spawned by the setup script (you can have a look at start-cluster.sh). > > > > > > Best Regards, > > Sihua Zhou > > > > > > 发自网易邮箱大师 > > > > > > On 03/15/2018 10:14，Renjie Liu wrote： > > Hi, Till: > > In fact I'm asking how to deploy other components such as dispatcher, > etc. > > > > Till Rohrmann 于 2018年3月15日周四 上午12:17写道： > > > > Hi Renjie, > > > > in the current master and release-1.5 branch flip-6 is activated by > > default. If you want to turn it off you have to add `mode: old` to your > > flink-conf.yaml. I'm really happy that you want to test it out :-) > > > > Cheers, > > Till > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu > > wrote: > > > > Hi Till: > > Is there any doc on deploying flink in flip6 mode? We want to help > > testing > > it. > > > > Till Rohrmann 于 2018年3月14日周三 下午7:08写道： > > > > Hi Renjie, > > > > in order to make Mesos work, we only needed to implement a Mesos > > specific > > ResourceManager. Look at MesosResourceManager for more details. As > > dispatcher, we use the StandaloneDispatcher which is spawned by > > the MesosSessionClusterEntrypoint. > > > > Cheers, > > Till > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu > > wrote: > > > > Hi all: > > I'm reading the source code and it seems that flip6 does not support > > mesos? > > According to the design, client send job graph to dispatcher and > > dispatcher > > spawn job mananger and resource manager for job execution. But I > > can't > > find > > dispatcher implementation for mesos. > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > -- > Liu, Renjie > Software Engineer, MVAD >
[jira] [Created] (FLINK-8953) Resolve unresolved field references in FieldComputer expressions
Timo Walther created FLINK-8953:
---
Summary: Resolve unresolved field references in FieldComputer
expressions
Key: FLINK-8953
URL: https://issues.apache.org/jira/browse/FLINK-8953
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Reporter: Timo Walther
When implementing the {{FieldComputer.getExpression}} method, it is not
possible to use API classes but only internal expression case classes.
It would be great to also define timestamp extractors like:
{code}
def getExpression(fieldAccesses: Array[ResolvedFieldReference]): Expression =
{
// 'x.cast(Types.LONG)
// ExpressionParser.parseExpression("x.cast(LONG)")
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8952) Support setting the parallelism of individual operators of a query
Fabian Hueske created FLINK-8952:
Summary: Support setting the parallelism of individual operators
of a query
Key: FLINK-8952
URL: https://issues.apache.org/jira/browse/FLINK-8952
Project: Flink
Issue Type: New Feature
Components: Table API & SQL
Reporter: Fabian Hueske
Right now it is not possible to set the parallelism of individual operators
that were generated for a SQL or Table API query.
We could expose the optimized plan before it is translated to a {{DataStream}}
or {{DataSet}} program to annotate operators with parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8951) Support OVER windows PARTITION BY (rounded) timestamp
Fabian Hueske created FLINK-8951:
Summary: Support OVER windows PARTITION BY (rounded) timestamp
Key: FLINK-8951
URL: https://issues.apache.org/jira/browse/FLINK-8951
Project: Flink
Issue Type: New Feature
Components: Table API & SQL
Reporter: Fabian Hueske
There are a few interesting use cases that can be addressed by queries that
follow the following pattern
{code:sql}
SELECT sensorId COUNT(*) OVER (PARTITION BY CEIL(rowtime TO HOUR) ORDER BY temp
ROWS BETWEEN UNBOUNDED preceding AND CURRENT ROW) FROM sensors
{code}
Such queries can be used to compute rolling cascading (tumbling) windows with
aggregates that are reset in regular intervals. This can be useful for TOP-K
per minute/hour/day queries.
Right now, such {{OVER}} windows are not supported, because we require that the
{{ORDER BY}} clause is defined on a timestamp (time indicator) attribute. In
order to support this kind of queries, we would require that the {{PARTITION
BY}} clause contains a timestamp (time indicator) attribute or a function that
is defined on it and which is monotonicity preserving. Once the optimizer
identifies this case, it could translate the query into a special
time-partitioned OVER window operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8950) "Materialize" Tables to avoid recomputation.
Fabian Hueske created FLINK-8950:
Summary: "Materialize" Tables to avoid recomputation.
Key: FLINK-8950
URL: https://issues.apache.org/jira/browse/FLINK-8950
Project: Flink
Issue Type: New Feature
Components: Table API & SQL
Affects Versions: 1.5.0
Reporter: Fabian Hueske
Currently, {{Table}} objects of the Table API / SQL are treated like virtual
views, i.e., all relational operators that have been applied on them are
recorded and translated when a {{Table}} is emitted to a {{TableSink}} or
converted into a {{DataSet}} or {{DataStream}}.
In case a {{Table}} is accessed twice, the (sub-)query that it represents is
translated twice into a {{DataSet}} or {{DataStream}} program and hence also
executed twice which is inefficient. Currently, the only way to avoid this is
to convert the {{Table}} into a {{DataSet}} or {{DataStream}}, which will cause
the optimizer to generate a plan and register it back as a new {{Table}}.
We should offer a method to internally "materialize" a {{Table}} object, i.e.,
to optimize, generate a plan, and register the plan as an internal table. All
queries / operations that are evaluated on the materialized {{Table}} will
start from the same {{DataSet}} or {{DataStream}} such that it is not computed
multiple times.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0
+1 on marking bugs as fixed for 1.5.0 only. Original message From: Piotr Nowojski Date: 3/15/18 12:48 AM (GMT-08:00) To: dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0 Same as Chesnay +1 for marking bugs as fixed 1.5.0 only > On 15 Mar 2018, at 07:57, Chesnay Schepler wrote: > > +1 to mark bugs as fixed in 1.5.0 only. > > On 15.03.2018 01:40, Aljoscha Krettek wrote: >> Hi, >> >> We currently have some issues that are marked as resolved for both 1.5.0 and >> 1.6.0 [1]. The reason is that we have the release-1.5 branch and the master >> branch, which will eventually become the branch for 1.6.0. >> >> I think this can lead to confusion because the release notes are created >> based on that data. Say, we fix a bug "foo" after we created the release-1.5 >> branch. Now we will have "[FLINK-] Fixed foo" in the release notes for >> 1.5.0 and 1.6.0. We basically start our Flink 1.6.0 release notes with >> around 50 issues that were never bugs in 1.6.0 because they were fixed in >> 1.5.0. Plus, having "[FLINK-] Fixed foo" in the 1.6.0 release notes >> indicates that "foo" was actually a bug in 1.5.0 (because we now had to fix >> it), but it wasn't. >> >> I would propose to remove fixVersion 1.6.0 from all issues that have 1.5.0 >> as fixVersion. What do you think? >> >> On a side note: a bug that is fixed in 1.5.1 should be marked as fixed for >> 1.6.0 separately, because 1.6.0 is not a direct successor to 1.5.1. >> >> Best, >> Aljoscha >> >> [1] >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20and%20fixVersion%20%3D%201.6.0%20and%20resolution%20!%3D%20unresolved > >
[jira] [Created] (FLINK-8949) Rest API failure with long URL
Truong Duc Kien created FLINK-8949:
--
Summary: Rest API failure with long URL
Key: FLINK-8949
URL: https://issues.apache.org/jira/browse/FLINK-8949
Project: Flink
Issue Type: Bug
Components: REST, Webfrontend
Affects Versions: 1.4.2
Reporter: Truong Duc Kien
When you have jobs with high parallelism, the URL for a REST request can get
very long. When the URL is longer than 4096 bytes, the REST API will return
error
{{Failure: 404 Not Found}}
This can easily be seen in the Web UI, when Flink queries for the watermark
using the REST API:
{{GET
/jobs/:jobId/vertices/:vertexId/metrics?get=0.currentLowWatermark,1.currentLowWatermark,2.currentLo...}}
The request will fail with more than 170 subtasks and the watermark will not be
displayed in the Web UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release 1.3.3, release candidate #2
+1 * verified that compilation works * ran a few test examples against a local cluster * src artifact contains no binaries * verified signatures On 14.03.2018 21:49, Ted Yu wrote: +1 Ran the following command - passed: mvn clean package -Pjdk8 On Wed, Mar 14, 2018 at 3:26 AM, Tzu-Li (Gordon) Tai wrote: Hi everyone, Please review and vote on release candidate #2 for Flink 1.3.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code branch “release-1.3.3-rc2” [5], * website pull request listing the new release [6]. * A complete list of all new commits in release-1.3.3-rc2, since release-1.3.2 [7] This release candidate contains fixes for only the following issues: [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint. checkpointsMatch() [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in infinite loop [FLINK-7783] Don’t always remove checkpoints in ZooKeeperCompletedCheckpointStore#recover() Since the last candidate was cancelled only due to incorrect binaries in the source artifacts, I think we can also have a shorter voting period for RC2. Please test the release and vote for the release candidate before Thursday (March 15th), 7pm CET. It is adopted by majority approval, with at least 3 PMC affirmative votes. Please let me know if you disagree with the shortened voting time. Thanks, Gordon [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? projectId=12315522&version=12341142 [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc2/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1151 [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a= shortlog;h=refs/heads/release-1.3.3-rc2 [6] https://github.com/apache/flink-web/pull/104 [7] - 90559b5413455d9d0f2b61c389a60e26e5c87800 [hotfix] Properly delete temp flink dir in create_source_release.sh - 99c0353a34c09af5bedb73f525f691dd7e78fcdd [hotfix] Ensure pristine release in tools/releasing/create_source_release.sh - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint.checkpointsMatch() - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in infinite loop - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always remove checkpoints in ZooKeeperCompletedCheckpointStore#recover()
Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0
Same as Chesnay +1 for marking bugs as fixed 1.5.0 only > On 15 Mar 2018, at 07:57, Chesnay Schepler wrote: > > +1 to mark bugs as fixed in 1.5.0 only. > > On 15.03.2018 01:40, Aljoscha Krettek wrote: >> Hi, >> >> We currently have some issues that are marked as resolved for both 1.5.0 and >> 1.6.0 [1]. The reason is that we have the release-1.5 branch and the master >> branch, which will eventually become the branch for 1.6.0. >> >> I think this can lead to confusion because the release notes are created >> based on that data. Say, we fix a bug "foo" after we created the release-1.5 >> branch. Now we will have "[FLINK-] Fixed foo" in the release notes for >> 1.5.0 and 1.6.0. We basically start our Flink 1.6.0 release notes with >> around 50 issues that were never bugs in 1.6.0 because they were fixed in >> 1.5.0. Plus, having "[FLINK-] Fixed foo" in the 1.6.0 release notes >> indicates that "foo" was actually a bug in 1.5.0 (because we now had to fix >> it), but it wasn't. >> >> I would propose to remove fixVersion 1.6.0 from all issues that have 1.5.0 >> as fixVersion. What do you think? >> >> On a side note: a bug that is fixed in 1.5.1 should be marked as fixed for >> 1.6.0 separately, because 1.6.0 is not a direct successor to 1.5.1. >> >> Best, >> Aljoscha >> >> [1] >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20and%20fixVersion%20%3D%201.6.0%20and%20resolution%20!%3D%20unresolved > >
[jira] [Created] (FLINK-8948) RescalingITCase fails on Travis
Chesnay Schepler created FLINK-8948:
---
Summary: RescalingITCase fails on Travis
Key: FLINK-8948
URL: https://issues.apache.org/jira/browse/FLINK-8948
Project: Flink
Issue Type: Improvement
Components: State Backends, Checkpointing, Tests, Travis
Affects Versions: 1.5.0
Reporter: Chesnay Schepler
https://travis-ci.org/apache/flink/jobs/353468272
{code}
testSavepointRescalingInKeyedStateDerivedMaxParallelism[0](org.apache.flink.test.checkpointing.RescalingITCase)
Time elapsed: 1.858 sec <<< ERROR!
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:891)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalStateException: null
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
at
org.apache.flink.runtime.io.network.buffer.BufferBuilder.finish(BufferBuilder.java:105)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.closeBufferBuilder(RecordWriter.java:218)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.clearBuffers(RecordWriter.java:178)
at
org.apache.flink.streaming.runtime.io.StreamRecordWriter.close(StreamRecordWriter.java:99)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.close(RecordWriterOutput.java:161)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.releaseOutputs(OperatorChain.java:239)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:402)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8947) Timeout handler issue
dhiraj prajapati created FLINK-8947:
---
Summary: Timeout handler issue
Key: FLINK-8947
URL: https://issues.apache.org/jira/browse/FLINK-8947
Project: Flink
Issue Type: Bug
Components: CEP
Affects Versions: 1.4.2
Reporter: dhiraj prajapati
The issue is same as FLINK-5753
I am using Event time and have used watermark interval. Still, I have observed
the timeout executes only after the next event
if: first event appears, second event not appear in the stream
and *no new events appear in a stream*, timeout handler is not executed.
Expected result: timeout handler should be executed in case if there are no new
events in a stream
My code snippet:
DataStream dataStream = env.socketTextStream("localhost", 1212);
dataStream.getExecutionConfig().setAutoWatermarkInterval(100L);
dataStream = dataStream.assignTimestampsAndWatermarks(new
BoundedOutOfOrdernessTimestampExtractor(
Time.seconds(0)) {
private static final long serialVersionUID = 4969170359023055566L;
@Override
public long extractTimestamp(JSONObject event) {
return System.currentTimeMillis();
}
});
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
