[jira] [Created] (FLINK-25339) Moving to hadoop-free flink runtime.
David Morávek created FLINK-25339: - Summary: Moving to hadoop-free flink runtime. Key: FLINK-25339 URL: https://issues.apache.org/jira/browse/FLINK-25339 Project: Flink Issue Type: Technical Debt Components: Runtime / Coordination Reporter: David Morávek Only remaining reason for having hadoop dependencies (even though these are _provided_) in `flink-runtime` is the Security / Kerberos setup, which is already hidden behind a service loader. This should be fairly straightforward to move into the separate module. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25338) Improvement of connection from TM to JM in session cluster
Shammon created FLINK-25338: --- Summary: Improvement of connection from TM to JM in session cluster Key: FLINK-25338 URL: https://issues.apache.org/jira/browse/FLINK-25338 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Affects Versions: 1.14.2, 1.13.5, 1.12.7 Reporter: Shammon When taskmanager receives slot request from resourcemanager for the specify job, it will connect to the jobmaster with given job address. Taskmanager register itself, monitor the heartbeat of job and update task's state by this connection. There's no need to create connections in one taskmanager for each job, and when the taskmanager is busy, it will increase the latency of job. One idea is that taskmanager manages the connection to `Dispatcher`, sends events such as heartbeat, state update to `Dispatcher`, and `Dispatcher` tell the local `JobMaster`. The main problem is that `Dispatcher` is an actor and can only be executed in one thread, it may be the performance bottleneck for deserialize event. The other idea it to create a netty service in `SessionClusterEntrypoint`, it can receive and deserialize events from taskmanagers in a threadpool, and send the event to the `Dispatcher` or `JobMaster`. Taskmanagers manager the connection to the netty service when it start. Thus a service can also receive the result of a job from taskmanager later. [~xtsong] What do you think? THX -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction
Hi Fabian, Very thanks for the update! I think the latest version in general looks good from my side and I think using separate feature interface would be much more easy to understand and extend in the future. I have some pending issues on the details though: 1. The first one is if we could support end-to-end exactly-once with post-committing topology in the batch mode ? Since for the batch mode, currently we could only commit all the transactions after the whole job is finished, otherwise if there are JM failover or the writer / committer get restarted due to indeterminstic (A -> [B1, B2], A, B1 have finished and B2 failed, if -> is rebalance / random / rescale, all of A, B1, B2 would restarted), there might be repeat records. Previously one possible thought is to move committer and global committer to the operator coordinator, but if it is a topology, we might need some other kind of solutions? 2. I also want to have a dobule confirmation with the compatibility: since the old sink is also named with Sink, do we want to put the Sink v2 in a new package ? Besides, since we might want to keep only have one `sinkTo(Sink sink)` , perhaps we also need to make the Sink v1 to be a subclass of Sink v2 and extends the stateful and two-phase-commit sinks, right? 3. I'd like also have a confirmation on ours thoughts with the `DataStreamSink` returned by the sinkTo method: The main issue is how do we implement the method like `setParallelism` or `setMaxParallelism` since now the sink would be translated to multiple transformations? perhaps we could make it the default values for all the transformations for the sink? A related issue would be for iceberg sink, I think it would need to have only one committer to avoid the competition of the optimistic locks (which would cause performance degradation), then it might need to have N writers with 1 committers, to build such topology, perhaps we might need to add new methods to specify the parallelism of the writers and committers separately? Best, Yun --Original Mail -- Sender:Fabian Paul Send Date:Mon Dec 13 23:59:43 2021 Recipients:dev Subject:Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction Hi all, After a lot of discussions with different, we received very fruitful feedback and reworked the ideas behind this FLIP. Initially, we had the impression that the compaction problem is solvable by a single topology that we can reuse across different sinks. We now have a better understanding that different external systems require different compaction mechanism i.e. Hive requires compaction before finally registering the file in the metastore or Iceberg compacts the files after they have been registered and just lazily compacts them. Considering all these different views we came up with a design that builds upon what @guowei@gmail.com and @yungao...@aliyun.com have proposed at the beginning. We allow inserting custom topologies before and after the SinkWriters and Committers. Furthermore, we do not see it as a downside. The Sink interfaces that will expose the DataStream to the user reside in flink-streaming-java in contrast to the basic Sink interfaces that reside fin flink-core deem it to be only used by expert users. Moreover, we also wanted to remove the global committer from the unified Sink interfaces and replace it with a custom post-commit topology. Unfortunately, we cannot do it without breaking the Sink interface since the GlobalCommittables are part of the parameterized Sink interface. Thus, we propose building a new Sink V2 interface consisting of composable interfaces that do not offer the GlobalCommitter anymore. We will implement a utility to extend a Sink with post topology that mimics the behavior of the GlobalCommitter. The new Sink V2 provides the same sort of methods as the Sink V1 interface, so a migration of sinks that do not use the GlobalCommitter should be very easy. We plan to keep the existing Sink V1 interfaces to not break externally built sinks. As part of this FLIP, we migrate all the connectors inside of the main repository to the new Sink V2 API. The FLIP document is also updated and includes the proposed changes. Looking forward to your feedback, Fabian https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction On Thu, Dec 2, 2021 at 10:15 AM Roman Khachatryan wrote: > > Thanks for clarifying (I was initially confused by merging state files > rather than output files). > > > At some point, Flink will definitely have some WAL adapter that can turn > > any sink into an exactly-once sink (with some caveats). For now, we keep > > that as an orthogonal solution as it has a rather high price (bursty > > workload with high latency). Ideally, we can keep the compaction > > asynchronously... > > Yes, that wou
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
Thanks for running and release Chesnay I hope so too. On Wed, Dec 15, 2021 at 9:56 PM Chesnay Schepler wrote: > The vote duration has passed and we have approved the releases. > > Binding votes: > * Till > * Yu > * Stephan > > I will now finalize the release. Let's hope this time no new CVE pops up... > > On 15/12/2021 03:55, Chesnay Schepler wrote: > > Hi everyone, > > > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and > > 1.14 to address CVE-2021-44228/CVE-2021-45046. > > It covers all 4 releases as they contain the same changes (upgrading > > Log4j to 2.16.0) and were prepared simultaneously by the same person. > > (Hence, if something is broken, it likely applies to all releases) > > > > Note: 1.11/1.12 are still missing the Python Mac releases. > > > > > > Please review and vote on the release candidate #1 for the versions > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > > [ ] +1, Approve the releases > > [ ] -1, Do not approve the releases (please provide specific comments) > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source releases and binary convenience releases > > to be deployed to dist.apache.org [2], which are signed with the key > > with fingerprint C2EED7B111D464BA [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tags [5], > > * website pull request listing the new releases and adding > > announcement blog post [6]. > > > > The vote will be open for at least 24 hours. The minimum vote time has > > been shortened as the changes are minimal and the matter is urgent. > > It is adopted by majority approval, with at least 3 PMC affirmative > > votes. > > > > Thanks, > > Chesnay > > > > [1] > > 1.11: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 > > 1.12: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 > > 1.13: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 > > 1.14: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 > > [2] > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > 1.11: > > https://repository.apache.org/content/repositories/orgapacheflink-1460 > > 1.12: > > https://repository.apache.org/content/repositories/orgapacheflink-1462 > > 1.13: > > https://repository.apache.org/content/repositories/orgapacheflink-1459 > > 1.14: > > https://repository.apache.org/content/repositories/orgapacheflink-1461 > > [5] > > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 > > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 > > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 > > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1 > > [6] https://github.com/apache/flink-web/pull/489 > > > > > > -- Israel Ekpo Lead Instructor, IzzyAcademy.com https://izzyacademy.com/
[jira] [Created] (FLINK-25337) Check whether the target table is valid when SqlToOperationConverter.convertSqlInsert
vim-wang created FLINK-25337: Summary: Check whether the target table is valid when SqlToOperationConverter.convertSqlInsert Key: FLINK-25337 URL: https://issues.apache.org/jira/browse/FLINK-25337 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Affects Versions: 1.14.0 Reporter: vim-wang when I execute insert sql like "insert into t1 select ...", If the t1 is not defined,sql will not throw an exception after SqlToOperationConverter.convertSqlInsert(), I think this is unreasonable, why not use catalogManager to check whether the target table is valid? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25336) Kafka connector compatible problem in Flink sql
Yuan Zhu created FLINK-25336: Summary: Kafka connector compatible problem in Flink sql Key: FLINK-25336 URL: https://issues.apache.org/jira/browse/FLINK-25336 Project: Flink Issue Type: Bug Components: Connectors / Kafka Affects Versions: 1.14.0 Environment: Flink 1.14.0 Kafka 0.10.2.1 Reporter: Yuan Zhu Attachments: log.jpg When I use sql to query kafka table, like {code:java} create table `kfk` ( user_id VARCHAR ) with ( 'connector' = 'kafka', 'topic' = 'test', 'properties.bootstrap.servers' = 'localhost:9092', 'format' = 'json', 'scan.startup.mode' = 'timestamp', 'scan.startup.timestamp-millis' = '163941120', 'properties.group.id' = 'test' ); CREATE TABLE print_table (user_id varchar) WITH ('connector' = 'print'); insert into print_table select user_id from kfk;{code} It will encounter an exception: org.apache.kafka.common.errors.UnsupportedVersionException: MetadataRequest versions older than 4 don't support the allowAutoTopicCreation field !log.jpg! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25335) Improvoment of task deployment by enable source split async enumerate
KevinyhZou created FLINK-25335: -- Summary: Improvoment of task deployment by enable source split async enumerate Key: FLINK-25335 URL: https://issues.apache.org/jira/browse/FLINK-25335 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Affects Versions: 1.12.1 Reporter: KevinyhZou When submit olap query by flink client to Flink Session Cluster, the JobMaster will start scheduling and enumerate the hive source split by `HiveSourceFileEnumerator`, and then deploy the query task and execute it. if the source table has a lot of partition and the partition file is big, the source split enumerate will cost a lot of time, which would block the task deployment & execution for a long time, and the dashboard can not appear JobMaster should async enumerate the hive split, and meanwhile deploy the query task and execute it. when the deployment is finished, source operator fetch split and read data, and the split enumeration is also going on. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
The vote duration has passed and we have approved the releases. Binding votes: * Till * Yu * Stephan I will now finalize the release. Let's hope this time no new CVE pops up... On 15/12/2021 03:55, Chesnay Schepler wrote: Hi everyone, This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 1.14 to address CVE-2021-44228/CVE-2021-45046. It covers all 4 releases as they contain the same changes (upgrading Log4j to 2.16.0) and were prepared simultaneously by the same person. (Hence, if something is broken, it likely applies to all releases) Note: 1.11/1.12 are still missing the Python Mac releases. Please review and vote on the release candidate #1 for the versions 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: [ ] +1, Approve the releases [ ] -1, Do not approve the releases (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source releases and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint C2EED7B111D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tags [5], * website pull request listing the new releases and adding announcement blog post [6]. The vote will be open for at least 24 hours. The minimum vote time has been shortened as the changes are minimal and the matter is urgent. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Chesnay [1] 1.11: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 1.12: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 1.13: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 1.14: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 [2] 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 1.11: https://repository.apache.org/content/repositories/orgapacheflink-1460 1.12: https://repository.apache.org/content/repositories/orgapacheflink-1462 1.13: https://repository.apache.org/content/repositories/orgapacheflink-1459 1.14: https://repository.apache.org/content/repositories/orgapacheflink-1461 [5] 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1 [6] https://github.com/apache/flink-web/pull/489
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
The pull request for Flink-web looks good to me +1 (non binding) from me On Wed, Dec 15, 2021 at 8:19 PM Yun Gao wrote: > > Ah actually we may have to merge another commit into the release > > branches for that to be the case for 1.11/1.12. > > > Anyway, I still wouldn't block the release on that. > > Yes, it should only be cases for 1.11 / 1.12~ With a double check the > other two versions contains the doc modification. Very thanks for the > explanation~ > > Best, > Yun > > > > -- > From:Yun Gao > Send Time:2021 Dec. 16 (Thu.) 09:14 > To:dev ; Yun Gao ; > Stephan Ewen > Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate > #1 > > Hi Chesnay, > > Very thanks for the explanation! > > Then +1 (non-binding) > > * Verifies the binary distributions could run successfully with the > example jobs. > * Verifies the new tags only contains the log4j version bump PRs. > * The latest blog post and release notes looks good to me. > > Very thanks for driving the emergency releases! > > Best, > Yun > > > > -- > From:Chesnay Schepler > Send Time:2021 Dec. 16 (Thu.) 09:03 > To:dev ; Yun Gao ; > Stephan Ewen > Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate > #1 > > That's not a problem as far as I'm concerned, because the documentation > of these releases is never actually published anywhere. > The documentation at nightlies.apache.org will reference 2.16.0. > > On 16/12/2021 02:01, Yun Gao wrote: > > Hi, > > > > May I have a double confirmation that it seems we still have log4j > version written as > > 2.15.0 in dev/project-configuration.md in the commit corresponding to > release tags, > > which seems to be not consistent with the PR in the github, is it > expected and would it > > have influence? > > > > Best, > > Yun > > > > > > > > --Original Mail -- > > Sender:Stephan Ewen > > Send Date:Thu Dec 16 08:34:10 2021 > > Recipients:dev > > Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release > candidate #1 > > +1 (binding) > > > > > > > > - Verified commit history, looks good > > > > => stumbled over the changes in the "create_release_branch.sh ", > > > > which are present in each release commit. [1] > > > > => agree that these are not an issue, because this is an out-of-band > > > > release > > > > - Release notes for 1.14.2 are off, contain incorrect entry > "FLINK-25222: > > > > Remove NetworkFailureProxy used for Kafka connector tests" > > > > - Checked that released binaries and jars reference correct Scala > versions > > > > - Ran streaming examples against binary releases for 1.12.7, 1.13.5, > > > > 1.14.2. Execution logs look correct. > > > > - Other checks (licenses, no binaries) carry over from previous > releases > > > > > > > > [1] > > > > > https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074 > > > > > > > > > > > > On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman wrote: > > > > > > > >> +1 (non-binding) > >> - Checked diff of all versions and verified dep upgrade > >> - Verified checksum and signatures > >> - Built 1.14 from source > >> - checked blog post > >> Seth > >> On Wed, Dec 15, 2021 at 10:22 AM Yu Li wrote: > >>> +1 > >>> * Verified checksums and signatures > >>> * Reviewed website PR > >>> - Minor: left a comment to mention CVE-2021-45046 > >>> * Checked and confirmed new tags only contain log4j version bump > >>> * Checked release notes and found no issues > >>> - I've moved FLINK-25317 to 1.14.3 > >>> Thanks for driving these releases Chesnay! > >>> Best Regards, > >>> Yu > >>> On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler > >> wrote: > FYI; the publication of the python release for 1.11/1.12 will be > >> delayed > because we hit the project size limit on pypi again, and increasing > >> that > limit may take a while. > On the positive side, this gives us more time to fix the mac builds. > On 15/12/2021 03:55, Chesnay Schepler wrote: > > Hi everyone, > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 > >> and > > 1.14 to address CVE-2021-44228/CVE-2021-45046. > > It covers all 4 releases as they contain the same changes (upgrading > > Log4j to 2.16.0) and were prepared simultaneously by the same person. > > (Hence, if something is broken, it likely applies to all releases) > > Note: 1.11/1.12 are still missing the Python Mac releases. > > Please review and vote on the release candidate #1 for the versions > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > > [ ] +1, Approve the releases > > [ ] -1, Do not approve the releases (please provide specific > >> comments) > > The complete staging area is available for your review, which > >> includes: > > * JIRA release no
Re: [DISCUSS] FLIP-198: Working directory for Flink processes
I am afraid creating a dedicated StatefulSet for each TaskManager is too expensive and using a shared StatefulSet for all the TaskManagers is not flexible enough. Maybe setting a proper restart policy for TaskManager pods could benefit from this FLIP. But we might need to tackle some other issues, e.g. duplicated registration, etc. All in all, this is out of the scope of this FLIP. I agree we could leave it in the future FLIPs. I have no more concerns. +1 Best, Yang Till Rohrmann 于2021年12月15日周三 19:06写道: > This is true. But this is not a new problem and I think that Flink should > be susceptible to this problem already. One solution for this concrete case > could be that the BlobServer stores some checksums and validates the file > before serving it to the TM. > > Cheers, > Till > > On Wed, Dec 15, 2021 at 11:59 AM Chesnay Schepler > wrote: > > > The issue with corrupted files is that some of them aren't read by the > > component that stores them. > > For example, a file can be corrupted in the blob server of the JM, but > > that it is corrupted will only be noticed by the TaskExecutor. > > > > On 15/12/2021 11:36, Till Rohrmann wrote: > > > Thanks everyone for your feedback. Let me try to address it by grouping > > > some of the individual comments: > > > > > > ### Will this feature work for native Yarn and K8s deployments? > > > > > > The working directory is an optional feature that can be used to > recover > > > additional information. You can think of it like a cache. If the > working > > > directory is there, then Flink can do certain things a bit faster but > in > > > the worst case it will have to retrieve the required information from > the > > > JobManager or persistent storage. > > > > > > In order to make it work with native Yarn and K8s, we would have to > > change > > > these modes slightly. First of all, we would have to be able to map > > working > > > directories to processes and then set a deterministic resource ids for > > the > > > processes. For K8s this could be easily achievable by using a > StatefulSet > > > as the deployment mechanism for TaskExecutors. For Yarn, we probably > > would > > > have to remember the prior locations of a process. Both things are > > > potential follow ups that I don't want to tackle in this FLIP. > > > > > > If one of the modes configures the working directory to be on a full or > > > broken disk, then the process will fail. I think this is not all that > > > different from the current state where some things in Flink will fail > if > > > they picked the wrong/full temporary directory (e.g. blob storage > > > directory). > > > > > > ### Cleanup > > > > > > The working directory will be cleaned up if the Flink process is > > gracefully > > > shut down. This means that the JobManager process will clean it up if > it > > > runs in application mode and the job is terminated. SIGTERM and SIGKILL > > > signals will be treated as an ungraceful shutdown and therefore they > > won't > > > clean up the working directory. This means that we probably also need a > > > graceful way for shutting TaskManager processes down in the future > > because > > > right now they are in most cases killed in order to shut them down. If > > the > > > user uses the tmp directory, then any left-over working directories > will > > be > > > cleaned up with the next system restart. This is somewhat similar to > how > > > RocksDB's working directory is currently cleaned up as well. > > > > > > ### Corrupted files > > > > > > The working directory itself won't give you any guarantees. It will be > > the > > > responsibility of the component that uses the working directory to make > > > sure that it can deal with corrupted files. E.g. if the component > cannot > > > read the file, then it should delete it and fall back to the remote > > > storage/ground truth to retrieve the required information. > > > > > > I hope this could answer your questions. Let me know if you have more > > > feedback. > > > > > > Cheers, > > > Till > > > > > > On Mon, Dec 13, 2021 at 5:05 AM 刘建刚 wrote: > > > > > >> I like the idea. It can reuse the disk to do many things. Isn't it > only > > >> for inner failover? If not, the cleaning may be a problem. Also, many > > >> resource components have their own disk schedule strategy. > > >> > > >> Chesnay Schepler 于2021年12月12日周日 19:59写道: > > >> > > >>> How do you intend to handle corrupted files, in particular due to > > >>> process crashes during a write? > > >>> Will all writes to a cached directory append some suffix (e.g., > > >>> ".pending") and do a rename? > > >>> > > >>> On 10/12/2021 17:54, Till Rohrmann wrote: > > Hi everyone, > > > > I would like to start a discussion about introducing an explicit > > working > > directory for Flink processes that can be used to store information > > [1]. > > Per default this working directory will reside in the temporary > > >>> directory > > of the node Flink runs on. However, if config
[jira] [Created] (FLINK-25334) Manage and share gateways of taskmanagers between jobs in session cluster
Shammon created FLINK-25334: --- Summary: Manage and share gateways of taskmanagers between jobs in session cluster Key: FLINK-25334 URL: https://issues.apache.org/jira/browse/FLINK-25334 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Affects Versions: 1.13.3, 1.12.4, 1.14.0 Reporter: Shammon Flink session cluster launches a certain number of taskmanagers and they won't exit during the cluster running. When a job is submitted to the cluster, taskmanagers will offer slots to `JobMaster` with location info. Then `JobMaster` will get `InetAddress` from the location and connect the taskmanager. This cause some performance loss and it isn't necessary in session cluster. It's better to manager the gateways of taskmanagers in JM and share them between jobs. When a taskmanager's gateway is exist and valid, the `JobMaster` can use it directly instead of creating a new one. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
> Ah actually we may have to merge another commit into the release > branches for that to be the case for 1.11/1.12. > Anyway, I still wouldn't block the release on that. Yes, it should only be cases for 1.11 / 1.12~ With a double check the other two versions contains the doc modification. Very thanks for the explanation~ Best, Yun -- From:Yun Gao Send Time:2021 Dec. 16 (Thu.) 09:14 To:dev ; Yun Gao ; Stephan Ewen Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 Hi Chesnay, Very thanks for the explanation! Then +1 (non-binding) * Verifies the binary distributions could run successfully with the example jobs. * Verifies the new tags only contains the log4j version bump PRs. * The latest blog post and release notes looks good to me. Very thanks for driving the emergency releases! Best, Yun -- From:Chesnay Schepler Send Time:2021 Dec. 16 (Thu.) 09:03 To:dev ; Yun Gao ; Stephan Ewen Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 That's not a problem as far as I'm concerned, because the documentation of these releases is never actually published anywhere. The documentation at nightlies.apache.org will reference 2.16.0. On 16/12/2021 02:01, Yun Gao wrote: > Hi, > > May I have a double confirmation that it seems we still have log4j version > written as > 2.15.0 in dev/project-configuration.md in the commit corresponding to release > tags, > which seems to be not consistent with the PR in the github, is it expected > and would it > have influence? > > Best, > Yun > > > > --Original Mail -- > Sender:Stephan Ewen > Send Date:Thu Dec 16 08:34:10 2021 > Recipients:dev > Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 > +1 (binding) > > > > - Verified commit history, looks good > > => stumbled over the changes in the "create_release_branch.sh ", > > which are present in each release commit. [1] > > => agree that these are not an issue, because this is an out-of-band > > release > > - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222: > > Remove NetworkFailureProxy used for Kafka connector tests" > > - Checked that released binaries and jars reference correct Scala versions > > - Ran streaming examples against binary releases for 1.12.7, 1.13.5, > > 1.14.2. Execution logs look correct. > > - Other checks (licenses, no binaries) carry over from previous releases > > > > [1] > > https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074 > > > > > > On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman wrote: > > > >> +1 (non-binding) >> - Checked diff of all versions and verified dep upgrade >> - Verified checksum and signatures >> - Built 1.14 from source >> - checked blog post >> Seth >> On Wed, Dec 15, 2021 at 10:22 AM Yu Li wrote: >>> +1 >>> * Verified checksums and signatures >>> * Reviewed website PR >>> - Minor: left a comment to mention CVE-2021-45046 >>> * Checked and confirmed new tags only contain log4j version bump >>> * Checked release notes and found no issues >>> - I've moved FLINK-25317 to 1.14.3 >>> Thanks for driving these releases Chesnay! >>> Best Regards, >>> Yu >>> On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler >> wrote: FYI; the publication of the python release for 1.11/1.12 will be >> delayed because we hit the project size limit on pypi again, and increasing >> that limit may take a while. On the positive side, this gives us more time to fix the mac builds. On 15/12/2021 03:55, Chesnay Schepler wrote: > Hi everyone, > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 >> and > 1.14 to address CVE-2021-44228/CVE-2021-45046. > It covers all 4 releases as they contain the same changes (upgrading > Log4j to 2.16.0) and were prepared simultaneously by the same person. > (Hence, if something is broken, it likely applies to all releases) > Note: 1.11/1.12 are still missing the Python Mac releases. > Please review and vote on the release candidate #1 for the versions > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > [ ] +1, Approve the releases > [ ] -1, Do not approve the releases (please provide specific >> comments) > The complete staging area is available for your review, which >> includes: > * JIRA release notes [1], > * the official Apache source releases and binary convenience releases > to be deployed to dist.apache.org [2], which are signed with the key > with fingerprint C2EED7B111D464BA [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tags [5], > * website pull request listing the new releases and adding > announcement blog post [6]. > T
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
Hi Chesnay, Very thanks for the explanation! Then +1 (non-binding) * Verifies the binary distributions could run successfully with the example jobs. * Verifies the new tags only contains the log4j version bump PRs. * The latest blog post and release notes looks good to me. Very thanks for driving the emergency releases! Best, Yun -- From:Chesnay Schepler Send Time:2021 Dec. 16 (Thu.) 09:03 To:dev ; Yun Gao ; Stephan Ewen Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 That's not a problem as far as I'm concerned, because the documentation of these releases is never actually published anywhere. The documentation at nightlies.apache.org will reference 2.16.0. On 16/12/2021 02:01, Yun Gao wrote: > Hi, > > May I have a double confirmation that it seems we still have log4j version > written as > 2.15.0 in dev/project-configuration.md in the commit corresponding to release > tags, > which seems to be not consistent with the PR in the github, is it expected > and would it > have influence? > > Best, > Yun > > > > --Original Mail -- > Sender:Stephan Ewen > Send Date:Thu Dec 16 08:34:10 2021 > Recipients:dev > Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 > +1 (binding) > > > > - Verified commit history, looks good > > => stumbled over the changes in the "create_release_branch.sh ", > > which are present in each release commit. [1] > > => agree that these are not an issue, because this is an out-of-band > > release > > - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222: > > Remove NetworkFailureProxy used for Kafka connector tests" > > - Checked that released binaries and jars reference correct Scala versions > > - Ran streaming examples against binary releases for 1.12.7, 1.13.5, > > 1.14.2. Execution logs look correct. > > - Other checks (licenses, no binaries) carry over from previous releases > > > > [1] > > https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074 > > > > > > On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman wrote: > > > >> +1 (non-binding) >> - Checked diff of all versions and verified dep upgrade >> - Verified checksum and signatures >> - Built 1.14 from source >> - checked blog post >> Seth >> On Wed, Dec 15, 2021 at 10:22 AM Yu Li wrote: >>> +1 >>> * Verified checksums and signatures >>> * Reviewed website PR >>> - Minor: left a comment to mention CVE-2021-45046 >>> * Checked and confirmed new tags only contain log4j version bump >>> * Checked release notes and found no issues >>> - I've moved FLINK-25317 to 1.14.3 >>> Thanks for driving these releases Chesnay! >>> Best Regards, >>> Yu >>> On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler >> wrote: FYI; the publication of the python release for 1.11/1.12 will be >> delayed because we hit the project size limit on pypi again, and increasing >> that limit may take a while. On the positive side, this gives us more time to fix the mac builds. On 15/12/2021 03:55, Chesnay Schepler wrote: > Hi everyone, > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 >> and > 1.14 to address CVE-2021-44228/CVE-2021-45046. > It covers all 4 releases as they contain the same changes (upgrading > Log4j to 2.16.0) and were prepared simultaneously by the same person. > (Hence, if something is broken, it likely applies to all releases) > Note: 1.11/1.12 are still missing the Python Mac releases. > Please review and vote on the release candidate #1 for the versions > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > [ ] +1, Approve the releases > [ ] -1, Do not approve the releases (please provide specific >> comments) > The complete staging area is available for your review, which >> includes: > * JIRA release notes [1], > * the official Apache source releases and binary convenience releases > to be deployed to dist.apache.org [2], which are signed with the key > with fingerprint C2EED7B111D464BA [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tags [5], > * website pull request listing the new releases and adding > announcement blog post [6]. > The vote will be open for at least 24 hours. The minimum vote time >> has > been shortened as the changes are minimal and the matter is urgent. > It is adopted by majority approval, with at least 3 PMC affirmative > votes. > Thanks, > Chesnay > [1] > 1.11: >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 > 1.12: >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 > 1.13: >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=123155
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
Ah actually we may have to merge another commit into the release branches for that to be the case for 1.11/1.12. Anyway, I still wouldn't block the release on that. On 16/12/2021 02:03, Chesnay Schepler wrote: That's not a problem as far as I'm concerned, because the documentation of these releases is never actually published anywhere. The documentation at nightlies.apache.org will reference 2.16.0. On 16/12/2021 02:01, Yun Gao wrote: Hi, May I have a double confirmation that it seems we still have log4j version written as 2.15.0 in dev/project-configuration.md in the commit corresponding to release tags, which seems to be not consistent with the PR in the github, is it expected and would it have influence? Best, Yun --Original Mail -- Sender:Stephan Ewen Send Date:Thu Dec 16 08:34:10 2021 Recipients:dev Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 +1 (binding) - Verified commit history, looks good => stumbled over the changes in the "create_release_branch.sh ", which are present in each release commit. [1] => agree that these are not an issue, because this is an out-of-band release - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222: Remove NetworkFailureProxy used for Kafka connector tests" - Checked that released binaries and jars reference correct Scala versions - Ran streaming examples against binary releases for 1.12.7, 1.13.5, 1.14.2. Execution logs look correct. - Other checks (licenses, no binaries) carry over from previous releases [1] https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074 On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman wrote: +1 (non-binding) - Checked diff of all versions and verified dep upgrade - Verified checksum and signatures - Built 1.14 from source - checked blog post Seth On Wed, Dec 15, 2021 at 10:22 AM Yu Li wrote: +1 * Verified checksums and signatures * Reviewed website PR - Minor: left a comment to mention CVE-2021-45046 * Checked and confirmed new tags only contain log4j version bump * Checked release notes and found no issues - I've moved FLINK-25317 to 1.14.3 Thanks for driving these releases Chesnay! Best Regards, Yu On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler wrote: FYI; the publication of the python release for 1.11/1.12 will be delayed because we hit the project size limit on pypi again, and increasing that limit may take a while. On the positive side, this gives us more time to fix the mac builds. On 15/12/2021 03:55, Chesnay Schepler wrote: Hi everyone, This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 1.14 to address CVE-2021-44228/CVE-2021-45046. It covers all 4 releases as they contain the same changes (upgrading Log4j to 2.16.0) and were prepared simultaneously by the same person. (Hence, if something is broken, it likely applies to all releases) Note: 1.11/1.12 are still missing the Python Mac releases. Please review and vote on the release candidate #1 for the versions 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: [ ] +1, Approve the releases [ ] -1, Do not approve the releases (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source releases and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint C2EED7B111D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tags [5], * website pull request listing the new releases and adding announcement blog post [6]. The vote will be open for at least 24 hours. The minimum vote time has been shortened as the changes are minimal and the matter is urgent. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Chesnay [1] 1.11: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 1.12: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 1.13: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 1.14: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 [2] 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 1.11: https://repository.apache.org/content/repositories/orgapacheflink-1460 1.12: https://repository.apache.org/content/repositories/orgapacheflink-1462 1.13: https://repository.apache.org/content/repositories/orgapacheflink-1459 1.14: https
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
That's not a problem as far as I'm concerned, because the documentation of these releases is never actually published anywhere. The documentation at nightlies.apache.org will reference 2.16.0. On 16/12/2021 02:01, Yun Gao wrote: Hi, May I have a double confirmation that it seems we still have log4j version written as 2.15.0 in dev/project-configuration.md in the commit corresponding to release tags, which seems to be not consistent with the PR in the github, is it expected and would it have influence? Best, Yun --Original Mail -- Sender:Stephan Ewen Send Date:Thu Dec 16 08:34:10 2021 Recipients:dev Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 +1 (binding) - Verified commit history, looks good => stumbled over the changes in the "create_release_branch.sh ", which are present in each release commit. [1] => agree that these are not an issue, because this is an out-of-band release - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222: Remove NetworkFailureProxy used for Kafka connector tests" - Checked that released binaries and jars reference correct Scala versions - Ran streaming examples against binary releases for 1.12.7, 1.13.5, 1.14.2. Execution logs look correct. - Other checks (licenses, no binaries) carry over from previous releases [1] https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074 On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman wrote: +1 (non-binding) - Checked diff of all versions and verified dep upgrade - Verified checksum and signatures - Built 1.14 from source - checked blog post Seth On Wed, Dec 15, 2021 at 10:22 AM Yu Li wrote: +1 * Verified checksums and signatures * Reviewed website PR - Minor: left a comment to mention CVE-2021-45046 * Checked and confirmed new tags only contain log4j version bump * Checked release notes and found no issues - I've moved FLINK-25317 to 1.14.3 Thanks for driving these releases Chesnay! Best Regards, Yu On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler wrote: FYI; the publication of the python release for 1.11/1.12 will be delayed because we hit the project size limit on pypi again, and increasing that limit may take a while. On the positive side, this gives us more time to fix the mac builds. On 15/12/2021 03:55, Chesnay Schepler wrote: Hi everyone, This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 1.14 to address CVE-2021-44228/CVE-2021-45046. It covers all 4 releases as they contain the same changes (upgrading Log4j to 2.16.0) and were prepared simultaneously by the same person. (Hence, if something is broken, it likely applies to all releases) Note: 1.11/1.12 are still missing the Python Mac releases. Please review and vote on the release candidate #1 for the versions 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: [ ] +1, Approve the releases [ ] -1, Do not approve the releases (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source releases and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint C2EED7B111D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tags [5], * website pull request listing the new releases and adding announcement blog post [6]. The vote will be open for at least 24 hours. The minimum vote time has been shortened as the changes are minimal and the matter is urgent. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Chesnay [1] 1.11: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 1.12: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 1.13: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 1.14: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 [2] 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 1.11: https://repository.apache.org/content/repositories/orgapacheflink-1460 1.12: https://repository.apache.org/content/repositories/orgapacheflink-1462 1.13: https://repository.apache.org/content/repositories/orgapacheflink-1459 1.14: https://repository.apache.org/content/repositories/orgapacheflink-1461 [5] 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 1.13: https://gith
Re: Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
Hi, May I have a double confirmation that it seems we still have log4j version written as 2.15.0 in dev/project-configuration.md in the commit corresponding to release tags, which seems to be not consistent with the PR in the github, is it expected and would it have influence? Best, Yun --Original Mail -- Sender:Stephan Ewen Send Date:Thu Dec 16 08:34:10 2021 Recipients:dev Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 +1 (binding) - Verified commit history, looks good => stumbled over the changes in the "create_release_branch.sh ", which are present in each release commit. [1] => agree that these are not an issue, because this is an out-of-band release - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222: Remove NetworkFailureProxy used for Kafka connector tests" - Checked that released binaries and jars reference correct Scala versions - Ran streaming examples against binary releases for 1.12.7, 1.13.5, 1.14.2. Execution logs look correct. - Other checks (licenses, no binaries) carry over from previous releases [1] https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074 On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman wrote: > +1 (non-binding) > > - Checked diff of all versions and verified dep upgrade > - Verified checksum and signatures > - Built 1.14 from source > - checked blog post > > Seth > > On Wed, Dec 15, 2021 at 10:22 AM Yu Li wrote: > > > +1 > > > > * Verified checksums and signatures > > * Reviewed website PR > > - Minor: left a comment to mention CVE-2021-45046 > > * Checked and confirmed new tags only contain log4j version bump > > * Checked release notes and found no issues > > - I've moved FLINK-25317 to 1.14.3 > > > > Thanks for driving these releases Chesnay! > > > > Best Regards, > > Yu > > > > > > On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler > wrote: > > > > > FYI; the publication of the python release for 1.11/1.12 will be > delayed > > > because we hit the project size limit on pypi again, and increasing > that > > > limit may take a while. > > > On the positive side, this gives us more time to fix the mac builds. > > > > > > On 15/12/2021 03:55, Chesnay Schepler wrote: > > > > Hi everyone, > > > > > > > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 > and > > > > 1.14 to address CVE-2021-44228/CVE-2021-45046. > > > > It covers all 4 releases as they contain the same changes (upgrading > > > > Log4j to 2.16.0) and were prepared simultaneously by the same person. > > > > (Hence, if something is broken, it likely applies to all releases) > > > > > > > > Note: 1.11/1.12 are still missing the Python Mac releases. > > > > > > > > > > > > Please review and vote on the release candidate #1 for the versions > > > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > > > > [ ] +1, Approve the releases > > > > [ ] -1, Do not approve the releases (please provide specific > comments) > > > > > > > > The complete staging area is available for your review, which > includes: > > > > * JIRA release notes [1], > > > > * the official Apache source releases and binary convenience releases > > > > to be deployed to dist.apache.org [2], which are signed with the key > > > > with fingerprint C2EED7B111D464BA [3], > > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > > * source code tags [5], > > > > * website pull request listing the new releases and adding > > > > announcement blog post [6]. > > > > > > > > The vote will be open for at least 24 hours. The minimum vote time > has > > > > been shortened as the changes are minimal and the matter is urgent. > > > > It is adopted by majority approval, with at least 3 PMC affirmative > > > > votes. > > > > > > > > Thanks, > > > > Chesnay > > > > > > > > [1] > > > > 1.11: > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 > > > > 1.12: > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 > > > > 1.13: > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 > > > > 1.14: > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 > > > > [2] > > > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ > > > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ > > > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ > > > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > [4] > > > > 1.11: > > > > > https://repository.apache.org/
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
+1 (binding) - Verified commit history, looks good => stumbled over the changes in the "create_release_branch.sh ", which are present in each release commit. [1] => agree that these are not an issue, because this is an out-of-band release - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222: Remove NetworkFailureProxy used for Kafka connector tests" - Checked that released binaries and jars reference correct Scala versions - Ran streaming examples against binary releases for 1.12.7, 1.13.5, 1.14.2. Execution logs look correct. - Other checks (licenses, no binaries) carry over from previous releases [1] https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074 On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman wrote: > +1 (non-binding) > > - Checked diff of all versions and verified dep upgrade > - Verified checksum and signatures > - Built 1.14 from source > - checked blog post > > Seth > > On Wed, Dec 15, 2021 at 10:22 AM Yu Li wrote: > > > +1 > > > > * Verified checksums and signatures > > * Reviewed website PR > >- Minor: left a comment to mention CVE-2021-45046 > > * Checked and confirmed new tags only contain log4j version bump > > * Checked release notes and found no issues > >- I've moved FLINK-25317 to 1.14.3 > > > > Thanks for driving these releases Chesnay! > > > > Best Regards, > > Yu > > > > > > On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler > wrote: > > > > > FYI; the publication of the python release for 1.11/1.12 will be > delayed > > > because we hit the project size limit on pypi again, and increasing > that > > > limit may take a while. > > > On the positive side, this gives us more time to fix the mac builds. > > > > > > On 15/12/2021 03:55, Chesnay Schepler wrote: > > > > Hi everyone, > > > > > > > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 > and > > > > 1.14 to address CVE-2021-44228/CVE-2021-45046. > > > > It covers all 4 releases as they contain the same changes (upgrading > > > > Log4j to 2.16.0) and were prepared simultaneously by the same person. > > > > (Hence, if something is broken, it likely applies to all releases) > > > > > > > > Note: 1.11/1.12 are still missing the Python Mac releases. > > > > > > > > > > > > Please review and vote on the release candidate #1 for the versions > > > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > > > > [ ] +1, Approve the releases > > > > [ ] -1, Do not approve the releases (please provide specific > comments) > > > > > > > > The complete staging area is available for your review, which > includes: > > > > * JIRA release notes [1], > > > > * the official Apache source releases and binary convenience releases > > > > to be deployed to dist.apache.org [2], which are signed with the key > > > > with fingerprint C2EED7B111D464BA [3], > > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > > * source code tags [5], > > > > * website pull request listing the new releases and adding > > > > announcement blog post [6]. > > > > > > > > The vote will be open for at least 24 hours. The minimum vote time > has > > > > been shortened as the changes are minimal and the matter is urgent. > > > > It is adopted by majority approval, with at least 3 PMC affirmative > > > > votes. > > > > > > > > Thanks, > > > > Chesnay > > > > > > > > [1] > > > > 1.11: > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 > > > > 1.12: > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 > > > > 1.13: > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 > > > > 1.14: > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 > > > > [2] > > > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ > > > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ > > > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ > > > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > [4] > > > > 1.11: > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1460 > > > > 1.12: > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1462 > > > > 1.13: > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1459 > > > > 1.14: > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1461 > > > > [5] > > > > 1.11: > https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 > > > > 1.12: > https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 > > > > 1.13: > https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 > > > > 1.14: > https://
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
+1 (non-binding) - Checked diff of all versions and verified dep upgrade - Verified checksum and signatures - Built 1.14 from source - checked blog post Seth On Wed, Dec 15, 2021 at 10:22 AM Yu Li wrote: > +1 > > * Verified checksums and signatures > * Reviewed website PR >- Minor: left a comment to mention CVE-2021-45046 > * Checked and confirmed new tags only contain log4j version bump > * Checked release notes and found no issues >- I've moved FLINK-25317 to 1.14.3 > > Thanks for driving these releases Chesnay! > > Best Regards, > Yu > > > On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler wrote: > > > FYI; the publication of the python release for 1.11/1.12 will be delayed > > because we hit the project size limit on pypi again, and increasing that > > limit may take a while. > > On the positive side, this gives us more time to fix the mac builds. > > > > On 15/12/2021 03:55, Chesnay Schepler wrote: > > > Hi everyone, > > > > > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and > > > 1.14 to address CVE-2021-44228/CVE-2021-45046. > > > It covers all 4 releases as they contain the same changes (upgrading > > > Log4j to 2.16.0) and were prepared simultaneously by the same person. > > > (Hence, if something is broken, it likely applies to all releases) > > > > > > Note: 1.11/1.12 are still missing the Python Mac releases. > > > > > > > > > Please review and vote on the release candidate #1 for the versions > > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > > > [ ] +1, Approve the releases > > > [ ] -1, Do not approve the releases (please provide specific comments) > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source releases and binary convenience releases > > > to be deployed to dist.apache.org [2], which are signed with the key > > > with fingerprint C2EED7B111D464BA [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tags [5], > > > * website pull request listing the new releases and adding > > > announcement blog post [6]. > > > > > > The vote will be open for at least 24 hours. The minimum vote time has > > > been shortened as the changes are minimal and the matter is urgent. > > > It is adopted by majority approval, with at least 3 PMC affirmative > > > votes. > > > > > > Thanks, > > > Chesnay > > > > > > [1] > > > 1.11: > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 > > > 1.12: > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 > > > 1.13: > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 > > > 1.14: > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 > > > [2] > > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ > > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ > > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ > > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] > > > 1.11: > > > https://repository.apache.org/content/repositories/orgapacheflink-1460 > > > 1.12: > > > https://repository.apache.org/content/repositories/orgapacheflink-1462 > > > 1.13: > > > https://repository.apache.org/content/repositories/orgapacheflink-1459 > > > 1.14: > > > https://repository.apache.org/content/repositories/orgapacheflink-1461 > > > [5] > > > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 > > > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 > > > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 > > > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1 > > > [6] https://github.com/apache/flink-web/pull/489 > > > > > > > > > > >
[jira] [Created] (FLINK-25333) Harden UID management when constructing statefun universe
Seth Wiesman created FLINK-25333: Summary: Harden UID management when constructing statefun universe Key: FLINK-25333 URL: https://issues.apache.org/jira/browse/FLINK-25333 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Seth Wiesman Assignee: Seth Wiesman Fix For: statefun-3.2.0 As described in this stack overflow, we do not set UIDs on routers. Because of how the stream graph is generated, the uids may be non-deterministic. We should manually set all UIDs and enforce this via configuration. https://stackoverflow.com/questions/70316498/flink-statefun-high-availability-exception-java-lang-illegalstateexception-th -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25332) When using Pyflink Table API, 'where' clause seems to work incorrectly
TongMeng created FLINK-25332: Summary: When using Pyflink Table API, 'where' clause seems to work incorrectly Key: FLINK-25332 URL: https://issues.apache.org/jira/browse/FLINK-25332 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.13.0 Environment: Python 3.6.9, Pyflink 1.13.0, kafka2.12-2.4.0 Reporter: TongMeng The UDF I used just returns a float, the first four data it returns 1.0, 2.0, 3.0 and 4.0, then it returns 0.0. I use 'where' in the sql to filter the 0.0 result. So the expected result I want to see in the kafka should be 1.0, 2.0, 3.0 and 4.0. However kafka consumer gives four 0.0. The sql is as follow: "insert into algorithmsink select dt.my_result from(select udf1(a) AS my_result from mysource) AS dt where dt.my_result > 0.0" After I removed the 'where dt.my_result > 0.0' part, it workd well. Kafka gave 1.0, 2.0, 3.0, 4.0, 0.0, 0.0, 0.0…… -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
+1 * Verified checksums and signatures * Reviewed website PR - Minor: left a comment to mention CVE-2021-45046 * Checked and confirmed new tags only contain log4j version bump * Checked release notes and found no issues - I've moved FLINK-25317 to 1.14.3 Thanks for driving these releases Chesnay! Best Regards, Yu On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler wrote: > FYI; the publication of the python release for 1.11/1.12 will be delayed > because we hit the project size limit on pypi again, and increasing that > limit may take a while. > On the positive side, this gives us more time to fix the mac builds. > > On 15/12/2021 03:55, Chesnay Schepler wrote: > > Hi everyone, > > > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and > > 1.14 to address CVE-2021-44228/CVE-2021-45046. > > It covers all 4 releases as they contain the same changes (upgrading > > Log4j to 2.16.0) and were prepared simultaneously by the same person. > > (Hence, if something is broken, it likely applies to all releases) > > > > Note: 1.11/1.12 are still missing the Python Mac releases. > > > > > > Please review and vote on the release candidate #1 for the versions > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > > [ ] +1, Approve the releases > > [ ] -1, Do not approve the releases (please provide specific comments) > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source releases and binary convenience releases > > to be deployed to dist.apache.org [2], which are signed with the key > > with fingerprint C2EED7B111D464BA [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tags [5], > > * website pull request listing the new releases and adding > > announcement blog post [6]. > > > > The vote will be open for at least 24 hours. The minimum vote time has > > been shortened as the changes are minimal and the matter is urgent. > > It is adopted by majority approval, with at least 3 PMC affirmative > > votes. > > > > Thanks, > > Chesnay > > > > [1] > > 1.11: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 > > 1.12: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 > > 1.13: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 > > 1.14: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 > > [2] > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > 1.11: > > https://repository.apache.org/content/repositories/orgapacheflink-1460 > > 1.12: > > https://repository.apache.org/content/repositories/orgapacheflink-1462 > > 1.13: > > https://repository.apache.org/content/repositories/orgapacheflink-1459 > > 1.14: > > https://repository.apache.org/content/repositories/orgapacheflink-1461 > > [5] > > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 > > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 > > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 > > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1 > > [6] https://github.com/apache/flink-web/pull/489 > > > > > >
[jira] [Created] (FLINK-25331) Allow testcontainer tests to run on Java 11
Chesnay Schepler created FLINK-25331: Summary: Allow testcontainer tests to run on Java 11 Key: FLINK-25331 URL: https://issues.apache.org/jira/browse/FLINK-25331 Project: Flink Issue Type: Sub-task Components: Tests Reporter: Chesnay Schepler -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
FYI; the publication of the python release for 1.11/1.12 will be delayed because we hit the project size limit on pypi again, and increasing that limit may take a while. On the positive side, this gives us more time to fix the mac builds. On 15/12/2021 03:55, Chesnay Schepler wrote: Hi everyone, This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 1.14 to address CVE-2021-44228/CVE-2021-45046. It covers all 4 releases as they contain the same changes (upgrading Log4j to 2.16.0) and were prepared simultaneously by the same person. (Hence, if something is broken, it likely applies to all releases) Note: 1.11/1.12 are still missing the Python Mac releases. Please review and vote on the release candidate #1 for the versions 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: [ ] +1, Approve the releases [ ] -1, Do not approve the releases (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source releases and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint C2EED7B111D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tags [5], * website pull request listing the new releases and adding announcement blog post [6]. The vote will be open for at least 24 hours. The minimum vote time has been shortened as the changes are minimal and the matter is urgent. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Chesnay [1] 1.11: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 1.12: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 1.13: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 1.14: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 [2] 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 1.11: https://repository.apache.org/content/repositories/orgapacheflink-1460 1.12: https://repository.apache.org/content/repositories/orgapacheflink-1462 1.13: https://repository.apache.org/content/repositories/orgapacheflink-1459 1.14: https://repository.apache.org/content/repositories/orgapacheflink-1461 [5] 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1 [6] https://github.com/apache/flink-web/pull/489
[jira] [Created] (FLINK-25330) Flink SQL doesn't retract all versions of Hbase data
Bruce Wong created FLINK-25330: -- Summary: Flink SQL doesn't retract all versions of Hbase data Key: FLINK-25330 URL: https://issues.apache.org/jira/browse/FLINK-25330 Project: Flink Issue Type: Bug Components: Connectors / HBase Affects Versions: 1.14.0 Reporter: Bruce Wong Attachments: image-2021-12-15-20-00-22-767.png h2. Background When we use CDC to synchronize mysql data to HBase, we find that HBase deletes only the last version of the specified rowkey when deleting mysql data. The data of the old version still exists. You end up using the wrong data. And I think its a bug of HBase connector. The following figure shows Hbase data changes before and after mysql data is deleted. !image-2021-12-15-19-59-31-837.png|width=925,height=680! h2. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [CANCELLED] Release 1.11.5/1.12.6/1.13.4/1.14.1, release candidate #1
That's right, they are referenced in POMs published with the jars, though. But that's minor. On Wed, Dec 15, 2021 at 12:28 PM Chesnay Schepler wrote: > AFAIK none of the jars we publish actually contains log4j. > It's only bundled by the distribution/python binaries/docker images. > > Hence I don't think the jars help in this case. > > On 15/12/2021 10:42, Stephan Ewen wrote: > > Given that these artifacts are published already, users can use them if > > they want to update now: > > > > For example: > > https://search.maven.org/artifact/org.apache.flink/flink-core/1.14.1/jar > > > > Just for the users that really want to update now (rather than rely on > the > > mitigation via config) and are not as much concerned about the remaining > > weakness in log4j 2.15.0 > > > > On Tue, Dec 14, 2021 at 11:18 PM Seth Wiesman > wrote: > > > >> Thank you for managing these updates Chesnay! > >> > >> > >> > >> On Tue, Dec 14, 2021 at 3:51 PM Chesnay Schepler > >> wrote: > >> > >>> Since the maven artifacts have already been published we will use the > >>> next patch version for each release, i.e.: > >>> 1.11.6 > >>> 1.12.7 > >>> 1.13.5 > >>> 1.14.2 > >>> > >>> (We could technically just update the source/binaries, but that seems > >>> fishy). > >>> > >>> On 14/12/2021 22:38, Chesnay Schepler wrote: > I'm canceling the release because the issue was not fully fixed in > Log4j 2.15.0; see CVE-2021-45046. > > I will start preparing new release candidates that use Log4j 2.16.0 . > > On 14/12/2021 21:28, Chesnay Schepler wrote: > > The vote duration has passed and we have approved the releases. > > > > Binding votes: > > * Stephan > > * Till > > * Xintong > > * Zhu > > * Gordon > > > > I will not finalize the release. > > > > On 13/12/2021 20:28, Chesnay Schepler wrote: > >> Hi everyone, > >> > >> This vote is for the emergency patch releases for 1.11, 1.12, 1.13 > >> and 1.14 to address CVE-2021-44228. > >> It covers all 4 releases as they contain the same changes (upgrading > >> Log4j to 2.15.0) and were prepared simultaneously by the same > person. > >> (Hence, if something is broken, it likely applies to all releases) > >> > >> Please review and vote on the release candidate #1 for the versions > >> 1.11.5, 1.12.6, 1.13.4 and 1.14.1, as follows: > >> [ ] +1, Approve the releases > >> [ ] -1, Do not approve the releases (please provide specific > >> comments) > >> The complete staging area is available for your review, which > >> includes: > >> * JIRA release notes [1], > >> * the official Apache source releases and binary convenience > >> releases to be deployed to dist.apache.org [2], which are signed > >> with the key with fingerprint C2EED7B111D464BA [3], > >> * all artifacts to be deployed to the Maven Central Repository [4], > >> * *the jars for 1.13/1.14 are still being built* > >> * source code tags [5], > >> * website pull request listing the new releases and adding > >> announcement blog post [6]. > >> > >> The vote will be open for at least 24 hours. The minimum vote time > >> has been shortened as the changes are minimal and the matter is > >> urgent. > >> It is adopted by majority approval, with at least 3 PMC affirmative > >> votes. > >> > >> Thanks, > >> Chesnay > >> > >> [1] > >> 1.11: > >> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350327 > >> 1.12: > >> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350328 > >> 1.13: > >> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350686 > >> 1.14: > >> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350512 > >> [2] > >> 1.11: > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.5-rc1/ > >> 1.12: > https://dist.apache.org/repos/dist/dev/flink/flink-1.12.6-rc1/ > >> 1.13: > https://dist.apache.org/repos/dist/dev/flink/flink-1.13.4-rc1/ > >> 1.14: > https://dist.apache.org/repos/dist/dev/flink/flink-1.14.1-rc1/ > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > >> [4] > >> 1.11/1.12: > >> > >> https://repository.apache.org/content/repositories/orgapacheflink-1455 > >> 1.13: > >> > >> https://repository.apache.org/content/repositories/orgapacheflink-1457 > >> 1.14: > >> > >> https://repository.apache.org/content/repositories/orgapacheflink-1456 > >> [5] > >> 1.11: > >> https://github.com/apache/flink/releases/tag/release-1.11.5-rc1 > >> 1.12: > >> https://github.com/apache/flink/releases/tag/release-1.12.6-rc1 > >> 1.13: > >> https://github.com/apache/flink/releases/tag/release-1.13.4-rc1 > >> 1.14: > >> https://github.com/apache/flink/releases/tag/release-1.1
[jira] [Created] (FLINK-25329) Improvement of disk operations in flink session cluster for jobs
Shammon created FLINK-25329: --- Summary: Improvement of disk operations in flink session cluster for jobs Key: FLINK-25329 URL: https://issues.apache.org/jira/browse/FLINK-25329 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Affects Versions: 1.13.3, 1.12.5, 1.14.0 Reporter: Shammon Flink session cluster uses files to store info of jobs after they reach termination with `FileExecutionGraphInfoStore`, each job will generate one file. When the cluster executes many small jobs concurrently, there will be many disk related operations, which will 1> Increase the CPU usage of `Dispatcher` 2> Decrease the performance of the jobs in the cluster. We hope to improve the disk operations in `FileExecutionGraphInfoStore` to increase the performance of session cluster. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [CANCELLED] Release 1.11.5/1.12.6/1.13.4/1.14.1, release candidate #1
AFAIK none of the jars we publish actually contains log4j. It's only bundled by the distribution/python binaries/docker images. Hence I don't think the jars help in this case. On 15/12/2021 10:42, Stephan Ewen wrote: Given that these artifacts are published already, users can use them if they want to update now: For example: https://search.maven.org/artifact/org.apache.flink/flink-core/1.14.1/jar Just for the users that really want to update now (rather than rely on the mitigation via config) and are not as much concerned about the remaining weakness in log4j 2.15.0 On Tue, Dec 14, 2021 at 11:18 PM Seth Wiesman wrote: Thank you for managing these updates Chesnay! On Tue, Dec 14, 2021 at 3:51 PM Chesnay Schepler wrote: Since the maven artifacts have already been published we will use the next patch version for each release, i.e.: 1.11.6 1.12.7 1.13.5 1.14.2 (We could technically just update the source/binaries, but that seems fishy). On 14/12/2021 22:38, Chesnay Schepler wrote: I'm canceling the release because the issue was not fully fixed in Log4j 2.15.0; see CVE-2021-45046. I will start preparing new release candidates that use Log4j 2.16.0 . On 14/12/2021 21:28, Chesnay Schepler wrote: The vote duration has passed and we have approved the releases. Binding votes: * Stephan * Till * Xintong * Zhu * Gordon I will not finalize the release. On 13/12/2021 20:28, Chesnay Schepler wrote: Hi everyone, This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 1.14 to address CVE-2021-44228. It covers all 4 releases as they contain the same changes (upgrading Log4j to 2.15.0) and were prepared simultaneously by the same person. (Hence, if something is broken, it likely applies to all releases) Please review and vote on the release candidate #1 for the versions 1.11.5, 1.12.6, 1.13.4 and 1.14.1, as follows: [ ] +1, Approve the releases [ ] -1, Do not approve the releases (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source releases and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint C2EED7B111D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * *the jars for 1.13/1.14 are still being built* * source code tags [5], * website pull request listing the new releases and adding announcement blog post [6]. The vote will be open for at least 24 hours. The minimum vote time has been shortened as the changes are minimal and the matter is urgent. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Chesnay [1] 1.11: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350327 1.12: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350328 1.13: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350686 1.14: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350512 [2] 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.5-rc1/ 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.6-rc1/ 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.4-rc1/ 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.1-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 1.11/1.12: https://repository.apache.org/content/repositories/orgapacheflink-1455 1.13: https://repository.apache.org/content/repositories/orgapacheflink-1457 1.14: https://repository.apache.org/content/repositories/orgapacheflink-1456 [5] 1.11: https://github.com/apache/flink/releases/tag/release-1.11.5-rc1 1.12: https://github.com/apache/flink/releases/tag/release-1.12.6-rc1 1.13: https://github.com/apache/flink/releases/tag/release-1.13.4-rc1 1.14: https://github.com/apache/flink/releases/tag/release-1.14.1-rc1 [6] https://github.com/apache/flink-web/pull/489
[jira] [Created] (FLINK-25328) Improvement of share memory manager between jobs if they use the same slot in TaskManager for flink olap queries
Shammon created FLINK-25328: --- Summary: Improvement of share memory manager between jobs if they use the same slot in TaskManager for flink olap queries Key: FLINK-25328 URL: https://issues.apache.org/jira/browse/FLINK-25328 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Affects Versions: 1.13.3, 1.12.5, 1.14.0 Reporter: Shammon We submit batch jobs to flink session cluster as olap queries, and these jobs' subtasks in TaskManager are frequently created and destroyed because they finish their work quickly. Each slot in taskmanager manages `MemoryManager` for multiple tasks in one job, and the `MemoryManager` is closed when all the subtasks are finished. Join/Aggregate/Sort and etc. operators in the subtasks allocate `MemorySegment` via `MemoryManager` and these `MemorySegment` will be free when they are finished. It causes too much memory allocation and free of `MemorySegment` in taskmanager. For example, a TaskManager contains 50 slots, one job has 3 join/agg operatos run in the slot, each operator will allocate 2000 segments and initialize them. If the subtasks of a job take 100ms to execute, then the taskmanager will execute 10 jobs' subtasks one second and it will allocate and free 2000 * 3 * 50 * 10 = 300w segments for them. Allocate and free too many segments from memory will cause two issues: 1) Increases the CPU usage of taskmanager 2) Increase the cost of subtasks in taskmanager, which will increase the latency of job and decrease the qps. To improve the usage of memory segment between jobs in the same slot, we propose not drop memory manager when all the subtasks in the slot are finished. The slot will hold the `MemoryManager` and not free the allocated `MemorySegment` in it immediately. When some subtasks of another job are assigned to the slot, they don't need to allocate segments from memory and can reuse the `MemoryManager` and `MemorySegment` in it. WDYT? [~xtsong] THX -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25327) ApplicationMode "DELETE /cluster" REST call leads to exit code 2, instead of 0
Robert Metzger created FLINK-25327: -- Summary: ApplicationMode "DELETE /cluster" REST call leads to exit code 2, instead of 0 Key: FLINK-25327 URL: https://issues.apache.org/jira/browse/FLINK-25327 Project: Flink Issue Type: Bug Components: Runtime / Coordination Reporter: Robert Metzger FLINK-24113 introduced a mode to keep the Application Mode JobManager running after the Job has been cancelled. Cluster shutdown needs to be initiated for example using the DELETE /cluster REST endpoint. The problem is that there can be a fatal error during the shutdown, making the JobManager exit with return code != 0 (making resource managers believe there was an error with the Flink application) Error {code} 2021-12-15 08:09:55,708 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error occurred in the cluster entrypoint. org.apache.flink.util.FlinkException: Application failed unexpectedly. at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$finishBootstrapTasks$1(ApplicationDispatcherBootstrap.java:177) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278) ~[?:1.8.0_312] at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:125) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:284) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at org.apache.flink.util.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:696) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) ~[?:1.8.0_312] at org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795) ~[?:1.8.0_312] at java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163) ~[?:1.8.0_312] at org.apache.flink.util.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:693) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at org.apache.flink.util.concurrent.FutureUtils.runAfterwards(FutureUtils.java:660) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:281) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214) ~[flink-dist-1.15-master-robert.jar:1.15-master-robert] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.lambda$terminate$0(AkkaRpcActor.java:580) ~[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert] at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83) ~[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:579) ~[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:191) ~[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-
Re: [DISCUSS] FLIP-198: Working directory for Flink processes
This is true. But this is not a new problem and I think that Flink should be susceptible to this problem already. One solution for this concrete case could be that the BlobServer stores some checksums and validates the file before serving it to the TM. Cheers, Till On Wed, Dec 15, 2021 at 11:59 AM Chesnay Schepler wrote: > The issue with corrupted files is that some of them aren't read by the > component that stores them. > For example, a file can be corrupted in the blob server of the JM, but > that it is corrupted will only be noticed by the TaskExecutor. > > On 15/12/2021 11:36, Till Rohrmann wrote: > > Thanks everyone for your feedback. Let me try to address it by grouping > > some of the individual comments: > > > > ### Will this feature work for native Yarn and K8s deployments? > > > > The working directory is an optional feature that can be used to recover > > additional information. You can think of it like a cache. If the working > > directory is there, then Flink can do certain things a bit faster but in > > the worst case it will have to retrieve the required information from the > > JobManager or persistent storage. > > > > In order to make it work with native Yarn and K8s, we would have to > change > > these modes slightly. First of all, we would have to be able to map > working > > directories to processes and then set a deterministic resource ids for > the > > processes. For K8s this could be easily achievable by using a StatefulSet > > as the deployment mechanism for TaskExecutors. For Yarn, we probably > would > > have to remember the prior locations of a process. Both things are > > potential follow ups that I don't want to tackle in this FLIP. > > > > If one of the modes configures the working directory to be on a full or > > broken disk, then the process will fail. I think this is not all that > > different from the current state where some things in Flink will fail if > > they picked the wrong/full temporary directory (e.g. blob storage > > directory). > > > > ### Cleanup > > > > The working directory will be cleaned up if the Flink process is > gracefully > > shut down. This means that the JobManager process will clean it up if it > > runs in application mode and the job is terminated. SIGTERM and SIGKILL > > signals will be treated as an ungraceful shutdown and therefore they > won't > > clean up the working directory. This means that we probably also need a > > graceful way for shutting TaskManager processes down in the future > because > > right now they are in most cases killed in order to shut them down. If > the > > user uses the tmp directory, then any left-over working directories will > be > > cleaned up with the next system restart. This is somewhat similar to how > > RocksDB's working directory is currently cleaned up as well. > > > > ### Corrupted files > > > > The working directory itself won't give you any guarantees. It will be > the > > responsibility of the component that uses the working directory to make > > sure that it can deal with corrupted files. E.g. if the component cannot > > read the file, then it should delete it and fall back to the remote > > storage/ground truth to retrieve the required information. > > > > I hope this could answer your questions. Let me know if you have more > > feedback. > > > > Cheers, > > Till > > > > On Mon, Dec 13, 2021 at 5:05 AM 刘建刚 wrote: > > > >> I like the idea. It can reuse the disk to do many things. Isn't it only > >> for inner failover? If not, the cleaning may be a problem. Also, many > >> resource components have their own disk schedule strategy. > >> > >> Chesnay Schepler 于2021年12月12日周日 19:59写道: > >> > >>> How do you intend to handle corrupted files, in particular due to > >>> process crashes during a write? > >>> Will all writes to a cached directory append some suffix (e.g., > >>> ".pending") and do a rename? > >>> > >>> On 10/12/2021 17:54, Till Rohrmann wrote: > Hi everyone, > > I would like to start a discussion about introducing an explicit > working > directory for Flink processes that can be used to store information > [1]. > Per default this working directory will reside in the temporary > >>> directory > of the node Flink runs on. However, if configured to reside on a > >>> persistent > volume, then this information can be used to recover from process/node > failures. Moreover, such a working directory can be used to > consolidate > some of our other directories Flink creates under /tmp (e.g. > >>> blobStorage, > RocksDB working directory). > > Here is a draft PR that outlines the required changes [2]. > > Looking forward to your feedback. > > [1] https://cwiki.apache.org/confluence/x/ZZiqCw > [2] https://github.com/apache/flink/pull/18083 > > Cheers, > Till > > >>> > >
Re: [DISCUSS] FLIP-198: Working directory for Flink processes
The issue with corrupted files is that some of them aren't read by the component that stores them. For example, a file can be corrupted in the blob server of the JM, but that it is corrupted will only be noticed by the TaskExecutor. On 15/12/2021 11:36, Till Rohrmann wrote: Thanks everyone for your feedback. Let me try to address it by grouping some of the individual comments: ### Will this feature work for native Yarn and K8s deployments? The working directory is an optional feature that can be used to recover additional information. You can think of it like a cache. If the working directory is there, then Flink can do certain things a bit faster but in the worst case it will have to retrieve the required information from the JobManager or persistent storage. In order to make it work with native Yarn and K8s, we would have to change these modes slightly. First of all, we would have to be able to map working directories to processes and then set a deterministic resource ids for the processes. For K8s this could be easily achievable by using a StatefulSet as the deployment mechanism for TaskExecutors. For Yarn, we probably would have to remember the prior locations of a process. Both things are potential follow ups that I don't want to tackle in this FLIP. If one of the modes configures the working directory to be on a full or broken disk, then the process will fail. I think this is not all that different from the current state where some things in Flink will fail if they picked the wrong/full temporary directory (e.g. blob storage directory). ### Cleanup The working directory will be cleaned up if the Flink process is gracefully shut down. This means that the JobManager process will clean it up if it runs in application mode and the job is terminated. SIGTERM and SIGKILL signals will be treated as an ungraceful shutdown and therefore they won't clean up the working directory. This means that we probably also need a graceful way for shutting TaskManager processes down in the future because right now they are in most cases killed in order to shut them down. If the user uses the tmp directory, then any left-over working directories will be cleaned up with the next system restart. This is somewhat similar to how RocksDB's working directory is currently cleaned up as well. ### Corrupted files The working directory itself won't give you any guarantees. It will be the responsibility of the component that uses the working directory to make sure that it can deal with corrupted files. E.g. if the component cannot read the file, then it should delete it and fall back to the remote storage/ground truth to retrieve the required information. I hope this could answer your questions. Let me know if you have more feedback. Cheers, Till On Mon, Dec 13, 2021 at 5:05 AM 刘建刚 wrote: I like the idea. It can reuse the disk to do many things. Isn't it only for inner failover? If not, the cleaning may be a problem. Also, many resource components have their own disk schedule strategy. Chesnay Schepler 于2021年12月12日周日 19:59写道: How do you intend to handle corrupted files, in particular due to process crashes during a write? Will all writes to a cached directory append some suffix (e.g., ".pending") and do a rename? On 10/12/2021 17:54, Till Rohrmann wrote: Hi everyone, I would like to start a discussion about introducing an explicit working directory for Flink processes that can be used to store information [1]. Per default this working directory will reside in the temporary directory of the node Flink runs on. However, if configured to reside on a persistent volume, then this information can be used to recover from process/node failures. Moreover, such a working directory can be used to consolidate some of our other directories Flink creates under /tmp (e.g. blobStorage, RocksDB working directory). Here is a draft PR that outlines the required changes [2]. Looking forward to your feedback. [1] https://cwiki.apache.org/confluence/x/ZZiqCw [2] https://github.com/apache/flink/pull/18083 Cheers, Till
Re: [DISCUSS] FLIP-198: Working directory for Flink processes
Thanks everyone for your feedback. Let me try to address it by grouping some of the individual comments: ### Will this feature work for native Yarn and K8s deployments? The working directory is an optional feature that can be used to recover additional information. You can think of it like a cache. If the working directory is there, then Flink can do certain things a bit faster but in the worst case it will have to retrieve the required information from the JobManager or persistent storage. In order to make it work with native Yarn and K8s, we would have to change these modes slightly. First of all, we would have to be able to map working directories to processes and then set a deterministic resource ids for the processes. For K8s this could be easily achievable by using a StatefulSet as the deployment mechanism for TaskExecutors. For Yarn, we probably would have to remember the prior locations of a process. Both things are potential follow ups that I don't want to tackle in this FLIP. If one of the modes configures the working directory to be on a full or broken disk, then the process will fail. I think this is not all that different from the current state where some things in Flink will fail if they picked the wrong/full temporary directory (e.g. blob storage directory). ### Cleanup The working directory will be cleaned up if the Flink process is gracefully shut down. This means that the JobManager process will clean it up if it runs in application mode and the job is terminated. SIGTERM and SIGKILL signals will be treated as an ungraceful shutdown and therefore they won't clean up the working directory. This means that we probably also need a graceful way for shutting TaskManager processes down in the future because right now they are in most cases killed in order to shut them down. If the user uses the tmp directory, then any left-over working directories will be cleaned up with the next system restart. This is somewhat similar to how RocksDB's working directory is currently cleaned up as well. ### Corrupted files The working directory itself won't give you any guarantees. It will be the responsibility of the component that uses the working directory to make sure that it can deal with corrupted files. E.g. if the component cannot read the file, then it should delete it and fall back to the remote storage/ground truth to retrieve the required information. I hope this could answer your questions. Let me know if you have more feedback. Cheers, Till On Mon, Dec 13, 2021 at 5:05 AM 刘建刚 wrote: > I like the idea. It can reuse the disk to do many things. Isn't it only > for inner failover? If not, the cleaning may be a problem. Also, many > resource components have their own disk schedule strategy. > > Chesnay Schepler 于2021年12月12日周日 19:59写道: > >> How do you intend to handle corrupted files, in particular due to >> process crashes during a write? >> Will all writes to a cached directory append some suffix (e.g., >> ".pending") and do a rename? >> >> On 10/12/2021 17:54, Till Rohrmann wrote: >> > Hi everyone, >> > >> > I would like to start a discussion about introducing an explicit working >> > directory for Flink processes that can be used to store information [1]. >> > Per default this working directory will reside in the temporary >> directory >> > of the node Flink runs on. However, if configured to reside on a >> persistent >> > volume, then this information can be used to recover from process/node >> > failures. Moreover, such a working directory can be used to consolidate >> > some of our other directories Flink creates under /tmp (e.g. >> blobStorage, >> > RocksDB working directory). >> > >> > Here is a draft PR that outlines the required changes [2]. >> > >> > Looking forward to your feedback. >> > >> > [1] https://cwiki.apache.org/confluence/x/ZZiqCw >> > [2] https://github.com/apache/flink/pull/18083 >> > >> > Cheers, >> > Till >> > >> >>
Re: [VOTE] FLIP-190: Support Version Upgrades for Table API & SQL Programs
+1 (binding) Best, Godfrey Ingo Bürk 于2021年12月15日周三 16:19写道: > > +1 (binding) > > Thanks for driving this much needed feature! > > On 14.12.21 17:45, Timo Walther wrote: > > Hi everyone, > > > > I'd like to start a vote on FLIP-190: Support Version Upgrades for Table > > API & SQL Programs [1] which has been discussed in this thread [2]. > > > > The vote will be open for at least 72 hours unless there is an objection > > or not enough votes. > > > > [1] https://cwiki.apache.org/confluence/x/KZBnCw > > [2] https://lists.apache.org/thread/n8v32j6o3d50mpblxydbz82q1q436ob4 > > > > Cheers, > > Timo
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
I had to monkey patch the release scripts a bit; it's unfortunate that the changes ended up in the releases, but I don't think it's a blocker. On 15/12/2021 11:01, Till Rohrmann wrote: Thanks for driving these releases Chesnay! +1 * Verified checksums and signatures * Reviewed website PR * Checked that build tags only contain log4j version bump In the release-1.14.2-rc1 there are some changes to the create_release_branch.sh script that don't seem necessary. But I think this does not matter since we won't build upon release-1.14.2-rc1 in the future. Cheers, Till On Wed, Dec 15, 2021 at 10:40 AM Yun Tang wrote: + 1 (non-binding) * Select several files of each release to verify the signature and sha512sum. * Reviewed the flink-web PR * checked that those 4 release-branchs only contains fix to bump log4j2 version to 2.15.0 and then to 2.16.0 * Checked the JIRA release notes and found that FLINK-25317 [1] was tagged as 1.14.2 version to resolve by mistake. [1] https://issues.apache.org/jira/browse/FLINK-25317 Best Yun Tang From: Chesnay Schepler Sent: Wednesday, December 15, 2021 10:55 To: dev@flink.apache.org Subject: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 Hi everyone, This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 1.14 to address CVE-2021-44228/CVE-2021-45046. It covers all 4 releases as they contain the same changes (upgrading Log4j to 2.16.0) and were prepared simultaneously by the same person. (Hence, if something is broken, it likely applies to all releases) Note: 1.11/1.12 are still missing the Python Mac releases. Please review and vote on the release candidate #1 for the versions 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: [ ] +1, Approve the releases [ ] -1, Do not approve the releases (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source releases and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint C2EED7B111D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tags [5], * website pull request listing the new releases and adding announcement blog post [6]. The vote will be open for at least 24 hours. The minimum vote time has been shortened as the changes are minimal and the matter is urgent. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Chesnay [1] 1.11: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 1.12: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 1.13: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 1.14: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 [2] 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 1.11: https://repository.apache.org/content/repositories/orgapacheflink-1460 1.12: https://repository.apache.org/content/repositories/orgapacheflink-1462 1.13: https://repository.apache.org/content/repositories/orgapacheflink-1459 1.14: https://repository.apache.org/content/repositories/orgapacheflink-1461 [5] 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1 [6] https://github.com/apache/flink-web/pull/489
[jira] [Created] (FLINK-25326) KafkaUtil.createKafkaContainer log levels are not set correctly
Alexander Preuss created FLINK-25326: Summary: KafkaUtil.createKafkaContainer log levels are not set correctly Key: FLINK-25326 URL: https://issues.apache.org/jira/browse/FLINK-25326 Project: Flink Issue Type: Bug Components: Connectors / Kafka Affects Versions: 1.15.0 Reporter: Alexander Preuss The internal kafka log levels set in KafkaUtils.createKafkaContainer method are wrong due to the order of the log hierarchy. If the test logger is set to e.g. 'DEBUG' it means that `logger.isErrorEnabled()` already evaluated to true and therefore the log level gets set to ERROR instead. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25325) Migration Flink from Junit4 to Junit5
Jing Ge created FLINK-25325: --- Summary: Migration Flink from Junit4 to Junit5 Key: FLINK-25325 URL: https://issues.apache.org/jira/browse/FLINK-25325 Project: Flink Issue Type: New Feature Components: Tests Affects Versions: 1.14.0 Reporter: Jing Ge Fix For: 1.15.0 Based on the consensus from the mailing list discussion[1][2], we have been starting working on the JUnit4 to JUnit5 migration. This is the umbrella ticket which describes the big picture of the migration with following steps: * AssertJ integration and guideline * Test Framework upgrade from JUnit4 to JUnit5 * JUnit5 migration guideline(document and reference migration) * Optimization for issues found while writing new test in JUint5 * [Long-term]Module based graceful migration of old tests in JUnit4 to JUnit5 [1] [[DISCUSS]Moving to JUnit5|https://lists.apache.org/thread/jsjvc2cqb91pyh47d4p6olk3c1vxqm3w] [2] [[DISCUSS] Conventions on assertions to use in tests|https://lists.apache.org/thread/33t7hz8w873p1bc5msppk65792z08rgt] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25324) Migration Flink from Junit4 to Junit5
Jing Ge created FLINK-25324: --- Summary: Migration Flink from Junit4 to Junit5 Key: FLINK-25324 URL: https://issues.apache.org/jira/browse/FLINK-25324 Project: Flink Issue Type: New Feature Components: Tests Affects Versions: 1.14.0 Reporter: Jing Ge Fix For: 1.15.0 Based on the consensus from the mailing list discussion[1][2], we have been starting working on the JUnit4 to JUnit5 migration. This is the umbrella ticket which describes the big picture of the migration with following steps: * AssertJ integration and guideline * Test Framework upgrade from JUnit4 to JUnit5 * JUnit5 migration guideline(document and reference migration) * Optimization for issues found while writing new test in JUint5 * [Long-term]Module based graceful migration of old tests in JUnit4 to JUnit5 [1] [[DISCUSS]Moving to JUnit5|https://lists.apache.org/thread/jsjvc2cqb91pyh47d4p6olk3c1vxqm3w] [2] [[DISCUSS] Conventions on assertions to use in tests|https://lists.apache.org/thread/33t7hz8w873p1bc5msppk65792z08rgt] -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
Thanks for driving these releases Chesnay! +1 * Verified checksums and signatures * Reviewed website PR * Checked that build tags only contain log4j version bump In the release-1.14.2-rc1 there are some changes to the create_release_branch.sh script that don't seem necessary. But I think this does not matter since we won't build upon release-1.14.2-rc1 in the future. Cheers, Till On Wed, Dec 15, 2021 at 10:40 AM Yun Tang wrote: > > + 1 (non-binding) > > > * Select several files of each release to verify the signature and > sha512sum. > * Reviewed the flink-web PR > * checked that those 4 release-branchs only contains fix to bump > log4j2 version to 2.15.0 and then to 2.16.0 > * Checked the JIRA release notes and found that FLINK-25317 [1] was > tagged as 1.14.2 version to resolve by mistake. > > [1] https://issues.apache.org/jira/browse/FLINK-25317 > > Best > Yun Tang > > From: Chesnay Schepler > Sent: Wednesday, December 15, 2021 10:55 > To: dev@flink.apache.org > Subject: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 > > Hi everyone, > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and > 1.14 to address CVE-2021-44228/CVE-2021-45046. > It covers all 4 releases as they contain the same changes (upgrading > Log4j to 2.16.0) and were prepared simultaneously by the same person. > (Hence, if something is broken, it likely applies to all releases) > > Note: 1.11/1.12 are still missing the Python Mac releases. > > > Please review and vote on the release candidate #1 for the versions > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: > [ ] +1, Approve the releases > [ ] -1, Do not approve the releases (please provide specific comments) > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source releases and binary convenience releases to > be deployed to dist.apache.org [2], which are signed with the key with > fingerprint C2EED7B111D464BA [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tags [5], > * website pull request listing the new releases and adding announcement > blog post [6]. > > The vote will be open for at least 24 hours. The minimum vote time has > been shortened as the changes are minimal and the matter is urgent. > It is adopted by majority approval, with at least 3 PMC affirmative votes. > > Thanks, > Chesnay > > [1] > 1.11: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 > 1.12: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 > 1.13: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 > 1.14: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 > [2] > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] > 1.11: > https://repository.apache.org/content/repositories/orgapacheflink-1460 > 1.12: > https://repository.apache.org/content/repositories/orgapacheflink-1462 > 1.13: > https://repository.apache.org/content/repositories/orgapacheflink-1459 > 1.14: > https://repository.apache.org/content/repositories/orgapacheflink-1461 > [5] > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1 > [6] https://github.com/apache/flink-web/pull/489 > > >
[jira] [Created] (FLINK-25323) Support side output late data for interval join
chenyuzhi created FLINK-25323: - Summary: Support side output late data for interval join Key: FLINK-25323 URL: https://issues.apache.org/jira/browse/FLINK-25323 Project: Flink Issue Type: Improvement Reporter: chenyuzhi Now, Flink just discard late data when using interval-join: [https://github.com/apache/flink/blob/83a2541475228a4ff9e9a9def4049fb742353549/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L231] Maybe we could use two outputTags to side output late data for left/right stream. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl
Congratulations, Matthias, and welcome to the Flink committers! On Mon, Dec 13, 2021 at 4:39 AM 刘建刚 wrote: > Congratulations! > > Best > Liu Jiangang > > Nicholas Jiang 于2021年12月13日周一 11:23写道: > > > Congratulations, Matthias! > > > > Best, > > Nicholas Jiang > > >
Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk
Congrats and welcome! On Mon, Dec 13, 2021 at 4:39 AM 刘建刚 wrote: > Congratulations! > > Best > Liu Jiangang > > Nicholas Jiang 于2021年12月13日周一 11:28写道: > > > Congratulations, Ingo! > > > > Best, > > Nicholas Jiang > > >
Re: [CANCELLED] Release 1.11.5/1.12.6/1.13.4/1.14.1, release candidate #1
Given that these artifacts are published already, users can use them if they want to update now: For example: https://search.maven.org/artifact/org.apache.flink/flink-core/1.14.1/jar Just for the users that really want to update now (rather than rely on the mitigation via config) and are not as much concerned about the remaining weakness in log4j 2.15.0 On Tue, Dec 14, 2021 at 11:18 PM Seth Wiesman wrote: > Thank you for managing these updates Chesnay! > > > > On Tue, Dec 14, 2021 at 3:51 PM Chesnay Schepler > wrote: > > > Since the maven artifacts have already been published we will use the > > next patch version for each release, i.e.: > > 1.11.6 > > 1.12.7 > > 1.13.5 > > 1.14.2 > > > > (We could technically just update the source/binaries, but that seems > > fishy). > > > > On 14/12/2021 22:38, Chesnay Schepler wrote: > > > I'm canceling the release because the issue was not fully fixed in > > > Log4j 2.15.0; see CVE-2021-45046. > > > > > > I will start preparing new release candidates that use Log4j 2.16.0 . > > > > > > On 14/12/2021 21:28, Chesnay Schepler wrote: > > >> The vote duration has passed and we have approved the releases. > > >> > > >> Binding votes: > > >> * Stephan > > >> * Till > > >> * Xintong > > >> * Zhu > > >> * Gordon > > >> > > >> I will not finalize the release. > > >> > > >> On 13/12/2021 20:28, Chesnay Schepler wrote: > > >>> Hi everyone, > > >>> > > >>> This vote is for the emergency patch releases for 1.11, 1.12, 1.13 > > >>> and 1.14 to address CVE-2021-44228. > > >>> It covers all 4 releases as they contain the same changes (upgrading > > >>> Log4j to 2.15.0) and were prepared simultaneously by the same person. > > >>> (Hence, if something is broken, it likely applies to all releases) > > >>> > > >>> Please review and vote on the release candidate #1 for the versions > > >>> 1.11.5, 1.12.6, 1.13.4 and 1.14.1, as follows: > > >>> [ ] +1, Approve the releases > > >>> [ ] -1, Do not approve the releases (please provide specific > comments) > > >>> > > >>> The complete staging area is available for your review, which > includes: > > >>> * JIRA release notes [1], > > >>> * the official Apache source releases and binary convenience > > >>> releases to be deployed to dist.apache.org [2], which are signed > > >>> with the key with fingerprint C2EED7B111D464BA [3], > > >>> * all artifacts to be deployed to the Maven Central Repository [4], > > >>> * *the jars for 1.13/1.14 are still being built* > > >>> * source code tags [5], > > >>> * website pull request listing the new releases and adding > > >>> announcement blog post [6]. > > >>> > > >>> The vote will be open for at least 24 hours. The minimum vote time > > >>> has been shortened as the changes are minimal and the matter is > urgent. > > >>> It is adopted by majority approval, with at least 3 PMC affirmative > > >>> votes. > > >>> > > >>> Thanks, > > >>> Chesnay > > >>> > > >>> [1] > > >>> 1.11: > > >>> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350327 > > >>> 1.12: > > >>> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350328 > > >>> 1.13: > > >>> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350686 > > >>> 1.14: > > >>> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12350512 > > >>> [2] > > >>> 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.5-rc1/ > > >>> 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.6-rc1/ > > >>> 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.4-rc1/ > > >>> 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.1-rc1/ > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > >>> [4] > > >>> 1.11/1.12: > > >>> > https://repository.apache.org/content/repositories/orgapacheflink-1455 > > >>> 1.13: > > >>> > https://repository.apache.org/content/repositories/orgapacheflink-1457 > > >>> 1.14: > > >>> > https://repository.apache.org/content/repositories/orgapacheflink-1456 > > >>> [5] > > >>> 1.11: > https://github.com/apache/flink/releases/tag/release-1.11.5-rc1 > > >>> 1.12: > https://github.com/apache/flink/releases/tag/release-1.12.6-rc1 > > >>> 1.13: > https://github.com/apache/flink/releases/tag/release-1.13.4-rc1 > > >>> 1.14: > https://github.com/apache/flink/releases/tag/release-1.14.1-rc1 > > >>> [6] https://github.com/apache/flink-web/pull/489 > > >>> > > >> > > > > > > > >
Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
+ 1 (non-binding) * Select several files of each release to verify the signature and sha512sum. * Reviewed the flink-web PR * checked that those 4 release-branchs only contains fix to bump log4j2 version to 2.15.0 and then to 2.16.0 * Checked the JIRA release notes and found that FLINK-25317 [1] was tagged as 1.14.2 version to resolve by mistake. [1] https://issues.apache.org/jira/browse/FLINK-25317 Best Yun Tang From: Chesnay Schepler Sent: Wednesday, December 15, 2021 10:55 To: dev@flink.apache.org Subject: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1 Hi everyone, This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 1.14 to address CVE-2021-44228/CVE-2021-45046. It covers all 4 releases as they contain the same changes (upgrading Log4j to 2.16.0) and were prepared simultaneously by the same person. (Hence, if something is broken, it likely applies to all releases) Note: 1.11/1.12 are still missing the Python Mac releases. Please review and vote on the release candidate #1 for the versions 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows: [ ] +1, Approve the releases [ ] -1, Do not approve the releases (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source releases and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint C2EED7B111D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tags [5], * website pull request listing the new releases and adding announcement blog post [6]. The vote will be open for at least 24 hours. The minimum vote time has been shortened as the changes are minimal and the matter is urgent. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Chesnay [1] 1.11: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351056 1.12: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351057 1.13: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351058 1.14: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351059 [2] 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/ 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/ 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/ 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 1.11: https://repository.apache.org/content/repositories/orgapacheflink-1460 1.12: https://repository.apache.org/content/repositories/orgapacheflink-1462 1.13: https://repository.apache.org/content/repositories/orgapacheflink-1459 1.14: https://repository.apache.org/content/repositories/orgapacheflink-1461 [5] 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1 [6] https://github.com/apache/flink-web/pull/489
[jira] [Created] (FLINK-25322) Support no-claim mode in changelog state backend
Dawid Wysakowicz created FLINK-25322: Summary: Support no-claim mode in changelog state backend Key: FLINK-25322 URL: https://issues.apache.org/jira/browse/FLINK-25322 Project: Flink Issue Type: Sub-task Components: Runtime / Checkpointing, Runtime / State Backends Reporter: Dawid Wysakowicz Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25321) standalone deploy on k8s,pod always OOM killed,actual heap memory usage is normal, gc is normal
Gao Fei created FLINK-25321: --- Summary: standalone deploy on k8s,pod always OOM killed,actual heap memory usage is normal, gc is normal Key: FLINK-25321 URL: https://issues.apache.org/jira/browse/FLINK-25321 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Affects Versions: 1.11.3 Environment: Flink 1.11.3 k8s v1.21.0 standlone deployment Reporter: Gao Fei Start a cluster on k8s, deploy in standalone mode, a jobmanager pod (1G) and a taskmanager pod (3372MB limit), the total memory configuration of the Flink TM process is 3072MB, and the managed configuration is 0, both of which are on the heap memory. Now the pod It will always be OOM killed, and the total process memory will always exceed 3072MB. I saw that the system has adopted jemlloc. There is no 64M problem. The application itself has not applied for direct memory. It is strange why the process is always killed by OOM after a period of time. INFO [] - Final TaskExecutor Memory configuration: INFO [] - Total Process Memory: 3.000gb (3221225472 bytes) INFO [] - Total Flink Memory: 2.450gb (2630667464 bytes) INFO [] - Total JVM Heap Memory: 2.080gb (2233382986 bytes) INFO [] - Framework: 128.000mb (134217728 bytes) INFO [] - Task: 1.955gb (2099165258 bytes) INFO [] - Total Off-heap Memory: 378.880mb (397284478 bytes) INFO [] - Managed: 0 bytes INFO [] - Total JVM Direct Memory: 378.880mb (397284478 bytes) INFO [] - Framework: 128.000mb (134217728 bytes) INFO [] - Task: 0 bytes INFO [] - Network: 250.880mb (263066750 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 307.200mb (322122552 bytes) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25320) The flink history web page shows the number of running jobs instead of just showing the number of completed jobs
Zhuang Liu created FLINK-25320: --- Summary: The flink history web page shows the number of running jobs instead of just showing the number of completed jobs Key: FLINK-25320 URL: https://issues.apache.org/jira/browse/FLINK-25320 Project: Flink Issue Type: New Feature Components: Runtime / Web Frontend Affects Versions: shaded-14.0 Reporter: Zhuang Liu -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] FLIP-190: Support Version Upgrades for Table API & SQL Programs
+1 (binding) Thanks for driving this much needed feature! On 14.12.21 17:45, Timo Walther wrote: Hi everyone, I'd like to start a vote on FLIP-190: Support Version Upgrades for Table API & SQL Programs [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/x/KZBnCw [2] https://lists.apache.org/thread/n8v32j6o3d50mpblxydbz82q1q436ob4 Cheers, Timo
[jira] [Created] (FLINK-25319) Quickstarts Scala nightly end-to-end test failed due to akka rpc server failed to start
Yun Gao created FLINK-25319: --- Summary: Quickstarts Scala nightly end-to-end test failed due to akka rpc server failed to start Key: FLINK-25319 URL: https://issues.apache.org/jira/browse/FLINK-25319 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.15.0 Reporter: Yun Gao {code:java} Dec 14 17:36:15 at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[?:?] Dec 14 17:36:15 at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_292] Dec 14 17:36:15 at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_292] Dec 14 17:36:15 at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_292] Dec 14 17:36:15 at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) ~[?:1.8.0_292] Dec 14 17:36:15 Caused by: java.lang.NullPointerException Dec 14 17:36:15 at org.apache.flink.runtime.rpc.RpcEndpoint.getAddress(RpcEndpoint.java:283) ~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT] Dec 14 17:36:15 at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:177) ~[?:?] Dec 14 17:36:15 at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ~[?:?] Dec 14 17:36:15 at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) ~[?:?] Dec 14 17:36:15 at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT] Dec 14 17:36:15 at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT] Dec 14 17:36:15 at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) ~[?:?] Dec 14 17:36:15 at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT] Dec 14 17:36:15 at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) ~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT] Dec 14 17:36:15 at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) ~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT] Dec 14 17:36:15 at akka.actor.Actor.aroundReceive(Actor.scala:537) ~[?:?] Dec 14 17:36:15 at akka.actor.Actor.aroundReceive$(Actor.scala:535) ~[?:?] Dec 14 17:36:15 at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) ~[?:?] Dec 14 17:36:15 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) ~[?:?] Dec 14 17:36:15 at akka.actor.ActorCell.invoke(ActorCell.scala:548) ~[?:?] Dec 14 17:36:15 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) ~[?:?] Dec 14 17:36:15 at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[?:?] Dec 14 17:36:15 at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[?:?] Dec 14 17:36:15 at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_292] Dec 14 17:36:15 at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_292] Dec 14 17:36:15 at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_292] Dec 14 17:36:15 at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) ~[?:1.8.0_292] Dec 14 17:36:15 2021-12-14 17:35:43,572 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Shutting StandaloneSessionClusterEntrypoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally.. Dec 14 17:36:15 2021-12-14 17:35:43,582 INFO org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB server at 0.0.0.0:42449 Dec 14 17:36:15 2021-12-14 17:35:43,584 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Shutting down rest endpoint. Dec 14 17:36:15 2021-12-14 17:35:40,722 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Dec 14 17:36:15 2021-12-14 17:35:40,724 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Preconfiguration: Dec 14 17:36:15 2021-12-14 17:35:40,724 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - {code} https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28117&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=3955 -- This message was sent by Atlassian Jira (v8.20.1#820001)