[jira] [Created] (FLINK-17140) AsyncLookupJoinITCase.testAsyncJoinTemporalTableOnMultiFieldsWithUdf failed on Azure

2020-04-14 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-17140: - Summary: AsyncLookupJoinITCase.testAsyncJoinTemporalTableOnMultiFieldsWithUdf failed on Azure Key: FLINK-17140 URL: https://issues.apache.org/jira/browse/FLINK-17140

[jira] [Created] (FLINK-17075) Add task status reconciliation between TM and JM

2020-04-09 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-17075: - Summary: Add task status reconciliation between TM and JM Key: FLINK-17075 URL: https://issues.apache.org/jira/browse/FLINK-17075 Project: Flink Issue

Re: [DISCUSS] Creating a new repo to host Flink benchmarks

2020-04-09 Thread Till Rohrmann
I think it is a good idea to make the benchmarks available to the community via a repo under the Apache project and to make updating it part of the release process. Hence +1 for the proposal. Cheers, Till On Thu, Apr 9, 2020 at 4:01 PM Piotr Nowojski wrote: > Hi Yun Tang, > > Thanks for

Re: Configuring autolinks to Flink JIRA ticket in github repos

2020-04-09 Thread Till Rohrmann
to > > >>>> the > > >>>>> commit details page[1] to see all Flink JIRA titles within commits > > has > > >>>> the > > >>>>> hyper link underline. Moreover, you don't need to use markdown > > language > > >>

[jira] [Created] (FLINK-17073) Slow checkpoint cleanup causing OOMs

2020-04-09 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-17073: - Summary: Slow checkpoint cleanup causing OOMs Key: FLINK-17073 URL: https://issues.apache.org/jira/browse/FLINK-17073 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-17072) Let Dispatcher and ResourceManager pick random endpoint id

2020-04-09 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-17072: - Summary: Let Dispatcher and ResourceManager pick random endpoint id Key: FLINK-17072 URL: https://issues.apache.org/jira/browse/FLINK-17072 Project: Flink

Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released

2020-04-08 Thread Till Rohrmann
Great news! Thanks a lot for being our release manager Gordon and to everyone who helped with the release. Cheers, Till On Wed, Apr 8, 2020 at 3:57 AM Congxian Qiu wrote: > Thanks a lot for the release and your great job, Gordon! > Also thanks to everyone who made this release possible! > >

Re: [VOTE] FLIP-111: Docker image unification

2020-04-07 Thread Till Rohrmann
Thanks for driving this effort Andrey. +1 (binding) Cheers, Till On Tue, Apr 7, 2020 at 4:48 PM Ufuk Celebi wrote: > Thanks for starting this FLIP. > > +1 > > On Tue, Apr 7, 2020 at 11:29 AM Andrey Zagrebin > wrote: > > > Hi All, > > > > As discussed in these threads [1] and [2], > > we

Re: [ANNOUNCE] New Flink committer: Seth Wiesman

2020-04-07 Thread Till Rohrmann
Congrats Seth! Cheers, Till On Tue, Apr 7, 2020 at 9:50 AM Danny Chan wrote: > Congratulations ~ > > Best, > Danny Chan > 在 2020年4月7日 +0800 PM2:56,Congxian Qiu ,写道: > > Congratulations, Seth! > > > > Best, > > Congxian > > > > > > Kurt Young 于2020年4月7日周二 下午2:53写道: > > > > > Congratulations,

Re: [DISCUSS] Consolidated log4j2-properties file

2020-04-06 Thread Till Rohrmann
Hi Chesnay, thanks for kicking this discussion off. I agree that deduplicating code is in general a good idea. The main benefit seems to be that all modules inherit a log4j2-test.properties file and that this file allows to control the logging output for several modules. The main drawback I see

Re: [DISCUSS] FLIP-111: Docker image unification

2020-04-06 Thread Till Rohrmann
in their custom > > > image which extends our base image. > > > > > > # Logging > > > > > > ## Web UI > > > Modify the *log4j-console.properties* to also output logs into the > files > > > for WebUI. Limit log file size. > > >

Re: [DISCUSS] FLIP-111: Docker image unification

2020-04-03 Thread Till Rohrmann
Hi everyone, just a small inline comment. On Fri, Apr 3, 2020 at 11:42 AM Ufuk Celebi wrote: > Hey Yang, > > thanks! See inline answers. > > On Fri, Apr 3, 2020 at 5:11 AM Yang Wang wrote: > > > Hi Ufuk, > > > > Thanks for make the conclusion and directly point out what need to be > done > >

Re: [PSA] Please report all occurrences of test failures

2020-04-03 Thread Till Rohrmann
Thanks for this reminder Aljoscha. Big +1 for following this practice. If you want to make the query a bit more generic (not only searching the summary field), you could also use project = FLINK AND resolution = Unresolved AND text ~ "FooBarTest*" ORDER BY priority DESC, updated DESC That way

Re: [ANNOUNCE] New Committers and PMC member

2020-04-02 Thread Till Rohrmann
Congratulations to all of you! Cheers, Till On Thu, Apr 2, 2020 at 7:57 AM Biao Liu wrote: > Congrats! > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Thu, 2 Apr 2020 at 11:59, Yuan Mei wrote: > > > Congrats :-) > > > > On Wed, Apr 1, 2020 at 4:52 PM Stephan Ewen wrote: > > > > > Hi all! > > > > > >

Re: Configuring autolinks to Flink JIRA ticket in github repos

2020-04-02 Thread Till Rohrmann
Nice, this is a cool feature. Thanks for asking INFRA for it. Cheers, Till On Wed, Apr 1, 2020 at 6:52 PM Yun Tang wrote: > Hi community. > > I noticed that Github supports autolink reference recently [1]. This is > helpful to allow developers could open Jira ticket link from pull requests >

Re: [VOTE] FLIP-108: Add GPU support in Flink

2020-04-02 Thread Till Rohrmann
Thanks for driving this effort Yangze. +1 Cheers, Till On Wed, Apr 1, 2020 at 12:41 PM Canbin Zheng wrote: > Thanks Yangze for driving the initial CPU support! > +1 (non-binding) from my side. > > > Xintong Song 于2020年4月1日周三 下午6:36写道: > > > Thanks Yangze, the FLIP looks good to me. > > +1

Re: Question about the ReadableConfigToConfigurationAdapter

2020-04-02 Thread Till Rohrmann
Hi Canbin, this looks indeed like a bug to me. I'm pulling in Dawid who worked on this part and might be able to tell us more about it. If he agrees, then I would suggest to open a JIRA issue and to fix it. Cheers, Till On Wed, Apr 1, 2020 at 11:26 AM Canbin Zheng wrote: > Hi everyone, > >

Re: [VOTE] FLIP-119: Pipelined Region Scheduling

2020-04-02 Thread Till Rohrmann
+1 Cheers, Till On Tue, Mar 31, 2020 at 5:52 PM Gary Yao wrote: > Hi all, > > I would like to start the vote for FLIP-119 [1], which is discussed and > reached a consensus in the discussion thread [2]. > > The vote will be open until April 3 (72h) unless there is an objection > or not enough

Re: [DISCUSS] FLIP-118: Improve Flink’s ID system

2020-03-31 Thread Till Rohrmann
hink > > > > > you could set the "deployment.meta.name". Since the pod name is > > > generated > > > > > by > > > > > K8s in the pattern {deployment.meta.nane}-{rc.uuid}-{uuid}. On the > > > > > contrary, we

[jira] [Created] (FLINK-16866) Make job submission non-blocking

2020-03-30 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16866: - Summary: Make job submission non-blocking Key: FLINK-16866 URL: https://issues.apache.org/jira/browse/FLINK-16866 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-16867) Simplify default timeout configuration

2020-03-30 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16867: - Summary: Simplify default timeout configuration Key: FLINK-16867 URL: https://issues.apache.org/jira/browse/FLINK-16867 Project: Flink Issue Type

Re: [ANNOUNCE] Flink on Zeppelin (Zeppelin 0.9 is released)

2020-03-30 Thread Till Rohrmann
This is great news Jeff! Thanks a lot for sharing it with the community. Looking forward trying Flink on Zeppelin out :-) Cheers, Till On Mon, Mar 30, 2020 at 2:47 PM Jeff Zhang wrote: > Hi Folks, > > I am very excited to announce the integration work of flink on apache > zeppelin notebook is

Re: [DISCUSS] FLIP-108: Add GPU support in Flink

2020-03-30 Thread Till Rohrmann
> 'external-resource.{resourceName}.yarn/kubernetes.key/amount', > > we can still fulfill that purpose, without the credential risks. > > > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Mon, Mar 30, 2020 at 5:17 PM Till R

Re: [DISCUSS] Introduce a new module 'flink-hadoop-utils'

2020-03-30 Thread Till Rohrmann
Hi Sivaprasanna, thanks for starting this discussion. In general I like the idea to remove duplications and move common code to a shared module. As a recommendation, I would exclude the whole part about Flink's Hadoop compatibility modules because they are legacy code and hardly used anymore.

Re: [DISCUSS] FLIP-108: Add GPU support in Flink

2020-03-30 Thread Till Rohrmann
tractKubernetesStepDecorator` directly manipulates on `Pod`) to > user > > > > codes. It gives more access to user codes than needed for defining > > > external > > > > resource, which might cause problems. Instead, I would suggest to > have > > > > in

[jira] [Created] (FLINK-16836) Losing leadership does not clear rpc connection in JobManagerLeaderListener

2020-03-27 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16836: - Summary: Losing leadership does not clear rpc connection in JobManagerLeaderListener Key: FLINK-16836 URL: https://issues.apache.org/jira/browse/FLINK-16836

[jira] [Created] (FLINK-16834) Examples cannot be run from IDE

2020-03-27 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16834: - Summary: Examples cannot be run from IDE Key: FLINK-16834 URL: https://issues.apache.org/jira/browse/FLINK-16834 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-119: Pipelined Region Scheduling

2020-03-27 Thread Till Rohrmann
Thanks for creating this FLIP Zhu Zhu and Gary! +1 for adding pipelined region scheduling. Concerning the extended SlotProvider interface I have an idea how we could further improve it. If I am not mistaken, then you have proposed to introduce the two timeouts in order to distinguish between

Re: [DISCUSS] FLIP-118: Improve Flink’s ID system

2020-03-26 Thread Till Rohrmann
Hi Yangze, thanks for creating this FLIP. I think it is a very good improvement helping our users and ourselves understanding better what's going on in Flink. Creating the ResourceIDs with host information/pod name is a good idea. Also deriving ExecutionGraph IDs from their superset ID is a

Re: [DISCUSS] Creating a new repo to host Stateful Functions Dockerfiles

2020-03-26 Thread Till Rohrmann
+1 for a separate repository. Cheers, Till On Thu, Mar 26, 2020 at 5:13 PM Ufuk Celebi wrote: > +1. > > The repo creation process is a light-weight, automated process on the ASF > side. When Patrick Lucas contributed docker-flink back to the Flink > community (as flink-docker), there was

Re: [DISCUSS] FLIP-108: Add GPU support in Flink

2020-03-26 Thread Till Rohrmann
Hi everyone, I'm a bit late to the party. I think the current proposal looks good. Concerning the ExternalResourceDriver interface defined in the FLIP [1], I would suggest to not include the decorator calls for Kubernetes and Yarn in the base interface. Instead I would suggest to segregate the

Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / switch off Travis

2020-03-26 Thread Till Rohrmann
Thanks for driving this effort Robert. I'd be in favour of disabling Travis for PRs once AZP is decently stable. Cheers, Till On Wed, Mar 25, 2020 at 8:28 PM Robert Metzger wrote: > Thank you for your responses. > > @Yu Li: In the current master, the log upload always fails, if the e2e job >

Re: [Dev Blog] Migrating Flink's CI Infrastructure from Travis CI to Azure Pipelines

2020-03-26 Thread Till Rohrmann
Thanks Robert for this dev blog post. It's a good read. Cheers, Till On Mon, Mar 23, 2020 at 10:24 PM Arvid Heise wrote: > Thank you Robert! (also thanks for incorporating my feedback so swiftly) > > On Mon, Mar 23, 2020 at 8:54 PM Seth Wiesman wrote: > > > Very interesting! No questions but

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

2020-03-26 Thread Till Rohrmann
LIP-102: > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager > POC web: > > http://101.132.122.69:8081/web/#/task-manager/8e1f1beada3859ee8e46d0960bb1da18/metrics > > Till Rohrmann 于2020年2月27日周四 下午10:27写道: > > > Thinking a bit more ab

[jira] [Created] (FLINK-16751) Expose bind port for Flink metric query service

2020-03-24 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16751: - Summary: Expose bind port for Flink metric query service Key: FLINK-16751 URL: https://issues.apache.org/jira/browse/FLINK-16751 Project: Flink Issue Type

Re: [DISCUSS] JMX remote monitoring integration with Flink

2020-03-20 Thread Till Rohrmann
Hi Rong Rong, you are right that it JMX is quite hard to use in production due to the mentioned problems with discovering the port. There is actually already a JIRA ticket [1] discussing this problem. It just never gained enough traction to be tackled. In general, I agree that it would be nice

Re: [DISCUSS] Link Stateful Functions from the Flink Website

2020-03-20 Thread Till Rohrmann
+1 for the list of proposed changes - What is "Stateful Functions"? link - Documentation -> Stateful Functions Docs - StateFun on GitHub Cheers, Till On Thu, Mar 12, 2020 at 12:22 PM Stephan Ewen wrote: > Thank you all, for chiming in! > > I like the ideas suggested to update the

Re: Flink YARN app terminated before the client receives the result

2020-03-20 Thread Till Rohrmann
Weike > > On Tue, Mar 17, 2020 at 11:11 PM tison wrote: > >> JIRA created as https://jira.apache.org/jira/browse/FLINK-16637 >> >> Best, >> tison. >> >> >> Till Rohrmann 于2020年3月17日周二 下午5:57写道: >> >>> @Tison could you create an

[jira] [Created] (FLINK-16684) StreamingFileSink builder does not work with Scala

2020-03-19 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16684: - Summary: StreamingFileSink builder does not work with Scala Key: FLINK-16684 URL: https://issues.apache.org/jira/browse/FLINK-16684 Project: Flink Issue

Re: [DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

2020-03-19 Thread Till Rohrmann
er complications > for > > > users. > > > I will adjust the FLIP. > > > > > > > change the default value of JM Metaspace size to 256 MB > > > > > > Indeed, no reason to assume that the user code would need less > Metaspace >

Re: [VOTE] FLIP 116: Unified Memory Configuration for Job Managers

2020-03-19 Thread Till Rohrmann
+1 (binding) Cheers, Till On Thu, Mar 19, 2020 at 7:29 AM Yang Wang wrote: > +1 (non-binding). > > This FLIP will make jobmanager memory configuration keep up with > taskmanager. > > > Best, > Yang > > Xintong Song 于2020年3月19日周四 上午11:23写道: > > > +1 (non-binding) from my side. > > > > Thank

Re: [DISCUSS] FLIP-111: Docker image unification

2020-03-18 Thread Till Rohrmann
Thanks for creating this FLIP Andrey. I like the general idea pretty much. I tried to group some of the above-mentioned points in order to give my 50 cent. # Configuration How to configure the Flink process seems to be the biggest question. Due to historical reasons we have a plethora of

Re: [DISCUSS] Releasing Flink 1.10.1

2020-03-17 Thread Till Rohrmann
+1 for a soonish bug fix release. Thanks for volunteering as our release manager Yu. I think we can soon merge the increase of metaspace size and improving the error message. The assumption is that we currently don't have too many small Flink 1.10 deployments with a process size <= 1GB. Of

Re: [DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

2020-03-17 Thread Till Rohrmann
Thanks for creating this FLIP Andrey. I agree with Xintong that we should rename jobmanager.memory.direct.size into jobmanager.memory.off-heap.size which accounts for native and direct memory usage. I think it should be good enough and is easier to understand for the user. Concerning the default

Re: [DISCUSS] Extend (or maintain) "shell" script support for Windows

2020-03-17 Thread Till Rohrmann
+1 for removing them. On Wed, Mar 11, 2020 at 10:06 AM Chesnay Schepler wrote: > +1 on removing them. > > They are so limited in terms of functionality that I doubt anyone would > be significantly impaired by us removing them. > > On 11/03/2020 02:13, Xintong Song wrote: > > Thanks for the

Re: [PROPOSAL] Reverse the dependency from flink-streaming-java to flink-client

2020-03-17 Thread Till Rohrmann
I'm a bit late to the party but also +1 from my side. Pulling the dependency graph straight is very good idea and will improve the maintainability in the long run. Cheers, Till On Tue, Mar 10, 2020 at 5:21 AM tison wrote: > Thanks for your attention! > > Best, > tison. > > > Aljoscha Krettek

[jira] [Created] (FLINK-16635) Incompatible okio dependency in flink-metrics-influxdb module

2020-03-17 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16635: - Summary: Incompatible okio dependency in flink-metrics-influxdb module Key: FLINK-16635 URL: https://issues.apache.org/jira/browse/FLINK-16635 Project: Flink

Re: Flink YARN app terminated before the client receives the result

2020-03-17 Thread Till Rohrmann
g strange that this behavior almost reproducible, it > >> should be a possible order but not always. Maybe previous we have to > >> firstly cancel the job which has a long call chain so that it happens we > >> have enough time to delivered the response. > >> >

Re: Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

2020-03-16 Thread Till Rohrmann
If you want to change the max parallelism then you need to take a savepoint and use Flink's state processor API [1] to rewrite the max parallelism by creating a new savepoint from the old one. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html Cheers,

Re: Flink YARN app terminated before the client receives the result

2020-03-16 Thread Till Rohrmann
Hi Weike, could you share the complete logs with us? Attachments are being filtered out by the Apache mail server but it works if you upload the logs somewhere (e.g. https://gist.github.com/) and then share the link with us. Ideally you run the cluster with DEBUG log settings. I assume that you

Re: SerializableHadoopConfiguration

2020-03-16 Thread Till Rohrmann
; > On Thu, Mar 5, 2020 at 4:22 PM Stephan Ewen wrote: > > > > > > > Do we have more cases of "common Hadoop Utils"? > > > > > > > > If yes, does it make sense to create a "flink-hadoop-utils" module > with > > > >

Re: SerializableHadoopConfiguration

2020-03-04 Thread Till Rohrmann
Hi Sivaprasanna, we don't upload the source jars for the flink-shaded modules. However you can build them yourself and install by cloning the flink-shaded repository [1] and then call `mvn package -Dshade-sources`. [1] https://github.com/apache/flink-shaded Cheers, Till On Tue, Mar 3, 2020 at

[jira] [Created] (FLINK-16408) Bind user code class loader to lifetime of a slot

2020-03-03 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16408: - Summary: Bind user code class loader to lifetime of a slot Key: FLINK-16408 URL: https://issues.apache.org/jira/browse/FLINK-16408 Project: Flink Issue

Re: Flink dev blog

2020-03-03 Thread Till Rohrmann
I like the idea. +1 from my side. Potential topics: - Scheduling - Cluster partitions - Memory configuration - Recovery Cheers, Till On Tue, Mar 3, 2020 at 3:56 AM Xintong Song wrote: > Big +1. Thanks for the idea, Arvid. > > I'd be excited to read such blogs. > > And we would also be happy

Re: [VOTE] FLIP-103: Better TM/JM Log Display

2020-02-28 Thread Till Rohrmann
review the pull request > and drive this FLIP forward? > We will start to create PR as soon as possible, thanks! > > Till Rohrmann 于2020年2月25日周二 下午6:03写道: > > > late +1 (binding) > > > > Cheers, > > Till > > > > On Tue, Feb 25, 2020 at 4:46 AM Yadong

Re: [VOTE] FLIP-98: Better Back Pressure Detection

2020-02-28 Thread Till Rohrmann
he pull request > and drive this FLIP forward? > We will start to create PR as soon as possible, thanks! > > Till Rohrmann 于2020年2月25日周二 下午6:12写道: > > > late +1 (binding). > > > > Cheers, > > Till > > > > On Tue, Feb 25, 2020 at 4:55 AM Yadong Xie

[jira] [Created] (FLINK-16346) BlobsCleanupITCase.testBlobServerCleanupCancelledJob fails on Travis

2020-02-28 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16346: - Summary: BlobsCleanupITCase.testBlobServerCleanupCancelledJob fails on Travis Key: FLINK-16346 URL: https://issues.apache.org/jira/browse/FLINK-16346 Project

Re: Flink 1.10 exception : Unable to instantiate java compiler

2020-02-28 Thread Till Rohrmann
Hi, with Flink 1.10 we changed the behaviour on the client side so that it also uses the child first class loader [1]. Due to that it might be the case that you have some conflicting dependencies bundled in your user code jar which don't play well together with what you have on the system class

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

2020-02-27 Thread Till Rohrmann
ried) > > if anyone has a better idea, please feels no hesitates to help me > > > > 4. What does JVM limit mean in Non-heap.JVM-Overhead? > > JVM limit is "non-heap max metric minus metaspace configuration" as > @Xintong > Song replyed in this mail thread > >

Re: [VOTE] FLIP-100: Add Attempt Information

2020-02-27 Thread Till Rohrmann
are ok for me. What do you think about it? > > Till Rohrmann 于2020年2月26日周三 下午10:53写道: > > > Fair enough. If this should become a problem we could introduce it later > as > > well. > > > > What about changing the SubtasksTimeInfo response type into an

Re: [VOTE] FLIP-100: Add Attempt Information

2020-02-26 Thread Till Rohrmann
he UI to help users filter the > desired attempt, and all the filtering process is running inside the > browser side, what do you think about this? > > > > > Till Rohrmann 于2020年2月25日周二 下午11:40写道: > > > Hi Yadong, > > > > thanks for creating this FLIP. I li

Re: [VOTE] FLIP-104: Add More Metrics to Jobmanager

2020-02-25 Thread Till Rohrmann
Hi Yadong, thanks for creating this FLIP. I like the idea of exposing more cluster information to the user. I share Xintong's concerns that we are about to rework the cluster entrypoint's memory management. It might make sense to wait for these changes before starting this effort. Otherwise, we

Re: [VOTE] FLIP-100: Add Attempt Information

2020-02-25 Thread Till Rohrmann
Hi Yadong, thanks for creating this FLIP. I like the idea to make the web-ui information richer wrt to subtask attempt information. I have a comment concerning the SubtasksTimesHandler: Should we change the response type SubtasksTimeInfo so that it simply contains an array of SubtaskTimeInfo?

Re: [VOTE] FLIP-101: Add Pending Slots Detail

2020-02-25 Thread Till Rohrmann
Hi Yadong, thanks for creating this FLIP. I like the idea to show the user if some slots are missing in order to run an operator. However, since we plan to rework the scheduling part and also how resources are acquired extensively, I'm not entirely sure whether we should add this functionality

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

2020-02-25 Thread Till Rohrmann
Thanks for creating this FLIP Yadong. I think your proposal makes it much easier for the user to understand what's happening on Flink TaskManager's. I have some comments: 1. Some of the newly introduced metrics involve computations on the TaskManager. I would like to avoid additional

Re: [VOTE] FLIP-98: Better Back Pressure Detection

2020-02-25 Thread Till Rohrmann
late +1 (binding). Cheers, Till On Tue, Feb 25, 2020 at 4:55 AM Yadong Xie wrote: > Thanks all for the votes. > > So far, we have > > - 3 binding +1 votes (Kurt, Jark, zhijiang) > - 4 non-binding +1 votes (Congxian, Wangyang, Benchao, Yun Gao) > - No -1 votes > > The voting time has past and

Re: [VOTE] FLIP-103: Better TM/JM Log Display

2020-02-25 Thread Till Rohrmann
late +1 (binding) Cheers, Till On Tue, Feb 25, 2020 at 4:46 AM Yadong Xie wrote: > Thanks all for the votes. > > So far, we have > > - 4 binding +1 votes (Kurt, Jark, jincheng, Zhu Zhu) > - 5 non-binding +1 votes (Yang Wang, Xintong Song, lining, Yangze, zhenya) > - No -1 votes > > The voting

Re: [VOTE] FLIP-99: Make Max Exception Configurable

2020-02-25 Thread Till Rohrmann
+1 (binding) Cheers, Till On Mon, Feb 24, 2020 at 3:04 AM zoudan wrote: > +1 (non-binding) > > Best, > Dan Zou >

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

2020-02-25 Thread Till Rohrmann
;> > >>>> BTW, the fabric8 kubernetes-client and Kubernetes api server already > has > >>>> the parameters check before starting to create the resources. I think > >>>> the exceptions > >>>> are usually meaning and enough for the users to get

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

2020-02-21 Thread Till Rohrmann
Thanks for starting this discussion Canbin. If I understand your proposal correctly, then you would like to evolve the existing decorator approach so that decorators are monadic and smaller in size and functionality. The latter aspect will allow to reuse them between the client and the cluster.

Re: Flink CEP greedy match of single pattern

2020-02-21 Thread Till Rohrmann
Hi Dominik, you can control FlinkCEP's consumption behaviour via the after match skip strategies [1]. They allow you to control how Flink treats events after a match has occurred. If you are interested in the longest possible window of events exceeding your threshold, then you could also add

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-21 Thread Till Rohrmann
Congratulations Jingsong! Cheers, Till On Fri, Feb 21, 2020 at 4:03 PM Yun Gao wrote: > Congratulations Jingsong! > >Best, >Yun > > -- > From:Jingsong Li > Send Time:2020 Feb. 21 (Fri.) 21:42 > To:Hequn Cheng >

Re: [DISCUSS] Kicking off the 1.11 release cycle

2020-02-19 Thread Till Rohrmann
Thanks for volunteering as one of our release managers Zhijiang. +1 for the *anticipated feature freeze date* end of April. As we go along and collect more data points we might be able to strengthen our initial anticipation. Cheers, Till On Wed, Feb 19, 2020 at 4:44 AM Zhijiang wrote: >

Re: [DISCUSS] Improvements on FLIP Process

2020-02-19 Thread Till Rohrmann
+1 for Aljoscha's proposal. This, of course, does not mean that one cannot use Google docs in order to prepare the FLIP discussion. Cheers, Till On Wed, Feb 19, 2020 at 11:15 AM Xintong Song wrote: > I'm also not a fun of discussing FLIPs with google docs. > > I think google docs is probably

Re: [ANNOUNCE] Apache Flink-shaded 10.0 released

2020-02-19 Thread Till Rohrmann
Thanks for making the release possible Chesnay and everyone who was involved! Cheers, Till On Wed, Feb 19, 2020 at 7:47 AM jincheng sun wrote: > Thanks a lot for the release Chesnay! > And thanks to everyone who make this release possible! > > Best, > Jincheng > > > Chesnay Schepler

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

2020-02-19 Thread Till Rohrmann
+1 for removing ES2 and updating ES5 in 1.11. +1 for revisiting the removal of ES5 for 1.12. Cheers, Till On Tue, Feb 18, 2020 at 1:28 PM Chesnay Schepler wrote: > Since one of the reasons for dropping ES2 was that it blocks some > critical updates for the ES5 connector I'd prefer to keep ES5

Re: [Discuss] Update the pull request description template.

2020-02-19 Thread Till Rohrmann
I actually wanted to second Chesnay but apparently my impression is a bit wrong. Out of the last 10 closed PRs (admittedly a small sample size) only 2 did not fill out the template. I did not check for correctness though. Assuming that people use the template, I believe it is a good idea to

Re: Total recovery time estimation after checkpoint recovery

2020-02-19 Thread Till Rohrmann
____ > From: Till Rohrmann > Sent: Tuesday, February 18, 2020 4:43 PM > To: dev > Subject: Re: Total recovery time estimation after checkpoint recovery > > Hi Jessica, > > did you receive my previous email with the explanation? > > Cheers, >

Re: Hotfixes on the master

2020-02-18 Thread Till Rohrmann
gt; > > > Is our Jira and PR process sufficiently agile to handle these hotfixes? > > > Will committers simply include hotfixes with other PRs, and would it be > > > better to retain these as smaller, separate commits? > > > > > > For these cosmetic changes and s

[jira] [Created] (FLINK-16156) TableAggregateITCase.testGroupByFlatAggregate failed on Travis

2020-02-18 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-16156: - Summary: TableAggregateITCase.testGroupByFlatAggregate failed on Travis Key: FLINK-16156 URL: https://issues.apache.org/jira/browse/FLINK-16156 Project: Flink

Re: Total recovery time estimation after checkpoint recovery

2020-02-18 Thread Till Rohrmann
Hi Jessica, did you receive my previous email with the explanation? Cheers, Till On Sat, Feb 15, 2020 at 11:45 PM Woods, Jessica Hui < jessica.wo...@campus.tu-berlin.de> wrote: > ??Hi, > > I am working with Apache Flink and am interested in knowing how one could > estimate the total amount of

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

2020-02-18 Thread Till Rohrmann
Thanks for updating the 1.9.2 release wrt Flink's Python API Jincheng! Cheers, Till On Thu, Feb 13, 2020 at 12:25 PM Hequn Cheng wrote: > Thanks a lot for the release, Jincheng! > Also thanks to everyone that make this release possible! > > Best, > Hequn > > On Thu, Feb 13, 2020 at 2:18 PM

Re: Total recovery time estimation after checkpoint recovery

2020-02-14 Thread Till Rohrmann
Hi Jessica, multiple factors affect the total recovery time. First of all, Flink needs to detect that something went wrong. In the worst case this happens through the missing heartbeat of a died machine. The default heartbeat value is configured to 50s but one can tune it. Next, Flink needs to

Re: [DISCUSS] FLINK-15831: Add Docker image publication to release documentation

2020-02-09 Thread Till Rohrmann
Sounds good to me Patrick. +1 for these changes. Cheers, Till On Fri, Feb 7, 2020 at 3:25 PM Patrick Lucas wrote: > Hi all, > > For FLINK-15831[1], I think the way to start is for the flink-docker > repo[2] itself to sufficiently document the workflow for publishing new > Dockerfiles, and then

Re: [DISCUSS] FLIP-75: Flink Web UI Improvement Proposal

2020-02-09 Thread Till Rohrmann
new practice if > possible. > > Till Rohrmann 于2020年2月3日周一 下午6:28写道: > > > I think there is no such description because we never did it before. I > just > > figured that FLIP-75 could actually be a good candidate to start this > > practice. We would need a

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

2020-02-06 Thread Till Rohrmann
on reducing the amount of jars we ship with the distribution. >> >> While there was no definitive conclusion there was a shared sentiment >> that APIs should be shipped with the distribution. >> >> On 04/02/2020 17:25, Till Rohrmann wrote: >> >> I think there i

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-02-06 Thread Till Rohrmann
it a bit easier to extract information from the JobGraph. Maybe Aljoscha can chime in on this topic. Cheers, Till On Wed, Feb 5, 2020 at 7:35 PM Gyula Fóra wrote: > @Till Rohrmann > You are completely right that the Atlas hook itself should not live inside > Flink. All ot

Re: [DISCUSS] Release flink-shaded 10.0

2020-02-05 Thread Till Rohrmann
Thanks for starting this discussion Chesnay. +1 for starting a new flink-shaded release. Cheers, Till On Wed, Feb 5, 2020 at 2:10 PM Chesnay Schepler wrote: > Hello, > > I would like to kick off the next release of flink-shaded. The main > feature are new modules that bundle zookeeper, that

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-02-05 Thread Till Rohrmann
Hi Gyula, thanks for starting this discussion. Before diving in the details of how to implement this feature, I wanted to ask whether it is strictly required that the Atlas integration lives within Flink or not? Could it also work if you have tool which receives job submissions, extracts the

[jira] [Created] (FLINK-15924) Detect and log blocking main thread operations

2020-02-05 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-15924: - Summary: Detect and log blocking main thread operations Key: FLINK-15924 URL: https://issues.apache.org/jira/browse/FLINK-15924 Project: Flink Issue Type

[jira] [Created] (FLINK-15911) Flink does not work over NAT

2020-02-04 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-15911: - Summary: Flink does not work over NAT Key: FLINK-15911 URL: https://issues.apache.org/jira/browse/FLINK-15911 Project: Flink Issue Type: Bug

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

2020-02-04 Thread Till Rohrmann
o find ways to support more > > versions. > > On the other hand, the size of the flink-ml libraries now is very > > small(about 246K), so it would not bring much impact on the size of dist. > > > > What do you think? > > > > Best, > > Hequn > > &g

[jira] [Created] (FLINK-15864) Upgrade jackson-databind dependency to 2.10.1 for security reasons

2020-02-03 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-15864: - Summary: Upgrade jackson-databind dependency to 2.10.1 for security reasons Key: FLINK-15864 URL: https://issues.apache.org/jira/browse/FLINK-15864 Project: Flink

Re: [DISCUSS] FLIP-75: Flink Web UI Improvement Proposal

2020-02-03 Thread Till Rohrmann
didn’t find how to create of sub flip at cwiki.apache.org > do you mean to create 9 more FLIPS instead of FLIP-75? > > Till Rohrmann 于2020年1月30日周四 下午11:12写道: > > > Would it be easier if FLIP-75 would be the umbrella FLIP and we would > vote > > on the individual impr

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

2020-02-03 Thread Till Rohrmann
An alternative solution would be to offer the flink-ml libraries as optional dependencies on the download page. Similar to how we offer the different SQL formats and Hadoop releases [1]. [1] https://flink.apache.org/downloads.html Cheers, Till On Mon, Feb 3, 2020 at 10:19 AM Hequn Cheng wrote:

Re: Quick Introduction of myself - Kartheek.

2020-02-03 Thread Till Rohrmann
Hi Kartheek, welcome to the Flink community. The best way to get started is to read the contribution guidelines [1]. [1] https://flink.apache.org/contributing/how-to-contribute.html Cheers, Till On Sun, Feb 2, 2020 at 6:52 AM Kartheek kark wrote: > Good Morning everyone, > I am kartheek,

Re: REST Monitoring Savepoint failed

2020-02-03 Thread Till Rohrmann
At the moment this is the case Ramya. We plan to add the auto scaling feature back again in one of the future Flink versions, though. Cheers, Till On Mon, Feb 3, 2020 at 5:27 AM Ramya Ramamurthy wrote: > Thanks Till Rohrmann for the update. > > So even if we upgrade to the newe

Re: [VOTE] Release 1.10.0, release candidate #1

2020-01-31 Thread Till Rohrmann
Hi everyone, -1, because flink-kubernetes does not have the correct NOTICE file. Here is the issue to track the problem [1]. [1] https://issues.apache.org/jira/browse/FLINK-15837 Cheers, Till On Fri, Jan 31, 2020 at 2:34 PM Xintong Song wrote: > Thanks Gary & Yu, > > +1 (non-binding) from

[jira] [Created] (FLINK-15837) flink-kubernetes module does not have correct NOTIC file

2020-01-31 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-15837: - Summary: flink-kubernetes module does not have correct NOTIC file Key: FLINK-15837 URL: https://issues.apache.org/jira/browse/FLINK-15837 Project: Flink

Re: [ANNOUNCE] Apache Flink 1.9.2 released

2020-01-31 Thread Till Rohrmann
Thanks for being our release manager Hequn and thanks for everyone who made the 1.9.2 release possible. Cheers, Till On Fri, Jan 31, 2020 at 1:46 PM Hequn Cheng wrote: > @jincheng sun That's great. Thank you! > > On Fri, Jan 31, 2020 at 7:57 PM jincheng sun > wrote: > >> Thanks for being the

<    5   6   7   8   9   10   11   12   13   14   >