Re: Flink job cannot find recover path after using entropy injection for s3 file systems

2021-03-24 Thread Till Rohrmann
There is indeed a ticket which tried to fix it for 1.11. release [1, 2]. Maybe the fix is not working properly. [1] https://issues.apache.org/jira/browse/FLINK-17359 [2] https://github.com/apache/flink/pull/11891 On Wed, Mar 24, 2021 at 12:08 PM Till Rohrmann wrote: > Thanks for look

Re: Flink job cannot find recover path after using entropy injection for s3 file systems

2021-03-24 Thread Till Rohrmann
Thanks for looking into this issue Chenqin. To me this looks like a bug in Flink. I am not entirely sure but somehow the wrapping order might have been changed when using files systems from the plugin system. Maybe Arvid knows about any changes in this area. I think we should open a JIRA ticket

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-22 Thread Till Rohrmann
. So I'm afraid that splitting the package is the > only solution we have if we want to reduce the package size of pyflink. > > [1] https://www.python.org/dev/peps/pep-0427/ > [2] https://realpython.com/python-wheels/#advantages-of-python-wheels > > Best, > Xingbo > > Till

Re: Re: Re: [DISCUSSION] Introduce a separated memory pool for the TM merge shuffle

2021-03-22 Thread Till Rohrmann
R now. Any suggestions about that? > > Best, > Yingjie (Kevin) > > -- > 发件人:Guowei Ma > 日 期:2021年03月09日 17:28:35 > 收件人:曹英杰(北牧) > 抄 送:Till Rohrmann; Stephan Ewen; > dev; user; Xintong Song< > tonysong...@

[jira] [Created] (FLINK-21873) CoordinatedSourceRescaleITCase.testUpscaling fails on AZP

2021-03-19 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21873: - Summary: CoordinatedSourceRescaleITCase.testUpscaling fails on AZP Key: FLINK-21873 URL: https://issues.apache.org/jira/browse/FLINK-21873 Project: Flink

Re: [VOTE] Move Flink ML pipeline API and library code to a separate repository named flink-ml

2021-03-19 Thread Till Rohrmann
+1 Cheers, Till On Fri, Mar 19, 2021 at 9:02 AM Becket Qin wrote: > +1. Thanks for driving the effort, Dong. > > Jiangjie (Becket) Qin > > On Fri, Mar 19, 2021 at 10:07 AM Dong Lin wrote: > > > Hi all, > > > > I would like to start the vote for moving Flink ML pipeline API and > library > >

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-19 Thread Till Rohrmann
> [2] https://pypi.org/search/?q=apache-airflow-*= > > Best, > Xingbo > > Till Rohrmann 于2021年3月17日周三 下午9:22写道: > > > How do other projects solve this problem? > > > > Cheers, > > Till > > > > On Wed, Mar 17, 2021 at 3:45 AM Xingbo Huang w

Re: Flink job cannot find recover path after using entropy injection for s3 file systems

2021-03-18 Thread Till Rohrmann
Rainie > > On Wed, Mar 17, 2021 at 6:38 AM Till Rohrmann > wrote: > > > Hi Rainie, > > > > 1. I think what you need to do is to look for the {job_id} in all the > > possible sub folders of the dev/checkpoints/ folder or you extract the > > entropy from

Re: flink 1.11 class loading question

2021-03-17 Thread Till Rohrmann
Hi Chen, You are right that Flink changed its memory model with Flink 1.10. Now the memory model is better defined and stricter. You can find information about it here [1]. For some pointers towards potential problems please take a look at [2]. What you need to do is to figure out where the

Re: Flink job cannot find recover path after using entropy injection for s3 file systems

2021-03-17 Thread Till Rohrmann
Hi Rainie, 1. I think what you need to do is to look for the {job_id} in all the possible sub folders of the dev/checkpoints/ folder or you extract the entropy from the logs. 2. According to [1] entropy should only be used for the data files and not for the metadata files. The idea was to keep

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-17 Thread Till Rohrmann
How do other projects solve this problem? Cheers, Till On Wed, Mar 17, 2021 at 3:45 AM Xingbo Huang wrote: > Hi Chesnay, > > Yes, in most cases, we can indeed download the required jars in `setup.py`, > which is also the solution I originally thought of reducing the size of > wheel packages.

[jira] [Created] (FLINK-21846) Rethink whether failure of ExecutionGraph creation should directly fail the job

2021-03-17 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21846: - Summary: Rethink whether failure of ExecutionGraph creation should directly fail the job Key: FLINK-21846 URL: https://issues.apache.org/jira/browse/FLINK-21846

Re: [DISCUSS] Move Flink ML pipeline API and library code to a separate repository named flink-ml

2021-03-15 Thread Till Rohrmann
+1 for moving Flink ML to a separate repository. Thanks for driving this discussion and effort Dong! Cheers, Till On Fri, Mar 12, 2021 at 1:19 PM Becket Qin wrote: > Thanks for raising the discussion, Dong. +1 on moving the Flink ML to a > separate repository. > > Machine learning is a big

[jira] [Created] (FLINK-21723) Hadoop integration link is broken on download page

2021-03-10 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21723: - Summary: Hadoop integration link is broken on download page Key: FLINK-21723 URL: https://issues.apache.org/jira/browse/FLINK-21723 Project: Flink Issue

Re: [DISCUSS] Support registering custom JobStatusListeners when scheduling a job

2021-03-10 Thread Till Rohrmann
ether it will introduce any stability problem or potential risks. > > Thanks, > Wenhao > > On Sat, Jan 9, 2021 at 12:31 AM Jeff Zhang wrote: > > > Hi Till, > > > > IIUC for application mode, we already allow to run user code in job > manager >

[jira] [Created] (FLINK-21693) TestStreamEnvironment does not implement executeAsync

2021-03-09 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21693: - Summary: TestStreamEnvironment does not implement executeAsync Key: FLINK-21693 URL: https://issues.apache.org/jira/browse/FLINK-21693 Project: Flink

[jira] [Created] (FLINK-21691) KafkaSource fails with NPE when setting it up

2021-03-09 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21691: - Summary: KafkaSource fails with NPE when setting it up Key: FLINK-21691 URL: https://issues.apache.org/jira/browse/FLINK-21691 Project: Flink Issue Type

Re: [ANNOUCE][DISCUSS] Roadmap Update for Website

2021-03-09 Thread Till Rohrmann
Thanks for creating the draft for the updated roadmap Stephan. It looks good to me. I also like the feature radar. This will help our users to navigate more safely in the growing Flink ecosystem. Cheers, Till On Mon, Mar 8, 2021 at 8:32 PM Stephan Ewen wrote: > Thanks, Robert! > > As far as I

Re: QUESTION: Local Dev Setup

2021-03-09 Thread Till Rohrmann
Hi Jan, if you run mvn install of your modified Flink, then you can simply use the maven artefacts from the local repository. You should then either specify --no-snapshot-updates or assign a different maven version to your Flink build which you can reference from your test job. Cheers, Till On

Re: [DISCUSSION] Introduce a separated memory pool for the TM merge shuffle

2021-03-05 Thread Till Rohrmann
Thanks for this proposal Guowei. +1 for it. Concerning the default size, maybe we can run some experiments and see how the system behaves with different pool sizes. Cheers, Till On Fri, Mar 5, 2021 at 2:45 PM Stephan Ewen wrote: > Thanks Guowei, for the proposal. > > As discussed offline

Re: [VOTE] FLIP-165: Operator's Flame Graphs

2021-03-05 Thread Till Rohrmann
+1 (binding) Cheers, Till On Fri, Mar 5, 2021 at 2:43 PM Alexander Fedulov wrote: > Hi all, > > I would like to start a vote on FLIP-165 [1] which was discussed in [2] and > proposed to be moved to the voting stage. > > The vote will be open until March 11th, 2021 10:00 AM CET. > > [1]

Re: [ANNOUNCE] Apache Flink 1.12.2 released

2021-03-05 Thread Till Rohrmann
Great work! Thanks a lot for being our release managers Roman and Yuan and to everyone who has made this release possible. Cheers, Till On Fri, Mar 5, 2021 at 10:43 AM Yuan Mei wrote: > Cheers! > > Thanks, Roman, for doing the most time-consuming and difficult part of the > release! > > Best,

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

2021-03-05 Thread Till Rohrmann
those data are > >> > sampled, > >> > > > and point to limitations such as: > >> > > > - data from all threads/operators are squashed together, so if > there > >> > is a > >> > > > data skew it w

[jira] [Created] (FLINK-21602) Creation of ExecutionGraph happens in main thread

2021-03-04 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21602: - Summary: Creation of ExecutionGraph happens in main thread Key: FLINK-21602 URL: https://issues.apache.org/jira/browse/FLINK-21602 Project: Flink Issue

[jira] [Created] (FLINK-21580) Remove JobGraph.ScheduleMode

2021-03-03 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21580: - Summary: Remove JobGraph.ScheduleMode Key: FLINK-21580 URL: https://issues.apache.org/jira/browse/FLINK-21580 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-21559) Python DataStreamTests::test_process_function failed on AZP

2021-03-02 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21559: - Summary: Python DataStreamTests::test_process_function failed on AZP Key: FLINK-21559 URL: https://issues.apache.org/jira/browse/FLINK-21559 Project: Flink

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

2021-03-02 Thread Till Rohrmann
Alexander Fedulov | Solutions Architect > > <https://www.ververica.com/> > > Follow us @VervericaData > > > On Tue, Mar 2, 2021 at 12:28 PM Till Rohrmann > wrote: > > > Why does the caller of TaskExecutorGateway.requestThreadInfoSamples need > to > > specif

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

2021-03-02 Thread Till Rohrmann
Why does the caller of TaskExecutorGateway.requestThreadInfoSamples need to specify the request id? Is it because the caller can send a second request with the same id? Or can the caller query the result of a previous request by specifying the requestId? If the TaskExecutor does not need to know

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

2021-03-02 Thread Till Rohrmann
Thanks for starting this discussion Alex. I really like this feature because it gives better insights into what a Flink job is doing. Quick question on the TaskExecutorGateway extension: * What is the requestId used for in the RPC call? * Would it make sense to group numSubSamples,

Re: Watermarks when reading from file

2021-03-01 Thread Till Rohrmann
Hi Dominik, I think the problem could be that TumblingTimeWindows don't start with the timestamp of the first arriving event but start at a multiple of the window length. So when defining a 90 day tumbling window you define a window from 0 - 89, 90 - 179, If your data ranges from day 79 -

Re: [DISCUSS] Enabling more dynamic, or metadata-driven behaviors

2021-03-01 Thread Till Rohrmann
Hi Maciej, The Flink community highly appreciates any kind of feedback and improvement suggestions. W/o going into more detail for the problems you've encountered, it is very likely that other people have run into something similar as well. Hence, it is a good idea to share these limitations and

Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-02-28 Thread Till Rohrmann
t; > FLIP-147 demands no new records from end-of-stream-flushing, but source > will emit “MAX_WATERMARK” on ending. Previously, I thought it is not a > valid issue, but turn out that it could cause trouble under scenario listed > by @Piotr if I am not wrong. > > > PR#14831: https://

Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-02-27 Thread Till Rohrmann
> > Best, > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > --Original Mail -- > > > > > Sender:Guowei Ma > > > > > Send Date:

Re: [DISCUSS] Apache Flink Jira Process

2021-02-26 Thread Till Rohrmann
Thanks for starting this discussion Konstantin. I like your proposal and also the idea of automating the tedious parts of it via a bot. Cheers, Till On Fri, Feb 26, 2021 at 4:17 PM Konstantin Knauf wrote: > Dear Flink Community, > > I would like to start a discussion on improving and to some

[jira] [Created] (FLINK-21521) Pretty print K8s specifications

2021-02-26 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21521: - Summary: Pretty print K8s specifications Key: FLINK-21521 URL: https://issues.apache.org/jira/browse/FLINK-21521 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-21518) CheckpointCoordinatorTest.testMinCheckpointPause fails fatally on AZP

2021-02-26 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21518: - Summary: CheckpointCoordinatorTest.testMinCheckpointPause fails fatally on AZP Key: FLINK-21518 URL: https://issues.apache.org/jira/browse/FLINK-21518 Project

[jira] [Created] (FLINK-21516) UpsertKafkaTableITCase.testTemporalJoin fails on AZP

2021-02-26 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21516: - Summary: UpsertKafkaTableITCase.testTemporalJoin fails on AZP Key: FLINK-21516 URL: https://issues.apache.org/jira/browse/FLINK-21516 Project: Flink Issue

Re: [DISCUSS] Releasing Apache Flink 1.12.2

2021-02-25 Thread Till Rohrmann
gt; > +1 for releasing 1.12.2 and Yuan as the release manager. > > >>> > > > > > >>> > > > Thank you~ > > >>> > > > > > >>> > > > Xintong Song > > >&g

[jira] [Created] (FLINK-21492) ActiveResourceManager swallows exception stack trace

2021-02-24 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21492: - Summary: ActiveResourceManager swallows exception stack trace Key: FLINK-21492 URL: https://issues.apache.org/jira/browse/FLINK-21492 Project: Flink Issue

[jira] [Created] (FLINK-21487) Replace TaskExecutor with TaskManager in ConfigOption description

2021-02-24 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21487: - Summary: Replace TaskExecutor with TaskManager in ConfigOption description Key: FLINK-21487 URL: https://issues.apache.org/jira/browse/FLINK-21487 Project: Flink

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-02-23 Thread Till Rohrmann
Thanks for the explanation Yun and Guowei. I have to admit that I do not fully understand why this is strictly required but I think that we are touching two very important aspects which might have far fetching consequences for how Flink works: 1) Do we want to allow that multiple checkpoints are

Re: [ANNOUNCE] New Apache Flink Committers - Wei Zhong and Xingbo Huang

2021-02-21 Thread Till Rohrmann
Congratulations Wei & Xingbo. Great to have you as committers in the community now. Cheers, Till On Mon, Feb 22, 2021 at 5:08 AM Xintong Song wrote: > Congratulations, Wei & Xingbo~! Welcome aboard. > > Thank you~ > > Xintong Song > > > > On Mon, Feb 22, 2021 at 11:48 AM Dian Fu wrote: > > >

[jira] [Created] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls

2021-02-16 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21383: - Summary: Docker image does not play well together with ConfigMap based flink-conf.yamls Key: FLINK-21383 URL: https://issues.apache.org/jira/browse/FLINK-21383

[jira] [Created] (FLINK-21382) Standalone K8s documentation does not explain usage of standby JobManagers

2021-02-16 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21382: - Summary: Standalone K8s documentation does not explain usage of standby JobManagers Key: FLINK-21382 URL: https://issues.apache.org/jira/browse/FLINK-21382 Project

[jira] [Created] (FLINK-21381) Kubernetes HA documentation does not state required service account and role

2021-02-16 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21381: - Summary: Kubernetes HA documentation does not state required service account and role Key: FLINK-21381 URL: https://issues.apache.org/jira/browse/FLINK-21381

Re: [ANNOUNCE] Randomized tests

2021-02-11 Thread Till Rohrmann
Thanks for letting the community know Arvid. Can we document this specific behaviour somewhere easy to find for others? Maybe we could add it to the Flink wiki or the how to contribute guide. Cheers, Till On Wed, Feb 10, 2021 at 2:44 PM Arvid Heise wrote: > Dear devs, > > I just merged

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Till Rohrmann
Congratulations Roman :-) Cheers, Till On Wed, Feb 10, 2021 at 1:01 PM Konstantin Knauf wrote: > Congratulations Roman! > > On Wed, Feb 10, 2021 at 11:29 AM Piotr Nowojski > wrote: > > > Hi everyone, > > > > I'm very happy to announce that Roman Khachatryan has accepted the > > invitation to

Re: Activate bloom filter in RocksDB State Backend via Flink configuration

2021-02-08 Thread Till Rohrmann
Hi Jun, Making things easier to use and configure is a good idea. Hence, +1 for this proposal. Maybe create a JIRA ticket for it. For the concrete default values it would be nice to hear the opinion of a RocksDB expert. Cheers, Till On Sun, Feb 7, 2021 at 7:23 PM Jun Qin wrote: > Hi, > >

Re: Flink upset-kaka connector not working with Avro confluent

2021-02-08 Thread Till Rohrmann
Hi Shamit, thanks for reaching out to the community. I am pulling in Timo who might know more about this problem. Cheers, Till On Sun, Feb 7, 2021 at 6:22 AM shamit jain wrote: > Hello Team, > > I am facing issue with "upsert-kafka" connector which should read the Avro > message serialized

Re: Flink 1.11.4 release? And usual plan for maintenance etc

2021-02-05 Thread Till Rohrmann
Hi Adam, Currently there are no concrete plans for the 1.11.4 release. Usually the community creates a new release after enough fixes have been merged or if there is enough demand for it. We have indeed accumulated 35 fixes for 1.11.4 already which could justify a new release. Which fixes do you

Re: [DISCUSS] Releasing Apache Flink 1.12.2

2021-02-05 Thread Till Rohrmann
Thanks for volunteering for being our release manager Yuan :-) +1 for a timely bug fix release. I will try to review the PR for FLINK- 20417 [1] which is a good fix to include in the next bug fix release. We don't have to block the release on this fix though. [1]

[RESULT][VOTE] FLIP-160: Declarative scheduler

2021-02-04 Thread Till Rohrmann
-binding) - Yu Li (binding) - Till Rohrmann (binding) There were no -1 votes. Thus, FLIP-160 has been accepted. Thanks everyone for joining the discussion and giving feedback! [1] https://lists.apache.org/thread.html/r5476d63b718bed70784b0d91ef03d28f4c4f6cf4cbe8760c7a7d5d73%40

Re: [VOTE] FLIP-160: Declarative scheduler

2021-02-04 Thread Till Rohrmann
gt; > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 29, 2021 at 10:41 PM Robert Metzger < > > rmetz...@apache.org > > > > > > > > > wrote: &

[jira] [Created] (FLINK-21262) Remove SlotPool.suspend

2021-02-03 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21262: - Summary: Remove SlotPool.suspend Key: FLINK-21262 URL: https://issues.apache.org/jira/browse/FLINK-21262 Project: Flink Issue Type: Bug

Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-03 Thread Till Rohrmann
Hmm, maybe it was deleted again. I received an email notification yesterday. On Wed, Feb 3, 2021 at 11:57 AM Xintong Song wrote: > @Till, > Did I overlook anything? I don't find Galen's post on the old PR. > > Thank you~ > > Xintong Song > > > > On Wed, Feb 3,

Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-03 Thread Till Rohrmann
offering the contribution. > > As Till has already suggested, please comment on FLINK-11838 your solution > proposal. > Once we reach consensus on the proposal, I'll assign you to the ticket. > > Thank you~ > > Xintong Song > > > > On Tue, Feb 2, 2021 at 5:19 PM Ti

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

2021-02-03 Thread Till Rohrmann
om logical slots and SSG, if it > gets > > ResourceSpec.ZERO, it is > > just a default sized slot. > > > > In short, turn SSG resource speciying as "add" and drop > > ResourceSpec.UNKNOWN. > > > > > > Questions/Issues: > > * Could

Re: [DISCUSS] FLINK-21110: Optimize Scheduler Performance for Large-Scale Jobs

2021-02-02 Thread Till Rohrmann
Thanks for making the community aware of these performance improvements, Zhilong. I like them and I am looking forward to a faster Flink :-) Cheers, Till On Tue, Feb 2, 2021 at 11:00 AM Zhilong Hong wrote: > Hello, everyone: > > I would like to start the discussion about FLINK-21110: Optimize

Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-02 Thread Till Rohrmann
Hi Galen, I think that adding support for GCS using the StreamingFileSink sounds like a very good idea to me. Looking at FLINK-11838 I believe that this effort has been abandoned. I think that you could take this ticket over if you want. Maybe you could update this ticket with your solution

Re: [VOTE] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

2021-02-02 Thread Till Rohrmann
+1 (binding) Cheers, Till On Mon, Feb 1, 2021 at 5:38 AM Xintong Song wrote: > +1 (binding) > > Thank you~ > > Xintong Song > > > > On Mon, Feb 1, 2021 at 11:56 AM Yangze Guo wrote: > > > Hi everyone, > > > > I'd like to start the vote of FLIP-156 [1]. This FLIP is discussed in > > the

Re: [VOTE] FLIP-159: Reactive Mode

2021-01-29 Thread Till Rohrmann
LGTM. Thanks for the work Robert! +1 (binding) Cheers, Till On Thu, Jan 28, 2021 at 11:27 AM Yang Wang wrote: > Thanks Robert for your great work on this FLIP. This is really a big step > to make Flink auto scalable. > > +1 (non-binding) > > > Best, > Yang > > Robert Metzger 于2021年1月28日周四

[VOTE] FLIP-160: Declarative scheduler

2021-01-29 Thread Till Rohrmann
Hi all, since the discussion [1] about FLIP-160 [2] seems to have reached a consensus, I'd like to start a formal vote for the FLIP. Please vote +1 to approve the FLIP, or -1 with a comment. The vote will be open at least until Wednesday, Feb 3rd. Cheers, Till [1]

Re: [DISCUSS] FLIP-160: Declarative scheduler

2021-01-29 Thread Till Rohrmann
planations, Till. > >> Keeping the initial design as simple as possible sounds good to me. > >> There's no further concern from my side. > >> > >> Thank you~ > >> > >> Xintong Song > >> > >> > >> > >> On Tue, J

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

2021-01-29 Thread Till Rohrmann
gt; Default requirements are whatever the default requirements are, > setting > > >>> the requirements for one operator has no effect on other operators. > > >>> > > >>> With these rules, and some API enhancements, the following mockup > would &

Re: [ANNOUNCE] Apache Flink 1.10.3 released

2021-01-29 Thread Till Rohrmann
Thanks Xintong for being our release manager. Well done! Cheers, Till On Fri, Jan 29, 2021 at 9:50 AM Yang Wang wrote: > Thanks Xintong for driving this release. > > Best, > Yang > > Yu Li 于2021年1月29日周五 下午3:52写道: > >> Thanks Xintong for being our release manager and everyone else who made >>

Re: [DISCUSS] Removal of flink-swift-fs-hadoop module

2021-01-27 Thread Till Rohrmann
+1, Cheers, Till On Wed, Jan 27, 2021 at 11:03 AM Ufuk Celebi wrote: > +1 > > On Wed, Jan 27, 2021, at 11:02 AM, Dawid Wysakowicz wrote: > > +1 > > > > On 27/01/2021 11:00, Chesnay Schepler wrote: > > > +1 > > > > > > While I don't think the filesystem is really useless, in it's current > > >

[jira] [Created] (FLINK-21155) FileSourceTextLinesITCase.testBoundedTextFileSourceWithTaskManagerFailover does not pass

2021-01-26 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21155: - Summary: FileSourceTextLinesITCase.testBoundedTextFileSourceWithTaskManagerFailover does not pass Key: FLINK-21155 URL: https://issues.apache.org/jira/browse/FLINK-21155

Re: [DISCUSS] FLIP-161: Configuration through environment variables

2021-01-26 Thread Till Rohrmann
Thinking a bit more about the DYNAMIC_PROPERTIES, I have to admit that I like the fact that it works around the problem of encoding the key names and that it is more powerful wrt to bulk changes. Also the fact that one can copy past configuration snippets is quite useful. Given these aspects and

[jira] [Created] (FLINK-21153) yarn-per-job deployment target ignores yarn options

2021-01-26 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21153: - Summary: yarn-per-job deployment target ignores yarn options Key: FLINK-21153 URL: https://issues.apache.org/jira/browse/FLINK-21153 Project: Flink Issue

Re: [DISCUSS] FLIP-160: Declarative scheduler

2021-01-26 Thread Till Rohrmann
On Tue, Jan 26, 2021 at 9:53 AM Zhu Zhu wrote: > Thanks for the proposal! @Till Rohrmann > > The design looks generally good to me. I have 2 concerns though: > > # performance on failover > I can see is that an ExecutionGraph will be built and initialized on each > task

Re: [DISCUSS] FLIP-159: Reactive Mode

2021-01-26 Thread Till Rohrmann
Thanks a lot for all the feedback Steven, Yang Wang and Xintong. I have a few more comments to add. # Keep it simple and stupid As Robert said we would like to keep the new feature initially as simple as possible in order to quickly implement it. Once we have a basic implementation, we want to

Re: [DISCUSS] FLIP-161: Configuration through environment variables

2021-01-26 Thread Till Rohrmann
arding the > "magic > > syntax" (s3.access-key → FLINK_CONFIG_S3_ACCESS__KEY) you mentioned > > earlier? > > > > Regards, > > Roman > > > > > > On Mon, Jan 25, 2021 at 6:48 PM Till Rohrmann > > wrote: > > > > > T

Re: [DISCUSS] FLIP-161: Configuration through environment variables

2021-01-25 Thread Till Rohrmann
r docker >> images we already have a mechanism to support that. >> >> In any case, wouldn't a single DYNAMIC_PROPERTIES variable suffice that >> is automatically evaluated by all scripts? >> Why go through all the hassle with the environment variable names

Re: Apache Pinot Sink

2021-01-25 Thread Till Rohrmann
Hi Mats and Jakob, In the general case, I don't think that elements from upstream Flink tasks always arrive at the same subtask of the sink. One problem is that user computations can be non-deterministic. Moreover, a rebalance operation can distribute the events of a task A among several

[jira] [Created] (FLINK-21129) Long metric names cause Graphite to not store them

2021-01-25 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21129: - Summary: Long metric names cause Graphite to not store them Key: FLINK-21129 URL: https://issues.apache.org/jira/browse/FLINK-21129 Project: Flink Issue

Re: [ANNOUNCE] New formatting rules are now in effect

2021-01-25 Thread Till Rohrmann
t; .forEach( >>>> > (name, entry) -> >>>> > modules.put(name, createModule(entry.asMap(), >>>> > classLoader; >>>> > >>>> > >>>> > >>>> &g

[jira] [Created] (FLINK-21102) Add ScaleUpController

2021-01-22 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21102: - Summary: Add ScaleUpController Key: FLINK-21102 URL: https://issues.apache.org/jira/browse/FLINK-21102 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-21098) Add SlotAllocator

2021-01-22 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21098: - Summary: Add SlotAllocator Key: FLINK-21098 URL: https://issues.apache.org/jira/browse/FLINK-21098 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-21101) Set up cron job to run CI with declarative scheduler

2021-01-22 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21101: - Summary: Set up cron job to run CI with declarative scheduler Key: FLINK-21101 URL: https://issues.apache.org/jira/browse/FLINK-21101 Project: Flink Issue

[jira] [Created] (FLINK-21100) Add DeclarativeScheduler skeleton

2021-01-22 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21100: - Summary: Add DeclarativeScheduler skeleton Key: FLINK-21100 URL: https://issues.apache.org/jira/browse/FLINK-21100 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-21099) Introduce JobType to distinguish between batch and streaming jobs

2021-01-22 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21099: - Summary: Introduce JobType to distinguish between batch and streaming jobs Key: FLINK-21099 URL: https://issues.apache.org/jira/browse/FLINK-21099 Project: Flink

[DISCUSS] FLIP-160: Declarative scheduler

2021-01-22 Thread Till Rohrmann
Hi everyone, I would like to start a discussion about adding a new type of scheduler to Flink. The declarative scheduler will first declare the required resources and wait for them before deciding on the actual parallelism of a job. Thereby it can better handle situations where resources cannot

Re: [DISCUSS] FLIP-161: Configuration through environment variables

2021-01-22 Thread Till Rohrmann
The updated design LGTM as well. Nice work Ingo! Cheers, Till On Fri, Jan 22, 2021 at 3:33 PM Ingo Bürk wrote: > Thanks, Ufuk. I think that makes sense, so I moved it from a footnote to an > addition to prevent that in the future as well. > > Ingo > > On Fri, Jan 22, 2021 at 3:10 PM Ufuk

Re: [ANNOUNCE] 1.12.1 may still produce corrupted checkpoints

2021-01-22 Thread Till Rohrmann
Thanks for the update Arvid. This fix warrants a quick 1.12.2 release imo. Cheers, Till On Fri, Jan 22, 2021 at 1:42 PM Arvid Heise wrote: > Hi Xintong, > > yes, I'm on it. > > Best, > > Arvid > > On Fri, Jan 22, 2021 at 1:01 PM Xintong Song > wrote: > > > Hi Arvid, > > > > Thanks for the

[jira] [Created] (FLINK-21078) Add DeclarativeSlotPoolService

2021-01-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21078: - Summary: Add DeclarativeSlotPoolService Key: FLINK-21078 URL: https://issues.apache.org/jira/browse/FLINK-21078 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-21077) Abstract SlotPool from JobManager by introducing SlotPoolService

2021-01-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21077: - Summary: Abstract SlotPool from JobManager by introducing SlotPoolService Key: FLINK-21077 URL: https://issues.apache.org/jira/browse/FLINK-21077 Project: Flink

[jira] [Created] (FLINK-21076) Document how to use the declarative scheduler and its limitations

2021-01-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21076: - Summary: Document how to use the declarative scheduler and its limitations Key: FLINK-21076 URL: https://issues.apache.org/jira/browse/FLINK-21076 Project: Flink

[jira] [Created] (FLINK-21075) FLIP-160: Declarative scheduler

2021-01-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21075: - Summary: FLIP-160: Declarative scheduler Key: FLINK-21075 URL: https://issues.apache.org/jira/browse/FLINK-21075 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-21074) Document how to use the reactive mode

2021-01-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-21074: - Summary: Document how to use the reactive mode Key: FLINK-21074 URL: https://issues.apache.org/jira/browse/FLINK-21074 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-161: Configuration through environment variables

2021-01-21 Thread Till Rohrmann
Thanks for starting this discussion Ingo! I like the flexibility in terms of supported EV formats the lazy evaluation gives us. However, I am a bit concerned by the fact that it is quite hard to compute the configurational ground truth w/o knowing all requested config keys. Moreover, it makes the

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

2021-01-21 Thread Till Rohrmann
gt; > (And before anyone argues what happens if slotsharing groups change or > whatnot, that's a plain API issue that we could surely solve. (A plain > iteration over slotsharing groups and therein contained operators would > suffice)). > > On 1/20/2021 6:48 PM, Till Rohrmann wrote:

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

2021-01-20 Thread Till Rohrmann
he moment what kind of changes we may want in > > future that might conflict with this FLIP proposal, as the question of > > switching data exchange mode answered above. I'd suggest to not give up > the > > user friendliness we may gain now for the future problems that may or may > &g

Re: Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache Flink Committer

2021-01-20 Thread Till Rohrmann
Congrats, Guowei! Cheers, Till On Wed, Jan 20, 2021 at 8:32 AM Matthias Pohl wrote: > Congrats, Guowei! > > On Wed, Jan 20, 2021 at 8:22 AM Congxian Qiu > wrote: > > > Congrats Guowei! > > > > Best, > > Congxian > > > > > > Danny Chan 于2021年1月20日周三 下午2:59写道: > > > > > Congratulations Guowei!

Re: [Announce] SQL docs are now Blink only

2021-01-19 Thread Till Rohrmann
Awesome. Thanks a lot Seth! This will help us to keep the docs more easily up to date. Cheers, Till On Tue, Jan 19, 2021 at 4:30 PM Seth Wiesman wrote: > Hi Everyone, > > I just merged in a PR to make the SQL / Table docs Blink planner only. > Going forward, you do not need to mark something

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

2021-01-19 Thread Till Rohrmann
essure, etc.) to improve the specification. > > > > > > To sum up: > > > If the user is capable of providing proper resource requirements for > > every > > > operator, that's definitely a good thing and we would not need to rely > on > > > the

Re: [DISCUSS] Flink configuration from environment variables

2021-01-19 Thread Till Rohrmann
and it's > minimally more convenient to have something like > > flinkConfiguration: > foo: ${BAR} > > for us. I don't think this approach would feel natural when manually > deploying Flink. There would be a clear migration path for our customers, > so I'm not concerned abo

Re: Re: [DISCUSS] Dealing with deprecated and legacy code in Flink

2021-01-19 Thread Till Rohrmann
Thanks a lot for starting this discussion Timo. I like the idea of setting more explicit guidelines for deprecating functionality. I really like the idea of adding with the @Deprecated annotation since when the function is deprecated. Based on that one can simply search for features which should

Re: [DISCUSS] Flink configuration from environment variables

2021-01-19 Thread Till Rohrmann
Hi everyone, Thanks for starting this discussion Ingo. I think being able to use env variables to change Flink's configuration will be a very useful feature. Concerning the two approaches I would be in favour of the second approach ($FLINK_CONFIG_S3_ACCESS_KEY) because it does not require the

Re: [ANNOUNCE] Apache Flink 1.12.1 released

2021-01-19 Thread Till Rohrmann
Thanks a lot for driving this release Xintong. This was indeed a release with some obstacles to overcome and you did it very well! Cheers, Till On Tue, Jan 19, 2021 at 5:59 AM Xingbo Huang wrote: > Thanks Xintong for the great work! > > Best, > Xingbo > > Peter Huang 于2021年1月19日周二 下午12:51写道:

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Till Rohrmann
+1, Cheers, Till On Mon, Jan 18, 2021 at 4:12 PM Chesnay Schepler wrote: > +1 > On 1/18/2021 3:50 PM, Seth Wiesman wrote: > > Addendum, 72 hours from now is Thursday the 21st :) > > > > sorry for the mistake. > > > > Seth > > > > On Mon, Jan 18, 2021 at 8:41 AM Timo Walther wrote: > > > >> +1

<    1   2   3   4   5   6   7   8   9   10   >