Questions regarding adaptive scheduler with YARN and application mode

2023-06-26 Thread Leon Xu
Hi Flink users, I am trying to use Adaptive Scheduler to auto scale our Flink streaming jobs (NOT batch job). Our jobs are running on YARN with application mode. There isn't much doc around how adaptive scheduler works. So I have some questions: 1. How does Adaptive Scheduler work with

Identifying a flink dashboard

2023-06-26 Thread Mike Phillips
G'day all, Not sure if this is the correct place but... We have a number of flink dashboards and it is difficult to know what dashboard we are looking at. Is there a configurable way to change the 'Apache Flink Dashboard' heading on the dashboard? Or some other way of uniquely identifying what

Re: Very long launch of the Flink application in BATCH mode

2023-06-26 Thread Brendan Cortez
No, I'm using a collection source + 20 same JDBC lookups + Kafka sink. On Mon, 26 Jun 2023 at 19:17, Yaroslav Tkachenko wrote: > Hey Brendan, > > Do you use a file source by any chance? > > On Mon, Jun 26, 2023 at 4:31 AM Brendan Cortez < > brendan.cortez...@gmail.com> wrote: > >> Hi all! >> >>

Re: Very long launch of the Flink application in BATCH mode

2023-06-26 Thread Yaroslav Tkachenko
Hey Brendan, Do you use a file source by any chance? On Mon, Jun 26, 2023 at 4:31 AM Brendan Cortez wrote: > Hi all! > > I'm trying to submit a Flink Job in Application Mode in the Kubernetes > cluster. > > I see some problems when an application has a big number of operators > (more than 20

Very long launch of the Flink application in BATCH mode

2023-06-26 Thread Brendan Cortez
Hi all! I'm trying to submit a Flink Job in Application Mode in the Kubernetes cluster. I see some problems when an application has a big number of operators (more than 20 same operators) - it freezes for ~6 minutes after *2023-06-21 15:46:45,082 WARN

Re: Flink Table API watermark after a select operation on a table

2023-06-26 Thread feng xiangyu
Hi Eugenio, According to docs[1], there are two ways to define the watermark in a table: 1. Defining in DDL 2. During DataStream-to-Table Conversion In your case, I think could use CREATE TABLE DDL to create a new table from filteredPhasesDurationsTable with watermark. See more in CREATE