Re: apache-flink +spring boot - logs related to spring boot application start up not printed in file in flink 1.12

Hi Abhishek, have you also tried to apply the instructions listed in [1]? Best, Matthias [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/advanced/logging.html#configuring-log4j1 On Mon, Mar 1, 2021 at 4:42 AM Abhishek Shukla wrote: > Hi Matthias, > Thanks for replyi

Re: Kubernetes HA - attempting to restore from wrong (non-existing) savepoint

Hi Alexey, It seems that the KubernetesHAService works well since all the checkpoints have been cleaned up when the job is canceled. And we could find related logs "Found 0 checkpoints in KubernetesStateHandleStore{configMapName='gsp--jobmanager-leader'}.". However

Re: apache-flink +spring boot - logs related to spring boot application start up not printed in file in flink 1.12

Hi Matthias, Thanks for replying, I checked both of these pages, And I downloaded the zip of flink 1.12.1 so the changes related to log4j2 are there in property file, I am able to see the logs of pipeline once application in up, but the logs related to application failure or successful bean creati

Re: java Flink local test failure (Could not create actor system)

Hi Vijay, Since version 1.7 Flink builds with Scala version 2.11 (default) and 2.12. Flink has APIs, libraries, and runtime modules written in Scala. Users of the Scala API and libraries may have to match the Scala version of Flink with the Scala version of their projects (because Scala is not str

Re: Union fields with time attributes have different types

Hi Sebastián, `endts` in your case is a time attribute which is slightly different than a regular TIMESTAMP type. You can manually `cast(endts as timestamp(3)` to make this query work which removes the time attribute meta. SELECT `evt`, `value`, `startts`, cast(endts as timestamp(3) FROM aggs_1m

Re: Best way to handle BIGING to TIMESTAMP conversions

Hi Sebastián, You can use `TO_TIMESTAMP(FROM_UNIXTIME(e))` to get a timestamp value. The BIGINT should be in seconds. Please note to declare the computed column in DDL schema and declare a watermark strategy on this computed field to make the field to be a rowtime attribute. Because streaming o

Re: Using Prometheus Client Metrics in Flink

It looks like I was finally able to get the expected labeling behavior that I was looking for by simply storing a reference to the underlying MetricGroup and then keeping track of any new metrics that I needed to dynamically create and use downstream: class MagicMetricRegistry(private val metricGr

Standard method to generate watermark forBoundedOutOfOrderness

Hello, everyone. I'm learning Flink but still not sure if I realise the topic of watermark mechanism. That is a simple common example of pipeline with event-time mode: I want to use

Re: Issues running multiple Jobs using the same JAR

Hi Morgan, You could check FLINK-11654, from its description, I think it is the problem you encountered. > We run multiple jobs on a cluster which write a lot to the same Kafka topic from identically named sinks. When EXACTLY_ONCE semantic is enabled for the KafkaProducers we run into a lot of Pr

Issues running multiple Jobs using the same JAR

Greetings all, I am having an issue instantiating multiple flink jobs uisng the same JAR in the same Flink native cluster (all 1.12.1). When processing events, the jobs fail with the following trace: org.apache.kafka.common.KafkaException: Cannotperform send because at least one previous tra

Re: Stateful functions 2.2 and stop with savepoint

Hi, You could also try `cancel —withSavepoint [savepointDir]` even it is in deprecation. Comparing to take-savepoints and then cancel approach, there will be no checkpoints in between. This may be important if there are two phase commit operators in your job. Best, Kezhu Wang On February 28, 2

Re: Using Prometheus Client Metrics in Flink

Thanks Dylan, Totally understandable. I already have the appropriate exporters / monitors in place for scraping metrics from Flink, including custom ones, into Prometheus. The labeling challenge is really the big one as while I see lots of labels for the metrics being exported (e.g. job id, wor

Re: Setting max parallelism via properties

Thanks a lot Kezhu, this fits the bill perfectly. Thanks, Padarn On Sun, Feb 28, 2021 at 7:00 PM Kezhu Wang wrote: > Hi Padarn, > > There is a configuration option “pipeline.max-parallelism”. > > It is not a cluster wide configuration but client/job/pipeline side > configuration which means y

Re: Using Prometheus Client Metrics in Flink

Hi Rion, Regarding the question about adding Prometheus labels out of the box. This is common ask of all exporters, but Prometheus philosophy sees this as an "anti-pattern" as the metrics source can often be ambivalent about context. See [0] for example of such a discussion. Instead, we can es

Flink Metrics

Hi flinksters, Scenario: We have cdc messages from our rdbms(various tables) flowing to Kafka. Our flink job reads the CDC messages and creates events based on certain rules. I am using Prometheus and grafana. Following are there metrics that i need to calculate A) Number of CDC messages wrt

Re: Stateful functions 2.2 and stop with savepoint

Thank you for opening the bug and including the extra context. I'll track the progress and, in the meantime, I will work around by taking two separate actions when stopping job: take-savepoints, then cancel. From: Kezhu Wang Sent: Sunday, February 28, 2021 12:31

Independence of task parallelism

Hello, We are working on a project where we want to gather information about the job performance across different task level parallelism settings. Essentially, we want to see how the throughput of a single task varies across different parallelism settings, e.g. for a job of 5 tasks: 1-1-1-1-1 v

Re: Setting max parallelism via properties

Hi Padarn, There is a configuration option “pipeline.max-parallelism”. It is not a cluster wide configuration but client/job/pipeline side configuration which means you should bring this configuration from flink conf file to pipeline generation stage. If I understand correctly, `flink-on-k8s-op

Re: Window Process function is not getting trigger

Hi, Glad to hear. Normally, you would not encounter this if there are massive data. `WatermarkStrategy.withIdleness` could be more appropriate in production. Best, Kezhu Wang On February 24, 2021 at 22:35:11, sagar (sagarban...@gmail.com) wrote: Thanks Kezhu, It worked!!! On Wed, Feb 24, 20

How to emit after a merge?

I define a `Transaction` class: ```scala case class Transaction(accountId: Long, amount: Long, timestamp: Long) ``` The `TransactionSource` simply emits `Transaction` with some time interval. Now I want to compute the last 2 transaction timestamp of each account id, see code below: ```scala impo

Re: Using Prometheus Client Metrics in Flink

Hi Prassana, Thanks for that. It’s what I was doing previously as a workaround however I was just curious if there was any Flink-specific functionality to handle this prior to Prometheus. Additionally from the docs on metrics [0], it seems that there’s a pattern in place to use supported third

Setting max parallelism via properties

Hi all, Sorry for the basic question, but is it possible to set max parallelism using the flink conf file, rather than explicitly in code: https://ci.apache.org/projects/flink/flink-docs-stable/dev/parallel.html#setting-the-maximum-parallelism Need this for a PR I am working on for the flink ope

Re: Stateful functions 2.2 and stop with savepoint

Hi, Thanks for reporting. I think it is a Flink bug and have created FLINK-21522 for it. You could track progress there. FLINK-21522: https://issues.apache.org/jira/browse/FLINK-21522 Best, Kezhu Wang On February 28, 2021 at 00:59:04, Meissner, Dylan ( dylan.t.meiss...@nordstrom.com) wrote: