Re: java.lang.NoSuchMethodError while writing to Kafka from Flink

Please double-check that your distribution and application jar were built against the same Flink version. This looks related to a binary-compatibility issues reporter in FLINK-13586 .

Re: Query Rest API from IDE during runtime

StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(config) does not actually create any resources yet, this only happens when you run a job. Upon execute() the Flink cluster is started, the job is run, and once the job finishes (and execute() returns) the cluster shuts down. So, you can

Re: Query Rest API from IDE during runtime

If you set DeploymentOptions.ATTACHED to false then execute() does not block until the job finishes, and returns a DetachedJobExecutionResult from which you can retrieve the Job ID. If you need to know when the job finishes you will have to continuously query the REST API. This is the only way

Re: History Server Not Showing Any Jobs - File Not Found?

/infra/big-data/hadoop/hdp-2.6.5.0/hadoop/bin::/gns/mw/dbclient/postgres/jdbc/pg-jdbc-9.3.v01/postgresql-9.3-1100-jdbc4.jar/ // You can see we have references to Hadoop mapred/yarn/hdfs libs in there. *// *ah** *From:*Chesnay Schepler *Sent:* Sunday, May 3, 2020 6:00 PM *To:* Hailu, Andreas [E

Re: History Server Not Showing Any Jobs - File Not Found?

-2.6.5.0/hadoop/bin::/gns/mw/dbclient/postgres/jdbc/pg-jdbc-9.3.v01/postgresql-9.3-1100-jdbc4.jar/ // You can see we have references to Hadoop mapred/yarn/hdfs libs in there. *// *ah *From:*Chesnay Schepler <mailto:ches...@apache.org> *Sent:* Sunday, May 3, 20

Re: History Server Not Showing Any Jobs - File Not Found?

lem. *// *ah** *From:*Hailu, Andreas [Engineering] *Sent:* Thursday, May 28, 2020 12:18 PM *To:* 'Chesnay Schepler' ; user@flink.apache.org *Subject:* RE: History Server Not Showing Any Jobs - File Not Found? Okay, I will look further to see if we’re mistakenly using a version that

Re: History Server Not Showing Any Jobs - File Not Found?

only keep the latest X archives in the dir - or is this something we need to manage ourselves? Thanks. *// *ah** *From:*Hailu, Andreas [Engineering] *Sent:* Friday, May 29, 2020 8:46 AM *To:* 'Chesnay Schepler' ; user@flink.apache.org *Subject:* RE: History Server Not Showing Any Jobs -

Re: Window Function use case;

If you input data already contains both the SensorID and FactoryID, why would the following not be sufficient? DataStream sensorEvents = ...; sensorEvents .filter(sensorEvent -> sensorEvent.Status.equals("alerte")) .map(sensorEvent -> sensorEvent.FactoryID) .addSink() If the problem is that

Re: Flink savepoints history

I can't quite find the answer right now, but the Web UI relies entirely on the REST API. So, whenever you see something in the UI, and wonder where that data comes from, open up the developer tools in your browser, go to the network tab, reload the page and see what requests are being made. On

Re: Simple stateful polling source

erally speaking, yes, you'd always want to extend RichParallelSourceFunction for a parallel data source; not out of necessity, but simplicity. On 07/06/2020 17:43, Ken Krugler wrote: Hi Chesnay, On Jun 19, 2019, at 6:05 AM, Chesnay Schepler <mailto:ches...@apache.org>> wrote:

Re: Run command after Batch is finished

This goes in the right direction; have a look at org.apache.flink.api.common.io.FinalizeOnMaster, which an OutputFormat can implement to run something on the Master after all subtasks have been closed. On 08/06/2020 17:25, Andrey Zagrebin wrote: Hi Mark, I do not know how you output the resu

Re: Timer metric in Flink

You cannot add custom metric types, just implementations of the existing ones. Your timer(wrapper) will have to implement Gauge or Histogram. On 10/06/2020 14:17, Vinay Patil wrote: Hi, As timer metric is not provided out of the box, can I create a new MetricGroup by implementing this interfa

Re: Timer metric in Flink

a plan or JIRA ticket to add Timer metric in future release, I think it is good to have Regards, Vinay Patil On Wed, Jun 10, 2020 at 5:55 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: You cannot add custom metric types, just implementations of the existing ones. Y

Re: Request: Documentation for External Communication with Flink Cluster

https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/rest_api.html#api On 15/06/2020 17:47, Morgan Geldenhuys wrote: Hi Community, I am interested in creating an external client for submitting and managing Flink jobs via a HTTP/REST endpoint. Taking a look at the documentat

Re: Unable to run flink job in dataproc cluster with jobmanager provided

Are you by any chance creating a local environment via (Stream)ExecutionEnvironment#createLocalEnvironment? On 17/06/2020 17:05, Sourabh Mehta wrote: Hi Team, I'm  exploring flink for one of my use case, I'm facing some issues while running a flink job in cluster mode. Below are the steps I

Re: Completed Job List in Flink UI

https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/config.html#jobstore-expiration-time On 18/06/2020 19:57, Ivan Yang wrote: Hello, In Flink web UI Overview tab, "Completed Job List” displays recent completed or cancelled job only for short period of time. After a while, they are

Re: Renaming the metrics

There's currently no way to change this. A related enhancement was proposed on FLINK-17495 that would at least allow you to attach a custom label, but the initial implementation wasn't general enough. On 22/06/2020 15:08, Arvid Heise wrote: Hi Ori, I see that the PrometheusPushGatewayReport

Re: Unable to run flink job in dataproc cluster with jobmanager provided

debugging the problem. Which version of Flink are you using? Cheers, Till On Wed, Jun 17, 2020 at 7:39 PM Sourabh Mehta mailto:sourabhmehta2...@gmail.com>> wrote: No, I am not. On Wed, 17 Jun 2020 at 10:48 PM, Chesnay Schepler mailto:che

Re: Faild to load dependency after migration to Flink 1.10

Did you make any modifications to the Flink scripts in order to get log4j2 to work with Flink 1.10? IIRC we had to modify the scripts when we migrated to log4j2 in 1.11; if this is done incorrectly it could break thing. Do all Flink processes have this issue, or only TaskExecutors? Can you pro

Re: Integrating prometheus

What metrics specifically are you interested in? On 03/07/2020 17:22, Robert Metzger wrote: Hi Manish, Currently, Flink's metric system does not support metrics via annotations. You need to go with the documented approach. But of course, you can try to build your own metrics abstraction based

Re: Integrating prometheus

ehow my custom metrics(a simple Counter) is still not shown on the prometheus board, though default metrics I can see. Anything I am missing? On Fri, Jul 3, 2020 at 8:57 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: What metrics specifically are you intere

Re: Logging Flink metrics

Have you looked at the SLF4J reporter? https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter On 06/07/2020 13:49, Manish G wrote: Hi, Is it possible to log Flink metrics in application logs apart from publishing it t

Re: Logging Flink metrics

invoked markEvent() method in the target code. But I don't see any related logs. I am doing this all on my local computer. Anything else I need to do? With regards Manish On Mon, Jul 6, 2020 at 5:24 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Have you looked at the

Re: Logging Flink metrics

is as: metrics.reporter.slf4j.class: org.apache.flink.metrics.slf4j.Slf4jReporter metrics.reporter.slf4j.interval: 30 SECONDS On Mon, Jul 6, 2020 at 7:57 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: How long did the job run for, and what is the configured interval?

Re: Logging Flink metrics

without having to remove the entire configuration. I'd suggest to just remove the option, try again, and report back. On 06/07/2020 16:35, Chesnay Schepler wrote: Please enable debug logging and search for warnings from the metric groups/registry/reporter. If you cannot find anything suspi

Re: Logging Flink metrics

orter metrics.reporter.slf4j.interval: 30 SECONDS // And while I can see custom metrics in Taskmanager logs, but prometheus dashboard logs doesn't show custom metrics. With regards On Mon, Jul 6, 2020 at 8:55 PM Chesnay Schepler <mailto:ches...@apach

Re: Logging Flink metrics

quot;58483036154d7f72ad1bbf10eb86bc2e",host="localhost",job_name="frauddetection",} 0.0 On Mon, Jul 6, 2020 at 9:51 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: You've said elsewhere that you do see some metrics in prometheus, w

Re: Logging Flink metrics

- targets: ['localhost:9250', 'localhost:9251']     metrics_path: /* * * This is the whole configuration I have done based on several tutorials and blogs available online. ** /**/ On Mon, Jul 6, 2020 at 10:20 PM Chesnay Schepler <mailto:ches...@apache.org>> wrot

Re: Logging Flink metrics

y for the JM/TM, which will likely entail manually starting the processes and updating the configuration in-between, e.g.: ./bin/jobmanager.sh start ./bin/taskmanager.sh start On 06/07/2020 19:16, Manish G wrote: Yes. On Mon, Jul 6, 2020 at 10:43 PM Chesnay Schepler <mailto:ches...@apache.org&

Re: SSL for QueryableStateClient

Queryable state does not support SSL. On 06/07/2020 22:42, mail2so...@yahoo.co.in wrote: Hello, I am running flink on Kubernetes, and from outside the Ingress to a proxy on Kubernetes is via SSL 443 PORT only. Can you please provide guidance on how to setup the SSL for /*QueryableStateClien

Re: Decompressing Tar Files for Batch Processing

I would probably go with a separate process. Downloading the file could work with Flink if it is already present in some supported filesystem. Decompressing the file is supported for selected formats (deflate, gzip, bz2, xz), but this seems to be an undocumented feature, so I'm not sure how us

Re: Chaining the creation of a WatermarkStrategy doesn't work?

WatermarkStrategy.forBoundedOutOfOrderness(Duration.of(1, ChronoUnit.MINUTES)) returns a WatermarkStrategy, but the exact type is entirely dependent on the variable declaration (i.e., it is not dependent on any argument). So, when you assign the strategy to a variable then the compiler can in

Re: Limiting metrics logs to custom metric

There's no built-in functionality for this. You could customize the reporter though. On 08/07/2020 17:19, Manish G wrote: Hi, I have added a Meter in my code and pushing it to app logs using slf4j reporter. I observe that apart from my custometrics, lots of other metrics like gauge, histog

Re: Limiting metrics logs to custom metric

same? On Wed, Jul 8, 2020, 9:38 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: There's no built-in functionality for this. You could customize the reporter though. On 08/07/2020 17:19, Manish G wrote: > Hi, > > I have added a Meter in my cod

Re: A query on Flink metrics in kubernetes

From Flink's perspective no metrics are aggregated, nor are metric requests forwarded to some other process. Each TaskExecutor has its own reporter, that each must be scraped to get the full set of metrics. On 09/07/2020 11:39, Manish G wrote: Hi, I have a query regarding prometheus scrapin

Re: Customised RESTful trigger

You can specify arguments to your job via query parameters or a json body (recommended) as documented here . On 10/07/2020 18:48, Jacek Grzebyta wrote: Hello, I am a newbie in the Apache Flin

Re: History Server Not Showing Any Jobs - File Not Found?

e future J *// *ah** *From:*Chesnay Schepler *Sent:* Tuesday, June 2, 2020 3:55 AM *To:* Hailu, Andreas [Engineering] ; user@flink.apache.org *Subject:* Re: History Server Not Showing Any Jobs - File Not Found? 1) It downloads all archives and stores them on disk; the only thing stored in memory

Re: RequiredParameters in Flink 1.11

/** * ... * * @deprecated These classes will be dropped in the next version. Use {@link ParameterTool} or a third-party * command line parsing library instead. */ On 13/07/2020 17:24, Flavio Pompermaier wrote: In Flink 1.11 RequiredParameters and Option have been deprecated. Is there any reco

Re: Missing jars

flink-formats is a pom artifact, meaning that there are no jars for it. You should add a dependency to the specific format(s) you are interested in, like flink-formats-csv. On 14/07/2020 17:41, Daves Open wrote: I added flink-formats to my pom.xml file, but the jar files are not found.  I che

Re: RestartStrategy failure count when losing a Task Manager

1) A restart in one region only increments the count by 1, independent of how many tasks from that region fail at the same time. If tasks from different regions fail at the same time, then the bound is incremented by the number of affected regions. 2) I would consider what failure rate is acce

Re: Communicating with my operators

Using an S3 bucket containing the configuration is the way to go. 1) web sockets, or more generally all approaches where you connect to the source The JobGraph won't help you; it doesn't contain the information on where tasks are deployed to at runtime. It is just an abstract representation

Re: Is there a way to access the number of 'keys' that have been used for partioning?

This information is not readily available; in fact Flink itself doesn't know how many keys there are at any point. You'd have to calculate it yourself. On 15/07/2020 17:11, orionemail wrote: Hi, I need to query the number of keys that a stream has been split by, is there a way to do this? T

Re: Using md5 hash while sinking files to s3

Please try configuring : fs.s3a.etag.checksum.enabled: true On 16/07/2020 03:11, nikita Balakrishnan wrote: Hello team, I’m developing a system where we are trying to sink to an immutable s3 bucket. This bucket has server side encryption set as KMS. The DataStream sink works perfectly fine wh

Re: backup configuration in Flink doc

They should be public yes; I do not know what the "Backup" category is supposed to mean, and I suspect this was a WIP title. On 16/07/2020 18:01, Steven Wu wrote: The configuration page has this "backup" section. Can I assume that they are public interfaces? The name "backup" is a little conf

Re: Using md5 hash while sinking files to s3

to set the md5 hash value manually while sinking? The fs.s3a.etag.checksum.enabled: true will do it for me? And Do I need to specify anywhere that we have to use md5 hashing? On Thu, Jul 16, 2020 at 12:04 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote:

Re: Question on Pattern Matching

Have you read this part of the documentation ? From what I understand, it provides you hooks for processing matched/timed out patterns. On 16/07/2020 20:23, Basanth Gowda wrote

Re: Flink 1.11 throws Unrecognized field "error_code"

Please double-check that the client and server are using the same Flink version. On 17/07/2020 02:42, Lian Jiang wrote: Hi, I am using java 1.8 and Flink 1.11 by following https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/local_installation.html on my MAC Mojave 10.14.6.

Re: REST API randomly returns Not Found for an existing job

How reproducible is this problem / how often does it occur? How is the cluster deployed? Is anything else happening to the cluster around that that time (like a JobMaster failure)? On 24/07/2020 13:28, Tomasz Dudziak wrote: Hi, I have come across an issue related to GET /job/:jobId endpoint

Re: How to start flink standalone session on windows ？

Flink no longer runs natively on Windows; you will have to use some unix-like environment like WSL or cygwin. On 24/07/2020 04:27, wangl...@geekplus.com.cn wrote: There's no  start-cluster.bat and flink.bat in bin directory. So how can i start flink on windowns OS? Thanks, Lei -

Re: problem with build from source flink 1.11

@Timo maven 3.2.5 is the recommended Maven version for building Flink. @Felipe Can you provide us the full stacktrace? This could be a library issue in regards to JDK compatibility. On 27/07/2020 15:23, Timo Walther wrote: Hi Felipe, are you sure that Maven and the TaskManagers are using the

Re: Count of records coming into the Flink App from KDS and at each step through Execution Graph

I'd recommend to do the aggregation over 5 seconds in graphite/prometheus etc., and expose a counter in Flink for each attribute/event_name. User variables are a good choice for encoding the attribute/event_name values. As for your remaining questions: Flink does not support aggregating ope

Re: Making sense of Meter metrics graph on Grafana

Yes; a rate of 1 means that 1 event occurred per second, which in your case means one call to markEvent() per second. Note that the default Meter implementation calculates the rate per second over the last minute (basically, rate(T) = (count(T) - count(T-60)) / 60; so short spikes tend to be f

Re: Count of records coming into the Flink App from KDS and at each step through Execution Graph

bases. relate to the Summary above from me Also, I am assuming that the Counter will get reset after every Window interval of 5 secs or do I need to do counter.dec(counter.getCount()) in the close() method as you showed above. TIA, || On Wed, Jul 29, 2020 at 2:53 AM Chesnay Schepler <m

Re: getting in an infinite loop while creating the dependency-reduced pom

https://issues.apache.org/jira/browse/MSHADE-148 On 03/08/2020 10:16, Dongwon Kim wrote: Hi, I create a new maven project (I'm using Maven 3.6.3) w/ the below command |curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash -s 1.11.1| and add the following dependencies to depend

Re: Any change in behavior related to the "web.upload.dir" behavior between Flink 1.9 and 1.11

From what I can tell we have not changed anything. Are you making any modifications to the image? This exception should only be thrown if there is already a file with the same path, and I don't think Flink would do that. On 03/08/2020 21:43, Avijit Saha wrote: Hello, Has there been any chan

Re: Metrics for number of events in a timeframe

meter * timeframe (in seconds) is the simplest option, although it will not be that accurate due to the flattening of spikes. You'd get the best results by using a time-series database, and calculating the difference between the current count and one 5 minutes ago. An example for Prometheus: h

Re: Metrics for number of events in a timeframe

No, because Flink counters are mapped to Prometheus gauges. On 04/08/2020 15:52, Manish G wrote: That documentation states: |delta| should only be used with gauges. Would that cause an issue as we are using counter. With regards On Tue, Aug 4, 2020 at 7:12 PM Chesnay Schepler <mailto:c

Re: The bytecode of the class does not match the source code

Please make sure you have loaded the correct source jar, and aren't by chance still using the 1.11.0 source jar. On 05/08/2020 09:57, 魏子涵 wrote: Hi, everyone:       I found  the 【org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl】 class in【flink-runtime_2.11-1.11.1.jar】does not match th

Re: The bytecode of the class does not match the source code

ttps://img-blog.csdnimg.cn/20200805180232929.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NTRE5yaG1t,size_16,color_FF,t_70 At 2020-08-05 16:46:11, "Chesnay Schepler" wrote: Please make sure you have loaded the cor

Re: Flink Promethues Metricsreporter Question

The PrometheusReporter acts as a scraping target for a single process. If you already have setup something in the Flink cluster that allows Prometheus/ServiceMonitor to scrape (Flink) metrics, then it shouldn't be necessary. It doesn't coordinate with other services in any way; it just has acces

Re: Metrics for the overall pipeline

You could create an abstract class that extends AbstractRichFunction, and all your remaining functions extend that class and implement the respective (Map/etc.)Function interface. On 06/08/2020 13:20, Manish G wrote: Adding metrics to individual RichMapFunction implementation classes would giv

Re: Hadoop_Compatability

We still offer a flink-shaded-hadoop-2 artifact that you can find on the download page: https://flink.apache.org/downloads.html#additional-components In 1.9 we changed the artifact name. Note that we will not release newer versions of this dependency. As for providing Hadoop class, there is som

Re: Dependency vulnerabilities with Apache Flink 1.10.1 version

log4j - If you don't use a Socket appender, you're good. Otherwise, you can replace the log4j jars in lib/ with a newer version. You could also upgrade to 1.11.1 which uses log4j2. guava - We do not use Guava for serialization AFAIK. We also do not use Java serialization for records. commons

Re: JobManager seems to be leaking temporary jar files

't be successfully deleted? As for FLINK-9844 - I understand that having code like if (userClassLoader instanceof Closeable) { ((Closeable) userClassloader).close() } is too "dirty trick" to be considered? thanks, maciek On 27.01.2021 13:00, Chesnay Schepler wrote: The prob

Re: Timers not firing until stream end

ttps://t.sidekickopen70.com/s2t/c/5/f18dQhb0S7kv8cpgQZVc6VPt59hl3kW7_k2842PjkFxW2R1KhZ7v4vclW2Rxbb82bzNKzf7GYHvr01?te=W3R5hFj4cm2zwW4fQ47l4fGCmnW3Fbt5S3H4THtF3F6jFSWsSg1&si=5987503666495488&pi=adc3545f-9610-4164-fa4a-2bddbd615e33>  +44 7961 125282 See our latest features <https://t.sidekicko

Re: Timers not firing until stream end

line":true,"log":"prevMsg.ts 3000 msg_in.ts 4000 Cancelled previous timer. "} -Pilgrim -- Learn more at https://devicepilot.com <https://devicepilot.com> @devicepilot <https://t.sidekickopen70.com/s2t/c/5/f18dQhb0S7kv8cpgQZVc6VPt59hl3kW7_k2842PjkFxW2R1KhZ7v4vclW2Rxbb82bzNKzf7GYHvr01?te=W3R5hFj4cm2zwW4fQ47l4fGCmnW3Fbt5S3H

Re: Flink 1.11.2 test cases fail with Scala 2.12.12

Scala 2.12.8 broke binary compatibility with 2.12.7 which Flink currently is compiled against. As a result you must either stay at 2.12.7, or recompile Flink yourself against 2.12.12 as shown here

Re: stopping with save points

It would be good if you could take a look at the Job-/TaskManager logs to see whether the operation is making progress or whether an exception has occurred. Does the job stop eventually? It could be that draining the jobs just takes longer than the client timeout allows by default (60 seconds)

Re: Question

Could you expand a bit on what you mean? Are you referring to /savepoints/? On 1/28/2021 3:24 PM, Abu Bakar Siddiqur Rahman Rocky wrote: Hi, Is there any library to use and remember the apache flink snapshot? Thank you -- Regards, Abu Bakar Siddiqur Rahman

Re: Cannot access state from a empty taskmanager - using kubernetes

Yes, it does sound quite a lot like FLINK-10225. I assume it is only happening for some task executors and not all of them? Unfortunately I don't think this issue will be fixed anytime soon. On 1/28/2021 12:59 PM, Daniel Peled wrote: Hi, We have followed the instructions in the following link

Re: question on checkpointing

1) An operator that just blocks for a long time (for example, because it does a synchronous call to some external service) can indeed cause a checkpoint timeout. 2) What kind of effects are you worried about? On 1/28/2021 8:05 PM, Marco Villalobos wrote: Is it possible that checkpointing times

Re: Is Flink able to parse strings into dynamic JSON?

Flink needs to know upfront what kind of types it deals with to setup the serialization stack between operators. As such, generally speaking, you will have to use some generic container for transmitting data (e.g., a String or a Jackson ObjectNode) and either work on them directly or map them

Re: Integration with Apache AirFlow

I'm sorry, but aren't these question better suited for the Airflow mailing lists? On 2/2/2021 12:35 PM, Flavio Pompermaier wrote: Thank you all for the hints. However looking at the REST API[1] of AirFlow 2.0 I can't find how to setup my DAG (if this is the right concept). Do I need to first c

Re: Flink Datadog Timeout

The reported exception looks quite similar to the one in this thread , which was supposedly caused by Datadog rate limits but I don't think this was thoroughly investig

Re: Question

/org/apache/flink/runtime/highavailability/zookeeper On 2/3/2021 3:08 AM, Abu Bakar Siddiqur Rahman Rocky wrote: Hi, Is there any source code for the checkpoints, snapshot and zookeeper mechanism? Thank you On Mon, Feb 1, 2021 at 4:23 AM Chesnay Schepler <mailto:ches...@apache.org>&

Re: Job submission failure via flink cli

Please make sure the client and server version are in sync. On 2/3/2021 4:12 PM, sidhant gupta wrote: I am getting following error while running the below command with the attached conf/flink-conf.yaml: bin/flink run -c firstflinkpackage.someJob ../somejob.jar arg1 arg2 arg3 2021-02-03 15:04

Re: ClassLoader leak when using s3a upload through DataSet.output

FileSystems must not be bundled in the user jar. You must place them in lib/ or plugins/, because by bundling it you break our assumption that they exist for the lifetime of the cluster (which in turn means we don't really have to worry about cleaning up). On 2/10/2021 4:01 PM, Vishal Santosh

Re: clarification on backpressure metrics in Apache Flink Dashboard

Yes. Unless operator 2 is also back-pressured of course, then you should take a look at the sink. On 2/11/2021 4:50 AM, Marco Villalobos wrote: given: [source] -> [operator 1] -> [operator 2] -> [sink]. If within the dashboard, operator 1 shows that it has backpressure, does that mean I need

Re: Tag flink metrics to job name

When you mean "job_name", are you referring to the Prometheus concept of jobs, of the one of Flink? Which of Flink prometheus reporters are you using? On 2/17/2021 7:37 PM, bat man wrote: Hello there, I am using prometheus to push metrics to prometheus and then use grafana for visualization.

Re: Sharding of Operators

When you change the parallelism then keys are re-distributed across operators instances. /However/, this re-distribution is limited to the set /maxParallelism /(set via the ExecutionConfig), which by default is 128 if no operators exceeded the parallelism on the first submission. This *cannot

Re: latency related to the checkpointing mode EXACTLY ONCE

Yes, if you are only reading committed data than it will take least the checkpoint interval for the data to be available to downstream consumers. On 2/18/2021 6:17 PM, Tan, Min wrote: Hi, We use the checkpointing mode EXACTLY ONCE for some of our flink jobs. I wonder how the chec

Re: Best practices around checkpoint intervals and sizes?

A lower checkpoint interval (== more checkpoints / time) will consume more resources and hence can affect the job performance. It ultimately boils down to how much latency you are willing to accept when a failure occurs and data has to be re-processed (more checkpoints => less data). How long

Re: Tag flink metrics to job name

Flink jobname. I’m using the below reporter - || |metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter| Is there any way to tag job names to the task and job manager metrics. Thanks, Hemant On Fri, 19 Feb 2021 at 12:40 AM, Chesnay Schepler <mailto:ches...@apache.org

Re: How do I increase number of db connections of the Flink JDBC Connector?

Every works uses exactly 1 connection, so in order to increase the number of connections you must indeed increase the worker parallelism. On 2/19/2021 6:51 AM, Li Peng wrote: Hey folks, I'm trying to use flink to write high throughput incoming data to a SQL db using the JDBC Connector as desc

Re: How is proctime represented?

Could you check whether this answers your question? https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/timely-stream-processing.html#notions-of-time-event-time-and-processing-time On 2/19/2021 7:29 AM, Rex Fenley wrote: Hello, When using PROCTIME() in CREATE DDL for a source

Re: Tag flink metrics to job name

multiple jobs are running on the same cluster. Also, the issue am trying to resolve is I’m seeing high memory usage for one of the containers I want isolate the issue with one job and then investigate further. Thanks, Hemant On Fri, 19 Feb 2021 at 12:18 PM, Chesnay Schepler <mailto:c

Re: How is proctime represented?

only gets filled in when used by an operator, then again, that's still not exactly what it says so I only feel 70% confident like that's what is happening. On Thu, Feb 18, 2021 at 10:55 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Could you check wheth

Re: Sharding of Operators

How will flink runtime handle such a situation? *From:*Chesnay Schepler *Sent:* Friday, February 19, 2021 12:52 AM *To:* yidan zhao ; Tripathi,Vikash *Cc:* user *Subject:* Re: Sharding of Operators When you change the parallelism then keys are re-distributed across operators instances.

Re: Flink 1.11.2 test cases fail with Scala 2.12.12

Coud you check your dependency tree for the version of scala-library? On 2/24/2021 7:28 AM, soumoks wrote: Thank you for the response but this error continues to happen with Scala 2.12.7. The app itself continues to compile without errors but the test cases fail with the same error. Seems to be

Re: Flink upgrade causes operator to lose state

It is currently not possible to upgrade table API / SQL applications via savepoints. This thread may provide some more insights: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-and-checkpoints-and-savepoints-td40749.html On 3/3/2021 6:53 PM, soumoks wrote: Hi,

Re: Unit Testing State Stores in KeyedProcessFunctions

I do not believe this to be possible. Given that the state will likely in some form affect the behavior of the function (usually in regards to what it outputs), it may be a better idea to test for that. (I suppose you'd want tests like that anyway) On 3/3/2021 8:10 PM, Rion Williams wrote: H

Re: Compile Error

The flink-clients dependency is correct. We will need additional information to debug the Job execution failures, because these can happen due to all kind of reasons. Things like the full stacktrace, or exceptions from the logging output. Additionally, I would recommend to base your project on

Re: Stop vs Cancel with savepoint

Your understanding of cancel vs stop(-with-savepoint) is correct. I agree that we should update the REST API documentation and have a section outlining the problems with cancel-with-savepoint. Would you like to open a ticket yourself? On 3/3/2021 11:16 AM, Thomas Eckestad wrote: Hi! Cancel w

Re: Flink Zookeeper leader change v 1.9.X

1) This could occur due to a number of reasons, like processes crashing, network issues between ZK and Flink, or the JobManager being stuck in some blocking operation for a long time. You will need to take a look at the ZK/Flink logs to narrow things down. 2) For FLINK-14091 the issue was not

Re: 1.12.2 docker image

they should be released in a day or two. On 3/3/2021 11:18 PM, Bohinski, Kevin wrote: Hi, Are there plans to provide a docker image for 1.12.2? Best kevin

Re: Defining GlobalJobParameters in Flink Unit Testing Harnesses

Could you show us how you create test harness? On 3/4/2021 5:13 AM, Rion Williams wrote: Hi all, Early today I had asked a few questions regarding the use of the many testing constructs available within Flink and believe that I have things in a good direction at present. I did run into a spec

Re: Defining GlobalJobParameters in Flink Unit Testing Harnesses

ion.of(String::class.java)     ) } I've also tried a few variants of that with a separate declaration for the function itself, etc. On Thu, Mar 4, 2021 at 6:47 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Could you show us how you create test harness? On 3/4/2021 5

Re: Defining GlobalJobParameters in Flink Unit Testing Harnesses

ngTimeBuffer") } Does anything look out of place here? I haven't gone spelunking into the source code for this yet, but I'm assuming that I'm setting the correct values on the execution configuration. Thanks again, Rion On Thu, Mar 4, 2021 at 7:57 AM Chesnay Schepler &

Re: Python Flink cluster: how to shut-down

run-application creates an application cluster that shuts down once the job is complete. As such, canceling the job in this case is equivalent to shutting down the cluster. AFAIK you also need to specify kubernetes arguments when using the list command. (without any argument it just assumes yo

Re: Compile Error

;count": so different from a streaming job. *Is something I missed due to not understanding those two lines?* * * *Thank you!* On Wed, Mar 3, 2021 at 2:49 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: The flink-clients dependency is correct. We will nee

<    1   2   3   4   5   6   7   8   9   10   >