RE: Flink cli Stop command exception

Hi Yang, Thanks for the response. I will collect the jobmanager logs and share. Is stop command applicable only for streaming jobs? As I can see in the documentation its mentioned for streaming jobs only. If so how I can handle the batch jobs. * Cancel a job with a savepoint (deprecated; u

Re: Flink UDF registration from jar at runtime

Hi, Jakub If I understand correctly you want the job, which you submitted, could load some table function which does not in the job jar. I don't think Flink could support this natively.(Maybe other guys know). But I think this requirement is like some code generated. You need to submit the "code"

Re: Application Mode support on VVP v2.3

Hi Narasimha, I think without a major change in vvp it will not be possible to submit multiple jobs within one jar in the foreseeable future. Maybe you can explain more about your use case why it is inconvenient for you to put the jobs in multiple jars? With ververica platform 2.4 we will add supp

Re: Flink UDF registration from jar at runtime

Hi Jakub, As Guowei said the UDF must be present in the user classloader. It must be there when compiling the program and when executing on the cluster. As of now the TableEnvironment uses the Thread context classloader as the "user classloader" when compiling the query. Therefore you can do the t

Re: How to debug a Flink Exception that does not have a stack trace?

It looks like the problem is that there's a problem in reading a null value in the AvroRowDataDeserializationSchema (see below for the snippet of code from Flink 1.11.1). The problem is due to the fact that there's a bad typing of the source so the call to createConverter() within the createNullabl

Accumulators storage path

I am having all the Accumulators store their data on /tmp - as this is the default. when running on docker - this is mapped on my VM’s “/” partition. a lot of accumulatos – cause low disk util => pods are evicted. Is there a way to set the Accumulators persistence to a different path then

Re: Flink 1.9Version State TTL parameter configuration it does not work

Hi,Yun TangThank for you help, checkpoint was failed due to expired， sorrry ，the jstack log was lost， I will restart the job and save the jstack log，The Flame Graph，we are trying to use it. Thank you very much. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.co

Re: Logs of JobExecutionListener

I would vote for ClusterClient not being internal. I use it a lot in my end-to-end tests to e.g. trigger savepoints and shut down the streaming jobs which I think is not possible via ExecutionEnvironments. So in my opinion, having a more powerful ClusterClient adds a lot of powerful features f

Putting record on kinesis (actually kinesalite) results with The security token included in the request is invalid.

Hi , Any help here will be greatly appreciated I am about to throw the towel, very frustrating... I am trying to put record on kinesalite with the following configuration : System.setProperty(com.amazonaws.SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true") System.setProperty(SDK

[ANNOUNCE] Apache Flink 1.12.0 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.12.0, which is the latest major release. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is avai

Re: [ANNOUNCE] Apache Flink 1.12.0 released

Thanks Dian and Robert for the great work as release manager ! And thanks everyone who makes the release possible ! Best, Leonard > 在 2020年12月10日，20:17，Robert Metzger 写道： > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.12.0, which is the latest maj

Re: How to reduce number of metrics pushed to Prometheus Push Gateway

There is no way to reduce the number of metrics. The only thing you can do is exclude specific variables (e.g., task_name), like this: metrics.reporter..scope.variables.exclude:task_name[; On 12/9/2020 7:15 PM, Alexander Filipchik wrote: Hi, Is there a way to reduce cardinality (preaggregate

Re: [ANNOUNCE] Apache Flink 1.12.0 released

Congratulations! Thanks Dian and Robert for the great work! Best, Wei > 在 2020年12月10日，20:26，Leonard Xu 写道： > > > Thanks Dian and Robert for the great work as release manager ! > And thanks everyone who makes the release possible ! > > > Best, > Leonard > >> 在 2020年12月10日，20:17，Robert Metz

Re: [ANNOUNCE] Apache Flink 1.12.0 released

Thanks Dian and Robert for driving this release and thanks everyone who makes this great work possible ! Best Yun Tang From: Wei Zhong Sent: Thursday, December 10, 2020 20:32 To: d...@flink.apache.org Cc: user ; annou...@apache.org Subject: Re: [ANNOUNCE] Apach

Re: Flink jobmanager TLS connectivity to Zookeeper

Hi Azeem, I haven't worked with Flink's SSL support, yet. But have you taken a look at the SSL configuration options listed under [1]? Best, Matthias [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/security/security-ssl.html#complete-list-of-ssl-options On Tue, Dec 8,

Re: Flink 1.12, Kafka Connector, JSON format - "Could not find a suitable table factory"

Hi! Thank you for the reply! I understand that the metadata syntax is only available as of 1.12, but I am indeed trying to use Flink 1.12. Regarding the option for ignoring parse errors: I have already noticed from before that, according to the docs, even in Flink 1.11 (which is the version that

Re: [ANNOUNCE] Apache Flink 1.12.0 released

Hi first good job and tank you i don't find in docker hub the new version 1.12 when it will be there ? nick ‫בתאריך יום ה׳, 10 בדצמ׳ 2020 ב-14:17 מאת ‪Robert Metzger‬‏ <‪ rmetz...@apache.org‬‏>:‬ > The Apache Flink community is very happy to announce the release of Apache > Flink 1.12.0, whic

Re: FlinkKafkaProducer Fails with "Topic not present in metadata"

Hi Joseph, Thanks for the thorough information. Do you happen to have the trace level logging available? If so, do you mind puttng it somewhere so we can take a look? Thanks, Jiangjie (Becket) Qin On Thu, Dec 3, 2020 at 8:55 PM Joseph Lorenzini wrote: > Hi all, > > I have a flink job that use

How do I pass the aggregated results of an SQL TUMBLING Window as one set of data to a Process Function?

I am sorry to task this twice. I reworded my question though, and I never got an answer. I am trying to learn how to use the SQL api, but mix-in the Streaming API where there is too much complex business logic. GIVEN two windows, window X an SQL tumbling processing time window of 15 minutes, a

Re: Accumulators storage path

Hi Hanan, thanks for reaching out to the Flink community. Have you considered changing io.tmp.dirs [1][2]? Best, Matthias [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/config.html#io-tmp-dirs [2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/clus

Re: How to debug a Flink Exception that does not have a stack trace?

Yea, the error makes sense and was an easy fix. Any idea what happened with the hidden stacktrace? The hidden stacktrace made this 100x more difficult. On Thu, Dec 10, 2020 at 12:59 AM Flavio Pompermaier wrote: > It looks like the problem is that there's a problem in reading a null > value in

Re: Putting record on kinesis (actually kinesalite) results with The security token included in the request is invalid.

Hi Avi, thanks for reaching out to the Flink community. I haven't worked with the KinesisConsumer. Unfortenately, I cannot judge whether there's something missing in your setup. But first of all: Could you confirm that the key itself is valid? Did you try to use it in other cases? Best, Matthias

Re: Putting record on kinesis (actually kinesalite) results with The security token included in the request is invalid.

Hi, Thanks for your reply, The problem is actually with the FlinkKinesisProducer and not the consumer ( i did consume from the stream successfully ). the keys are valid On Thu, Dec 10, 2020 at 6:53 PM Matthias Pohl wrote: > Hi Avi, > thanks for reaching out to the Flink community. I haven't work

Re: How to reduce number of metrics pushed to Prometheus Push Gateway

Thank you for replying! Will exclusion produce proper aggregates? If I drop subtask_index, will resulting metric be a sum of all the subtasks, or it will be just data from one that was reported the last? Alex On Thu, Dec 10, 2020 at 4:28 AM Chesnay Schepler wrote: > There is no way to reduce t

Help on the Split Distinct Aggregation from Table API

Hi, I am trying to understand and simulate the "Split Distinct Aggregation" [1] from Table API. I am executing the query: SELECT driverId, COUNT(DISTINCT dayOfTheYear) FROM TaxiRide GROUP BY driverId on the TaxiRide data from Flink exercises. As it is mentioned in the link [1], the optimization

Re: How User-Defined AggregateFunctions handle deletes of all aggregated rows.

Ok, that makes sense. > You just need to correct the acc state to what it expects to be (say re-evaluate the acc without the record that needs retraction) when you received the retraction message. So for example, if i just remove all items from acc.groupIdSet on retraction it will know to clear

Re: How to reduce number of metrics pushed to Prometheus Push Gateway

It will not produce aggregates. But it may reduce the load a bit without affecting correctness; some variables are not necessarily required for preventing metrics from overriding each other; like the job/task name (because the IDs are good enough). On 12/10/2020 6:37 PM, Alexander Filipchik wr

Re: Flink UDF registration from jar at runtime

Hi Dawid, According to your suggestion, given that a I spawn a LocalEnvironment, I tried the following: val root = new File("custom") val classLoader: URLClassLoader = new URLClassLoader(Array[URL](root.toURI.toURL), Thread.currentThread().getContextClassLoader) val cls = classLoader.loadClas

Re: Help on the Split Distinct Aggregation from Table API

I just realized that i have to use the dayOfTheYear on the gropuBy. I will test again. On Thu, 10 Dec 2020, 18:48 Felipe Gutierrez, wrote: > Hi, > > I am trying to understand and simulate the "Split Distinct > Aggregation" [1] from Table API. I am executing the query: > > SELECT driverId, COUNT

Re: FlinkKafkaProducer Fails with "Topic not present in metadata"

So I actually discovered what the problem is after some extensive debugging with confluent.   My Kafka Producer was attempting to send a record to specific topic partition. The topic partition did not exist. So what the error message should have said is something like “Record sent to pa

Re: Flink UDF registration from jar at runtime

Hey, I am not sure exactly what is going wrong in your case, but I put together an example to show you how I would do it: @Test public void testClassloader() throws IOException, ClassNotFoundException { URLClassLoader functionClassloader = ClassLoaderUtils.compileAndLoadJava( folder.new

latency monitoring

Hi, I would like to monitor the pipeline latency measured by timestamp when writing the output to sink - timestamp when ingested from the source. Now I'm able to get the timestamp writing to sink since the sink is implementing a RichSinkFunction and therefore I can report gauge there [1]. But I h

Re: latency monitoring

Hey Fanbin, I’m not sure if TimeCharacteristic.IngestionTime works with Flink SQL, but if you haven’t tried setting the stream time characteristic to ingestion time it’s worth a shot. Otherwise, one possibility that comes to mind is to use a custom TimestampAssigner to set the event time to the

Re: latency monitoring

1) i have to set the time characteristic to event time 2) I use Flink SQL create table DDL: create table a (user_id string, ts timestamp, data string) . so the timestampAssigner is automatically setup using the `ts` field. Still not sure how to get the source ingestion time to metrics system. O

Re: Flink 1.12, Kafka Connector, JSON format - "Could not find a suitable table factory"

One thing needs to note is that the old connectors are still in the release-1.11/release-1.12 jars. So the old option still works but with the old connector codes. You may need to find the root cause why the new options do not work, maybe some stack trace here ? abelm 于2020年12月10日周四 下午10:54写道：

Re: Flink 1.9Version State TTL parameter configuration it does not work

Hi Andrey, Thank for you help , my job is running on Flink 1.9 version，I restart the job without from checkpoint，it has been running for 12 hours so far. There has been no failure to execute checkpoint so far，but the time it takes to execute checkpoint varies greatly. Time change range from 1

Re: Flink 1.9Version State TTL parameter configuration it does not work

Hi Yun Tang， I restart the job ，it has been running for 12 hours so far. There has been no failure to execute checkpoint so far, but some checkpoint execution takes a long time. This is the