Apache Flink - Error on creating savepoints using REST interface

Hi: I am using Flink 1.6.2 and trying to create a savepoint using the following curl command using the following references (https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/rest_api.html) and  https://ci.apache.org/projects/flink/flink-docs-release-1.6/api/java/org/apache/

Does Flink use EMRFS?

Hi, I'm using Flink StreamingFileSink running in one AWS account (A) to another (B). I'm also leveraging a SecurityConfiguration in the CFN to assume a role in account B so that when I write there the files are owned by account B which then in turn allows account B to delegate to other AWS account

Apache Flink - Question about application restart

Hi Flink Folks: If I have a Flink Application with 10 restarts, if it fails and restarts, then: 1. Does the job have the same id ?2. Does the automatically restarting application, pickup from the last checkpoint ? I am assuming it does but just want to confirm. Also, if it is running on AWS EMR I

Re: Stateful functions Harness

Also there seems to be a problem in this class public final class Modules When I am running Harness example it fails with the error Exception in thread "main" java.lang.IllegalStateException: There are no routers defined. at org.apache.flink.statefun.flink.core.StatefulFunctionsUniverse

Re: kerberos integration with flink

Hi Guo, Even for HDFS I don't really need to set "security.kerberos.login.contexts" . As long as there is the right ticket in the ticket cache before starting the flink cluster it seems to work fine. I think even [4] from your reference seems to do the same thing. I have defined own ticket cache sp

Re: Flink Dashboard UI Tasks hard limit

Hi Xintong, Thx for your reply. Increasing network memory buffers (fraction, min, max) seems to increase tasks slightly. Streaming job Standalone Vijay On Fri, May 22, 2020 at 2:49 AM Xintong Song wrote: > Hi Vijay, > > I don't think your problem is related to number of opening files. The > p

Re: Why Flink Connector JDBC does't support LocalDateTime ?

Hi, forideal Thanks for your report, As Jingsong said, we do not support LocalDateTime in JDBC connector now, I create a JIRA[1] to track this. Best, Leonard Xu [1] https://issues.apache.org/jira/browse/FLINK-17890 > 在 2020年5月22日，22:07，Jing

Query Rest API from IDE during runtime

Hi, I want to query Flink's REST API in my IDE during runtime in order to get the jobID of the job that is currently running. Is there any way to do this? I found the RestClient class, but can't seem to figure out how to exactly make this work. Any help much appreciated. Best, Annemarie -- Se

onTimer method in CoProcessFunction in flink

How can I identify the type of element for which onTime is called in flink? I want to store the objects for which onTimer is called to sideOutputs and then streamout the sideoutput data to kafka topic. I am not understanding how to stream out the sideoutput data like where should I write that pro

Re: Why Flink Connector JDBC does't support LocalDateTime ?

Hi forideal, Just because we don't have time to support it. We just support LocalDateTime in Flink after 1.9. Welcome to contribute. Best, Jingsong Lee On Fri, May 22, 2020 at 2:48 PM forideal wrote: > Hello, my friends > > env: Flink 1.10, Blink Planner > table source > > CREATE TABL

Re: How do I get the IP of the master and slave files programmatically in Flink?

thanks. it worked! I add the following method at the org.apache.flink.streaming.api.operators.StreamingRuntimeContext class: public Environment getTaskEnvironment() { return this.taskEnvironment; } Then I am getting the IP using: ConfigOption restAddressOption = ConfigOptions .key("rest.addr

Re: Stateful functions Harness

Thanks, I will try your suggestions. One more question. Is StatefulFunctionJob the one that is used for Stateful function execution in the docker case? > On May 22, 2020, at 12:03 AM, Tzu-Li (Gordon) Tai wrote: > > Hi, > > Sorry, I need to correct my comment on using the Kafka ingress / egre

java.lang.AbstractMethodError when implementing KafkaSerializationSchema

public class MyKafkaSerializationSchema implements KafkaSerializationSchema> { @Override public ProducerRecord serialize(Tuple2 o, @Nullable Long aLong) { ProducerRecord record = new ProducerRecord<>(o.f0, o.f1.getBytes(StandardCharsets.UTF_8)); return record;

Re: Writing to SQL server

Well got it working. The varchars in the database were set too small. Thanks for your help! Den fre. 22. maj 2020 kl. 13.30 skrev Martin Frank Hansen < m...@berlingskemedia.dk>: > Arh ok thanks, no problem. > > My problem is now that nothing is sent, do I need to format it in another > way? Or

Re: Writing to SQL server

Is the sql-server jdbc jar in the flink dist lib folder? On Fri, May 22, 2020 at 1:30 PM Martin Frank Hansen wrote: > Arh ok thanks, no problem. > > My problem is now that nothing is sent, do I need to format it in another > way? Or did I miss something else? > > I tried to include Class.forName

Re: Writing to SQL server

Arh ok thanks, no problem. My problem is now that nothing is sent, do I need to format it in another way? Or did I miss something else? I tried to include Class.forName( "com.microsoft.sqlserver.jdbc.SQLServerDriver") but that didn't work. Den fre. 22. maj 2020 kl. 11.57 skrev Flavio Pompermaie

Re: Flink Window with multiple trigger condition

I was also thinking to have a processing time window but that will not work for me. I want to start the window when the user "*search*" event arrives. So for each user window will start from the *search* event. The Tumbling window has fixed start end time so that will not be suitable in my case.

Re: Flink 1.8.3 Kubernetes POD OOM

Hi Josson, Do you use state backend? is it RocksDB? Best, Andrey On Fri, May 22, 2020 at 12:58 PM Fabian Hueske wrote: > Hi Josson, > > I don't have much experience setting memory bounds in Kubernetes myself, > but my colleague Andrey (in CC) reworked Flink's memory configuration for > the las

How to control the banlence by using FlinkKafkaConsumer.setStartFromTimestamp(timestamp)

Hello everyone, First，a brief pipeline introduction: env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) consume multi kafka topic -> union them -> assignTimestampsAndWatermarks -> keyby -> window() and so on … It's a very normal way use flink to p

Re: Flink Dashboard UI Tasks hard limit

Hi Vijay, I don't think your problem is related to number of opening files. The parallelism of your job is decided before actually tries to open the files. And if the OS limit for opening files is reached, you should see a job execution failure, instead of a success execution with a lower parallel

Re: Using Queryable State within 1 job + docs suggestion

Hi Gordon, Yes we are well aware of the inconsistencies that can (and will) emerge while using queryable state like this. However, we will treat them manually for ensuring the correctness of our targeting applications. Therefore, could you help with Annemarie's question or are you aware of someon

Re: Flink 1.8.3 Kubernetes POD OOM

Hi Josson, I don't have much experience setting memory bounds in Kubernetes myself, but my colleague Andrey (in CC) reworked Flink's memory configuration for the last release to ease the configuration in container envs. He might be able to help. Best, Fabian Am Do., 21. Mai 2020 um 18:43 Uhr sch

Re: Writing to SQL server

No sorry, you're right. The JDBCOutputFormat should work..I get confused with the Table API On Fri, May 22, 2020 at 11:51 AM Martin Frank Hansen < m...@berlingskemedia.dk> wrote: > Hi again, > > I am a bit confused as to why the generic jdbc connector would not work > with sql-server? > > Can you

Re: Writing to SQL server

Hi again, I am a bit confused as to why the generic jdbc connector would not work with sql-server? Can you explain a bit more? Den fre. 22. maj 2020 kl. 11.33 skrev Martin Frank Hansen < m...@berlingskemedia.dk>: > Hi Flavio, > > Thanks for your reply. I will try another way then. > > > Den fr

Re: Adaptive Watermarks Generator

Hi, The code of the implementation is linked in the paper: https://github.com/DataSystemsGroupUT/Adaptive-Watermarks Since this is a prototype for a research paper, I'm doubtful that the project is maintained. I also didn't find an open-source license attached to the code. Hence adding the project

Re: Writing to SQL server

Hi Flavio, Thank you so much! Thought i had that import but misread it. The code does not give any errors, but no data is written to the sql server. Can you see why that is? Den fre. 22. maj 2020 kl. 09.02 skrev Flavio Pompermaier < pomperma...@okkam.it>: > SQL server should not be supported

Re: Writing to SQL server

Hi Flavio, Thanks for your reply. I will try another way then. Den fre. 22. maj 2020 kl. 11.31 skrev Flavio Pompermaier < pomperma...@okkam.it>: > I expect you to see some exception somewhere, that sql server dialect is > not supported yet. > > On Fri, May 22, 2020 at 11:29 AM Martin Frank Hans

Re: Writing to SQL server

I expect you to see some exception somewhere, that sql server dialect is not supported yet. On Fri, May 22, 2020 at 11:29 AM Martin Frank Hansen < m...@berlingskemedia.dk> wrote: > Hi Flavio, > > Thank you so much! Thought i had that import but misread it. > > The code does not give any errors, b

Re: Writing to SQL server

SQL server should not be supported from what I know..for this I opened a PR[1] that I should rebase. If someone is interested in I could do it [1] https://github.com/apache/flink/pull/12038 (FLINK-14101 ) On Fri, May 22, 2020 at 8:35 AM Timo Walther wrote: > Hi Martin, > > usually, this error o

Re: Adaptive Watermarks Generator

Thanks Robert! Could you point out the repository name ? I didn't find it from flink-packages.org Packages list. Robert Metzger 于2020年5月19日周二 下午5:02写道： > Hi, > I haven't seen this paper before. I'm not aware of any efforts in the > community to integrate this watermark generator. > I'm also no