Re: Timeout settings for Flink jobs?

Thanks Caizhi, our use case is for testing - before running a production job, we want to “dry run” it for a while and output the result to a temp location where we can verify the output. We are running a streaming job but the use case sounds more like a batch job. On Mon, Oct 11, 2021 at 7:34 PM

??????flink-1.14 ???? kafkasource ????watermark????

Hi , wm > window.end_time ,?? wm , Best JasonLee ??2021??10??12?? 11:26??kcz<573693...@qq.com.INVALID> ??

回复：flinksql有计划支持mysql catalog吗？

旭晨，你好。 目前这个feature已经在工作中。 欢迎 review / 讨论/改进。 https://github.com/apache/flink/pull/16962 祝好。 Roc. 发自 网易邮箱大师 回复的原邮件 | 发件人 | 赵旭晨 | | 日期 | 2021年10月12日 10:17 | | 收件人 | user-zh@flink.apache.org | | 抄送至 | | | 主题 | flinksql有计划支持mysql catalog吗？ |

flink-1.14 ???? kafkasource ????watermark????

times??+20??StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); KafkaSource

Re: Yarn job not exit when flink job exit

Hi, Jake In Flink 1.14, we recommend using "-t yarn-per-job"[1] for starting per-job cluster. Regarding your issue, I could not reproduce it with the Wordcount example. However, I think this is not the right way for Flink's SQL client, which might be the root cause of your issue. Would you like

Inconsistent parallelism in web UI when using reactive mode

Hi, We found that parallelism in web UI are inconsistent when using reactive mode. As in attachment, in overview page, all parallelism values are 1, which is not correct one. When clicking operator for detail information, the parallelism in detail information is the correct one. Is it possible

Re:​异步IO算子无法完成checkpoint

图片上传到附件中了 在 2021-10-12 10:33:12，"李一飞" 写道： 异步IO算子无法完成checkpoint，帮忙看下是什么原因

Re: Checkpoint size increasing even i enable increasemental checkpoint

Hi! Checkpoint sizes are highly related to your job. Incremental checkpointing will help only when the values in the state are converging (for example a distinct count aggregation). If possible, could you provide your user code or explain what jobs are you running? Lei Wang 于2021年10月11日周一

Re: Yarn job not exit when flink job exit

Hi! yarn-cluster is the mode for a yarn session cluster, which means the cluster will remain even after the job is finished. If you want to finish the Flink job as well as the yarn job, use yarn-per-job mode instead. Jake 于2021年10月9日周六 下午5:53写道： > Hi > > When submit job in yarn-cluster model,

Re: ​异步IO算子无法完成checkpoint

Hi！ 图片无法在邮件中显示，请检查。 李一飞 于2021年10月12日周二 上午10:33写道： > 异步IO算子无法完成checkpoint，帮忙看下是什么原因 > > > >

Re: Timeout settings for Flink jobs?

Hi! There is currently no such setting. You need to rely on an external system to read the execution time (from Flink's job information, see [1]) and cancel the job once it exceeds the time limit. Could you elaborate more on your use case? Are you running a streaming job or a batch job? For

​异步IO算子无法完成checkpoint

异步IO算子无法完成checkpoint，帮忙看下是什么原因

flinksql有计划支持mysql catalog吗？

目前flink的jdbccatalog只支持PG，有计划支持mysql的吗？由于公司的元数据存储统一用mysql，不太可能再引进PG，或者反过来问，flink社区到目前为止不支持mysqlcatalog的原因是什么？有什么顾虑么？

Re: Does the flink sql support checkpoints

(Forwarding this to the user mailing list as this mail is written in English) Hi! I think problem 1 is the expected behavior. Is this behavior inconvenient for you? If yes why it is the case? For problem 2, could you explain in detail how do you run the word count program and where do you store

Re: Does the flink sql support checkpoints

(Forwarding this to the user mailing list as this mail is written in English) Hi! I think problem 1 is the expected behavior. Is this behavior inconvenient for you? If yes why it is the case? For problem 2, could you explain in detail how do you run the word count program and where do you store

Re: PyFlink JDBC SQL Connector for SQL Server

Hi, Currently it only supports derby, mysql, postgresql dialect. The dialect 'sqlserver' is still not supported. There is a ticket https://issues.apache.org/jira/browse/FLINK-14101 for this. Regards, Dian On Mon, Oct 11, 2021 at 9:43 PM Schmid Christian wrote: > Hi all > > > > According to

Re: Does the flink sql support checkpoints

The flink version is v1.13.2 王小宅的蜗居生活 于2021年10月11日周一 下午9:01写道： > Use flink sql for real-time calculation (deployment mode: on yarn). To use > the checkpoint, you need to configure the following in the flink-conf.yaml: > > > state.backend: filesystem > > state.checkpoints.dir:

Does the flink sql support checkpoints

Use flink sql for real-time calculation (deployment mode: on yarn). To use the checkpoint, you need to configure the following in the flink-conf.yaml: state.backend: filesystem state.checkpoints.dir: hdfs:///flink/flink-checkpoints state.savepoints.dir: hdfs:///flink/flink-savepoints

Flink 1.11 loses track of event timestamps and watermarks after process function

Flink 1.11 I have a simple Flink application that reads from Kafka, uses event timestamps, assigns timestamps and watermarks and then key's by a field and uses a KeyedProcessFunciton. The keyed process function outputs events from with the `processElement` method using `out.collect`. No timers

Timeout settings for Flink jobs?

Hi there, We have a use case where we want to terminate a job when a time limit is reached. Is there a Flink setting that we can use for this use case? Thanks, Sharon

Re: Empty Kafka topics and watermarks

Great, thanks! pon., 11 paź 2021 o 17:24 James Sandys-Lumsdaine napisał(a): > Ah thanks for the feedback. I can work around for now but will upgrade as > soon as I can to the latest version. > > Thanks very much, > > James. > -- > *From:* Piotr Nowojski > *Sent:* 08

Re: Empty Kafka topics and watermarks

Ah thanks for the feedback. I can work around for now but will upgrade as soon as I can to the latest version. Thanks very much, James. From: Piotr Nowojski Sent: 08 October 2021 13:17 To: James Sandys-Lumsdaine Cc: user@flink.apache.org Subject: Re: Empty

Re: Reset of transient variables in state to default values.

It would be difficult to provide even a semblance of the complete product , however I could try to provide enough details to reproduce the problem. Standard source would do: DataStream stream = env.addSource( new FlinkKafkaConsumer<>(topic, new AbstractDeserializationSchema() {

Re: Reset of transient variables in state to default values.

Hi Alex, It is a little weird. Would you please provide the program which could reproduce the problem, including DataStream job code and related classes code. I need some debug to find out the reason. Best, JING ZHANG Alex Drobinsky 于2021年10月11日周一 下午5:50写道： > Hi Jing Zhang, > > I'm using the

Re: Unsubscribe

Hi, To unsubscribe emails from Flink user mail list, send an email to user-unsubscr...@flink.apache.org To unsubscribe emails from Flink dev mail list, send an email to dev-unsubscr...@flink.apache.org To unsubscribe emails from Flink user -zh mail list, send an email to

PyFlink JDBC SQL Connector for SQL Server

Hi all According to the official documentation (Table API / JDBC SQL Connector v.1.14.0) "the JDBC connector allows reading data from and writing data into any relational databases with a JDBC driver". At the moment we are using SQL Server in conjunction with Flink and Java, which works

Unsubscribe

Hello i want to unsubscribe

offset of TumblingEventTimeWindows

As in China (UTC+08:00)，we should use Time.hours(-8) as offset when state day's data, // daily tumbling event-time windows offset by -8 hours. input.keyBy().window(TumblingEventTimeWindows.of(Time.days(1),Time.hours(-8))).() shall we also set Time.hours(-8) as offset for minute's state ?

offset of TumblingEventTimeWindows

As in China (UTC+08:00)，we should use Time.hours(-8) as offset when state day's data, // daily tumbling event-time windows offset by -8 hours. input.keyBy().window(TumblingEventTimeWindows.of(Time.days(1),Time.hours(-8))).() shall we also set Time.hours(-8) as offset for minute's state ?

退订

退订

Re: 回复：Flink的api停止方式

可以尝试以下两种方法： 1、达到停止条件时，通过一定方式通知外界工具，外界工具来帮忙停止作业。 2、现在RichFunction里可以拿到jobId，但是拿不到applicationId，可以看看能否修改代码获取它，比如通过环境变量。然后再调用restful 接口停止作业。 lei-tian 于2021年10月11日周一 上午9:11写道： > 因为要在代码里面判断是否停止的条件，停止的时候还是要在代码里面停止吧。 > > > > > > > > > > > > > > > > > > 在 2021-10-11 09:06:17，"995626544"

Re: After Receiving Kafka Data ( getting this error) for s3 bucket access

sorry , there was issue with path of s3 bucket, Got this fixed .. Sorry for troubling you guys On Sunday, October 10, 2021, 12:33:16 PM EDT, Dhiru wrote: We have configured s3 bucket s3a://msc-sandbox-test-bucketI am not sure how come some extra characters get added for a bucket?

退订

退订

Re: Reset of transient variables in state to default values.

Hi Jing Zhang, I'm using the FileSystem backend. I also implemented ReadObject function to support proper restart procedure: private void readObject(ObjectInputStream ois) throws ClassNotFoundException, IOException { ois.defaultReadObject(); logger.info("Deserialized

Re: Reset of transient variables in state to default values.

Hi, Alex What state backend do you choose? If you choose MemoryStateBackend or FsStateBackend, `transient` keyword may not have effect because MemoryStateBackend does not serialize state for regular read/write accesses but keeps it as objects on the heap. If you choose RocksDBStateBackend, I

Reset of transient variables in state to default values.

Dear flink community, I have following state class ( irrelevant fields removed ) public class MultiStorePacketState implements Serializable { public transient RandomAccessFile currentFile = null; public long timerValue; public String fileName; public String exportedFileName;

RE: Flink S3 Presto Checkpointing Permission Forbidden

Hi Rommel, Thanks for getting back to me and for your time. I switched to the Hadoop plugin and used the following authentication method that worked: *fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"* Turns out that I was using the wrong credentials

Checkpoint size increasing even i enable increasemental checkpoint

[image: image.png] The checkpointed data size became bigger and bigger and the node cpu is very high when the job is doing checkpointing. But I have enabled incremental checkpointing: env.setStateBackend(new RocksDBStateBackend(checkpointDir, true)); I am using flink-1.11.2 and aliyun oss as

Re: OVER IGNORE NULLS support

Seems like IGNORE NULL is specified in SQL spec 2008 (paragraph 6.10), the opposite is called RESPECT NULLS: ::= RESPECT NULLS | IGNORE NULLS Perhaps this is worth supporting, I've opened an issue for that FLINK-24499 . @Adrian are you