Re: Unable to recover from checkpoint

Hi Sivaprasanna For RocksDBStateBackend incremental checkpoint, the latest checkpoint may contain the files of the previous checkpoint(the files in the shared directory), so delete the files belong to the previous checkpoint may lead to FileNotFoundException. Currently, we can only parse the

Re: flink1.10.1/1.11.1 使用sql 进行group 和 时间窗口 操作后 状态越来越大

Hi 鱼子酱 能否把在使用增量 checkpoint 的模式下，截图看一下 checkpoint size 的走势呢？另外可以的话，也麻烦你在每次 checkpoint 做完之后，到 hdfs 上 ls 一下 checkpoint 目录的大小。 另外有一个问题还需要回答一下，你的处理速度大概是多少，state 的更新频率能否评估一下呢？ Best, Congxian 鱼子酱 <384939...@qq.com> 于2020年7月30日周四 上午10:43写道： > 感谢！ > >

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

Hi Jincheng, Thanks a lot for bringing up this discussion and the proposal. Big +1 for improving the structure of PyFlink doc. It will be very friendly to give PyFlink users a unified entrance to learn PyFlink documents. Best, Xingbo Dian Fu 于2020年7月31日周五 上午11:00写道： > Hi Jincheng, > >

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

Hi Jincheng, Thanks a lot for bringing up this discussion and the proposal. Big +1 for improving the structure of PyFlink doc. It will be very friendly to give PyFlink users a unified entrance to learn PyFlink documents. Best, Xingbo Dian Fu 于2020年7月31日周五 上午11:00写道： > Hi Jincheng, > >

Re: flink1.9.1 在WebUI中查看committedOffsets指标为负值

如JasonLee所说，你可以在FlinkKafkaConsumerBase的notifyCheckpointComplete方法中看到提交offset的逻辑。 值得注意的是，此处的chk完成指的是整个链路上的chk完成，而不是kafka source的chk完成。 JasonLee <17610775...@163.com> 于2020年7月30日周四 下午9:59写道： > hi > 提交offset到Kafka是在ck成功之后 如果没有开启ck的话 需要设置自动提交提交offset > > > | | > JasonLee > | > | >

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

Hi Jincheng, Thanks a lot for bringing up this discussion and the proposal. +1 to improve the Python API doc. I have received many feedbacks from PyFlink beginners about the PyFlink doc, e.g. the materials are too few, the Python doc is mixed with the Java doc and it's not easy to find the

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

Hi Jincheng, Thanks a lot for bringing up this discussion and the proposal. +1 to improve the Python API doc. I have received many feedbacks from PyFlink beginners about the PyFlink doc, e.g. the materials are too few, the Python doc is mixed with the Java doc and it's not easy to find the

is it possible one task manager stuck and still fetching data from Kinesis?

We are running Flink 1.10 about 900+ task managers with kinesis as an input stream. The problem we are having now is that only Max Age of kinesis shard is growing and the average age of that kinesis is very low meaning most of shards having very low age. We already checked the data skew issue but

Re: Does Flink automatically apply any backpressure ?

Hi Suraj Puvvada Yes, Flink back pressure depend on the Flink task buffer。process task will sends buffer remaining size to source, source will slow down. https://www.ververica.com/blog/how-flink-handles-backpressure Jake > On

[DISCUSS] FLIP-133: Rework PyFlink Documentation

Hi folks, Since the release of Flink 1.11, users of PyFlink have continued to grow. As far as I know there are many companies have used PyFlink for data analysis, operation and maintenance monitoring business has been put into production(Such as 聚美优品[1](Jumei), 浙江墨芷[2] (Mozhi) etc.). According

[DISCUSS] FLIP-133: Rework PyFlink Documentation

Hi folks, Since the release of Flink 1.11, users of PyFlink have continued to grow. As far as I know there are many companies have used PyFlink for data analysis, operation and maintenance monitoring business has been put into production(Such as 聚美优品[1](Jumei), 浙江墨芷[2] (Mozhi) etc.). According

回复: Re: Flink实现Kafka到Mysql的 End-To-End Exactly-Once中遇到的问题

我参考你的代码，也遇到了同样的问题，有什么好的方法吗？ hdxg1101300...@163.com 发件人： 卢伟楠 发送时间： 2020-01-03 15:18 收件人： user-zh 主题： Re: Flink实现Kafka到Mysql的 End-To-End Exactly-Once中遇到的问题 我发这个用于测试的代码，里面的mysql-connector-java已经是最新的了，由于使用mysql-connector-java老版本趟过的坑已经处理过一遍了 > 在 2020年1月3日，下午3:13，残翅2008 <770968...@qq.com> 写道： >

Re: Colocating Compute

Hi Dawid, I am currently on Flink v1.10. Do streaming pipelines support unbounded InputFormat in v1.10? My current setup uses SourceFunction for streaming pipeline and InputFormat for batch queries. I see the documentation for Flink v1.11 describe concepts for Split, SourceReader, and

Re: [Compression] Flink DataStream[GenericRecord] Avrowriters

Hi Ravi, Perfect, This is looking good. Thanks for your help. Regards, Vijay On Thu, Jul 30, 2020 at 5:39 AM Ravi Bhushan Ratnakar < ravibhushanratna...@gmail.com> wrote: > Hi Vijayendra, > > There is an issue with the CustomeAvroWriters.java which i shared with you > earlier, i am sending you

StreamingFileSink: any risk parallelizing active buckets checkpointing?

Hi experts, I am trying to use S3 StreamingFileSink with a high number of active buckets (>1000). I found that checkpointing duration will grow linearly with the number of active buckets, which makes achieving high number of active buckets difficult. One reason for that is the each active

Re: Count of records coming into the Flink App from KDS and at each step through Execution Graph

If you do the aggregation in Prometheus I would think that you do not need to reset the counter; but it's been a while since I've used it. Flink will not automatically reset counters. If this is necessary then you will have to manually reset the counter every 5 seconds. The name under which

Re: Count of records coming into the Flink App from KDS and at each step through Execution Graph

Hi David, Thx for your reply. To summarize: Use a Counter: counter = getRuntimeContext() .getMetricGroup() .addGroup("MyMetricsKey", "MyMetricsValue") //specify my value for each custom event_name here- I might not know all custom event_names in advance .counter("myCounter"); This

Does Flink automatically apply the rebalance operator ?

Hello We are testing a simple use case where we read from kafka -> process and write to kafka. I have set parallelism of the job to 3 and parallelism for the process function to 6. When I looked at the job graph in the Flink UI noticed that between the source and process function a rebalance

Does Flink automatically apply any backpressure ?

Hello I am trying to understand if Flink has a mechanism to automatically apply any backpressure by throttling any operators ? For example if I have a Process function that reads from a Kafkaa source and writes to a Kafka sink. If the process function is slow will the kafka source be

Count of records in the Stream for a time window of 5s

Hi, Trying to get a count of records in the Stream for a time window of 5s. Always getting a count of 1 ?? Sent in 10 records.Expect the count to be 10 at the end. Tried to follow the advise here from Fabian Hueske-

BucketingSink & StreamingFileSink

Hi Flink Team, I'm Mariano & I'm working with Apache Flink to process data and sink from Kafka to Azure Datalake (ADLS Gen1). We are having problems with the sink in parquet format in the ADLS Gen1, also don't work with the gen2. We try to do the StreamingFileSink to store in parquet but we

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

We use runCustomOperation to group a set of operators and into a single functional unit, just to make the code more modular.. It's very comfortable indeed. On Thu, Jul 30, 2020 at 5:20 PM Aljoscha Krettek wrote: > That is good input! I was not aware that anyone was actually using >

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

That is good input! I was not aware that anyone was actually using `runCustomOperation()`. Out of curiosity, what are you using that for? We have definitely thought about the first two points you mentioned, though. Especially processing-time will make it tricky to define unified execution

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

I just wanted to be propositive about missing api.. :D On Thu, Jul 30, 2020 at 4:29 PM Seth Wiesman wrote: > +1 Its time to drop DataSet > > Flavio, those issues are expected. This FLIP isn't just to drop DataSet > but to also add the necessary enhancements to DataStream such that it works >

Options for limiting state size in TableAPI

Hi, I'm trying to investigate a SQL job using a time-windowed join that is exhibiting a large, growing state. The join syntax is most similar to the interval join ( https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/joins.html ). A few questions: 1. Am I correct in

Re: Flink state reconciliation

That is doable via the state processor API, though Arvid's idea does sound simpler :) You could read the operator with the rules, change the data as necessary and then rewrite it out as a new savepoint to start the job. On Thu, Jul 30, 2020 at 5:24 AM Arvid Heise wrote: > Another idea: since

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

+1 Its time to drop DataSet Flavio, those issues are expected. This FLIP isn't just to drop DataSet but to also add the necessary enhancements to DataStream such that it works well on bounded input. On Thu, Jul 30, 2020 at 8:49 AM Flavio Pompermaier wrote: > Just to contribute to the

Flink streaming job logging reserves space

Hi everyone, I have a strange issue with flink logging. I use pretty much standard log4 config, which is writing to standard output in order to see it in Flink GUI. Deployment is on YARN with job mode. I can see logs in UI, no problem. On the servers, where Flink YARN containers are running,

回复：flink1.9.1 在WebUI中查看committedOffsets指标为负值

hi 提交offset到Kafka是在ck成功之后 如果没有开启ck的话 需要设置自动提交提交offset | | JasonLee | | 邮箱：17610775...@163.com | Signature is customized by Netease Mail Master 在2020年07月30日 19:37，bradyMk 写道： 谢谢解答~ 这个确实是个不变的值，应该是没有成功提交；而且我发现了，只要是没有设置ck的任务，该指标都会显示这个值，如果设置了ck，就会正常；但是我不懂为什么会这样，请问您知道详细的原因么？ -- Sent from:

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

Just to contribute to the discussion, when we tried to do the migration we faced some problems that could make migration quite difficult. 1 - It's difficult to test because of https://issues.apache.org/jira/browse/FLINK-18647 2 - missing mapPartition 3 - missing DataSet

Re: Customization of execution environment

That's fine and it's basically what I do as well..I was arguing that it's bad (IMHO) that you could access the config from the BatchTableEnvironment (via bte.getConfig().getConfiguration()). You legitimately think that you are customizing the env but that's illusory. You should not be able to set

Re: Flink Kafka EXACTLY_ONCE causing KafkaException ByteArraySerializer is not an instance of Serializer

Has anyone had success with using exactly_once in a kafka producer in flink? As of right now I don't think the code shown in the docs (https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-producer) actually works. -- Sent from:

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

+1 for this effort. Great to see that we are making progress towards our goal of a truly unified batch and stream processing engine. Cheers, Till On Thu, Jul 30, 2020 at 2:28 PM Kurt Young wrote: > +1, looking forward to the follow up FLIPs. > > Best, > Kurt > > > On Thu, Jul 30, 2020 at 6:40

Re: flink 1.11 rest api saveppoint接口 异常

Hi taowang 你 Jira ID 是啥，我好像找不到你 Best, Congxian taowang 于2020年7月30日周四 下午8:58写道： > 好呀好呀~ > > > 原始邮件 > 发件人: Congxian Qiu > 收件人: user-zh; taowang > 发送时间: 2020年7月30日(周四) 20:53 > 主题: Re: flink 1.11 rest api saveppoint接口 异常 > > > Hi taowang FLINK-18748 在 Jiar

Re: Colocating Compute

Hi Satyam, I think you can use the InputSplitAssigner also for streaming pipelines through an InputFormat. You can use StreamExecutionEnvironment#createInput or for SQL you can write your source according to the documentation here:

Re: flink 1.11 rest api saveppoint接口 异常

好呀好呀~ 原始邮件 发件人: Congxian Qiu 收件人: user-zh; taowang 发送时间: 2020年7月30日(周四) 20:53 主题: Re: flink 1.11 rest api saveppoint接口 异常 Hi taowang FLINK-18748 在 Jiar 侧有一些讨论，这个问题修复起来应该会比较简单，你是否有意愿修复这个问题呢？ Best, Congxian taowang 于2020年7月29日周三 下午3:23写道： > 好哒，我的自动更新逻辑依赖了这个 api，不过现在我用其他方式先处理了。 > 感谢相应，祝好~ >

Re: flink 1.11 rest api saveppoint接口 异常

Hi taowang FLINK-18748 在 Jiar 侧有一些讨论，这个问题修复起来应该会比较简单，你是否有意愿修复这个问题呢？ Best, Congxian taowang 于2020年7月29日周三 下午3:23写道： > 好哒，我的自动更新逻辑依赖了这个 api，不过现在我用其他方式先处理了。 > 感谢相应，祝好~ > > > 原始邮件 > 发件人: Congxian Qiu > 收件人: user-zh > 发送时间: 2020年7月29日(周三) 13:34 > 主题: Re: flink 1.11 rest api saveppoint接口 异常 > >

Unable to recover from checkpoint

Hello, We recently ran into an unexpected scenario. Our stateful streaming pipeline uses RocksDB as the backend and has incremental checkpointing enabled. We have RETAIN_ON_CANCELATION enabled so some of the previous cancellation and restarts had left a lot of unattended checkpoint directories

Re: [Compression] Flink DataStream[GenericRecord] Avrowriters

Hi Vijayendra, There is an issue with the CustomeAvroWriters.java which i shared with you earlier, i am sending you the fixed version, hope this will resolve the issue of reading it from the avro tool. Please use below supported possible string value for codecName null - for nullCodec deflate -

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

+1, looking forward to the follow up FLIPs. Best, Kurt On Thu, Jul 30, 2020 at 6:40 PM Arvid Heise wrote: > +1 of getting rid of the DataSet API. Is DataStream#iterate already > superseding DataSet iterations or would that also need to be accounted for? > > In general, all surviving APIs

Re: flink1.9.1 在WebUI中查看committedOffsets指标为负值

谢谢解答~ 这个确实是个不变的值，应该是没有成功提交；而且我发现了，只要是没有设置ck的任务，该指标都会显示这个值，如果设置了ck，就会正常；但是我不懂为什么会这样，请问您知道详细的原因么？ -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

+1 of getting rid of the DataSet API. Is DataStream#iterate already superseding DataSet iterations or would that also need to be accounted for? In general, all surviving APIs should also offer a smooth experience for switching back and forth. On Thu, Jul 30, 2020 at 9:39 AM Márton Balassi

Re: Flink state reconciliation

Another idea: since your handling on Flink is idempotent, would it make sense to also periodically send the whole rule set anew? Going further, depending on the number of rules, their size, and the update frequency. Would it be possible to always transfer the complete rule set and just discard

Re: Customization of execution environment

I'm not entirely sure, if I completely understand the interaction of BTE and ExecEnv, but I'd create it this way Configuration conf = new Configuration(); conf.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, PARALLELISM); ExecutionEnvironment env =

Re: flink kafka SQL Connectors 传递kerberos 参数

不知道你的问题是能否通过这个解决 我看了下目前文档里缺少了传递kafka properties 的部分，我建了个issue[1]把文档补齐 Best Leonard [1] https://issues.apache.org/jira/browse/FLINK-18768 > 在 2020年7月30日，17:52，lydata 写道： > > > > > > > > 谢谢 ，我试试 > > > > > > > > > > > > > >

Re: flink1.9.1 在WebUI中查看committedOffsets指标为负值

可以检查下在patition3上有没有成功提交过offsets。负值可能是没有提交过情况下的默认值（我猜这是个不变值）。 bradyMk 于2020年7月29日周三 下午6:36写道： > flink1.9.1 > > 在WebUI中查看Source__Custom_Source.KafkaConsumer.topic.geek-event-target.partition.3.committedOffsets指标为负值，查看官网释义：对于每个分区，最后一次成功提交到Kafka的偏移量。 > 但我这里为什么是负值呢？ > < >

Re: Flink使用Kafka作为source时checkpoint成功提交offset的机制

似乎楼主一开始说的checkpoint成功是指source 算子的checkpoint成功？但notifyCheckpointComplete函数要求的是整个链路的chk成功。 这个时候offset为100的消息必然已经被sink算子处理完成了，因为触发chk的屏障消息必然在offset100的消息之后到达sink算子。 hk__lrzy 于2020年7月29日周三 下午5:53写道： > 你是说emit之后的offset commit么？可以看下 > `Kafka09Fetcher`的runFetchLoop方法 > > > 在2020年07月29日 15:09，shuwen

Re: State Restoration issue with flink 1.10.1

Hi I compared the implementation of CepOperator between Flink-1.10.1 and Flink-1.8.2, however they should behave the same as code for map state does not change much. The error you meet might be caused by the change of inputSerializer [1], could you check whether you have introduced any

Re:Re: flink kafka SQL Connectors 传递kerberos 参数

谢谢 ，我试试 在 2020-07-30 17:34:41，"Leonard Xu" 写道： >Hi, >kafka properties 的参数是可以透传的，你试试下面： > >‘properties.security.protocol'='SASL_PLAINTEXT', >‘properties.sasl.mechanism'='GSSAPI’, >‘properties.sasl.kerberos.service.name'='kafka', > >祝好 >Leonard > > >> 在 2020年7月30日，17:00，lydata

Re: Unable to submit high parallelism job in cluster

Hi! The problem was indeed a exponentially slow subtask that related to the parallelism, all working now, thanks! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Unable to submit high parallelism job in cluster

Hi Annemarie, could you please share your topology? If you have a shuffle, your job needs 2 slots per parallelism. So you'd only be able to scale up to 384/2. On Tue, Jul 28, 2020 at 6:32 PM Robert Metzger wrote: > Ah, the good old cloud-11 cluster at DIMA. I used that one as well in 2014 > to

Re: How to stream CSV from S3?

Hi John, I found an example on SO [1] in Scala. [1] https://stackoverflow.com/a/52093079/10299342 On Tue, Jul 28, 2020 at 4:29 PM John Smith wrote: > Hi, is there an example on how RowCsvInputFormat is initialized? > > On Tue, 28 Jul 2020 at 04:00, Jingsong Li wrote: > >> - `env.readCsvFile`

Re: flink kafka SQL Connectors 传递kerberos 参数

Hi, kafka properties 的参数是可以透传的，你试试下面： ‘properties.security.protocol'='SASL_PLAINTEXT', ‘properties.sasl.mechanism'='GSSAPI’, ‘properties.sasl.kerberos.service.name'='kafka', 祝好 Leonard > 在 2020年7月30日，17:00，lydata 写道： > > > > 是否需要这3个参数，或者下面参数是否支持？ > > > > >

Re: Recommended pattern for implementing a DLQ with Flink+Kafka

Hi Tom, using side outputs is actually the established Flink pattern in that regard. The advantage of side output is that you do not depend on the DLQ concept of the source system, which is incredibly useful if you read from multiple systems. Most commonly, the side-output is then outputted to

Re:flink????kafka??????????

complate

Re:flink????kafka??????????

checikpoint state complate ?? ?? 2020-07-30 10:56:52??"??" <201782...@qq.com> ?? >flink1.11kafkagroup >offset??kafka tooloffset??flink

Re: Handle idle kafka source in Flink 1.9

Hi Hemant, sorry for the late reply. You can just create your own watermark assigner and either copy the assigner from Flink 1.11 or take the one that we use in our trainings [1]. [1]

Re:flink kafka SQL Connectors 传递kerberos 参数

是否需要这3个参数，或者下面参数是否支持？ 'security.protocol'='SASL_PLAINTEXT', 'sasl.mechanism'='GSSAPI', 'sasl.kerberos.service.name'='kafka', 在 2020-07-30 16:38:11，"lydata" 写道： > flink v1.11.1 kafka使用了kerberos >下面DDL 是支持 kerberos 参数 > > >CREATETABLEkafkaTable( >... >)WITH('connector'='kafka',

Re: How to retain the column'name when convert a Table to DataStream

Hi, I am afraid you are facing an issue that was not checked for/was not considered. I think your use case is absolutely valid and should be supported. The problem you are facing as far as I can tell from an initial investigation is that the top-level projection/rename is not being applied.

[flink??????DataSet ??orc????????]

hi all ??flink??DataSet ??orc??

flink kafka SQL Connectors 传递kerberos 参数

flink v1.11.1 kafka使用了kerberos 下面DDL 是支持 kerberos 参数 CREATETABLEkafkaTable( ... )WITH('connector'='kafka', 'topic'='user_behavior', 'properties.bootstrap.servers'='localhost:9092', 'properties.group.id'='testGroup', 'security.protocol'='SASL_PLAINTEXT', 'sasl.mechanism'='GSSAPI',

?????? StatementSet ??????????insertsql????

statement.execute??streamEnv.execute app?? ---- ??:

Re: StatementSet 里添加多个insertsql执行

StatementSet 中的多个insert会被编译成一个job提交。 你能提供一下对应的代码样例吗？ op <520075...@qq.com> 于2020年7月30日周四 下午3:57写道： > 大家好，我发现StatementSet 里添加2个insertsql执行的时候会启动两个application， > 这两个任务除了sink都是一样的吗？这样是不是会重复计算和浪费资源，而且两边的数据可能不同步， > 有什么办法能解决？ > 谢谢

StatementSet ??????????insertsql????

??StatementSet ??2??insertsqlapplication?? ??sink ??

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

Hi All, Thanks for the write up and starting the discussion. I am in favor of unifying the APIs the way described in the FLIP and deprecating the DataSet API. I am looking forward to the detailed discussion of the changes necessary. Best, Marton On Wed, Jul 29, 2020 at 12:46 PM Aljoscha Krettek

Re:Flink操作 kafka ，hive遇到kerberos过期

遇到kerberos过期问题，应该是你使用的是ticket cache，而不是keytab文件做认证 在 2020-07-28 17:58:51，"lydata" 写道： > 请问下 Flink操作 kafka ，hive遇到kerberos过期 有什么解决方法吗？

RE: flink消费kafka提交偏移量

可以参考下这个： https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/connectors/ kafka.html#kafka-consumers-offset-committing-behaviour-configuration -Original Message- From: user-zh-return-6007-wxchunjhyy=163@flink.apache.org On Behalf Of 小学 生 Sent: 2020年7月30日 10:57 To: user-zh

Re: unsubscribe

Hi Please send an email to user-unsubscr...@flink.apache.org for unsubscribing. you can get more info here[1] [1] https://flink.apache.org/community.html#mailing-lists Best, Congxian Maatary Okouya 于2020年7月30日周四 上午12:09写道： > >