Re: When to use scan.incremental.snapshot.enabled

y MongoDBSource ? > > I don't see any option in the builder for the same. > > Regards > Sachin > > > > On Mon, Aug 19, 2024 at 8:00 PM Jiabao Sun wrote: > > > Sorry, in my previous reply, I mistakenly wrote Flink 2.0 instead of Flink > > CDC 2.0. > &

Re: When to use scan.incremental.snapshot.enabled

n of flink ? > > Thanks > Sachin > > > On Mon, Aug 19, 2024 at 2:29 PM Jiabao Sun wrote: > > > Hi Sachin, > > > > Incremental snapshot reading is a new feature introduced in Flink 2.0. > > > > It has the following capabilities: > > - Source c

Re: When to use scan.incremental.snapshot.enabled

ction faster. > > Am I understanding that correctly? > > Also I am using Flink 1.8, would it work with this version of flink ? > > Thanks > Sachin > > > On Mon, Aug 19, 2024 at 2:29 PM Jiabao Sun wrote: > > > Hi Sachin, > > > > Incremental snapsh

Re: How to set the number formatter for json convertor for mongo cdc connector

Hi Sachin, It is recommended to use org.bson.Document to convert MongoDB Extended JSON into Java types, and then perform further field mapping. .deserializer(new DebeziumDeserial

Re: When to use scan.incremental.snapshot.enabled

Hi Sachin, Incremental snapshot reading is a new feature introduced in Flink 2.0. It has the following capabilities: - Source can be parallel during snapshot reading to improve snapshot speed - Source can perform checkpoints in the chunk granularity during snapshot reading Limitation: - MongoD

Re: Mongo flink CDC connector not reading from the source

Hi Sachin, The 'collectionList' needs to be filled with fully qualified names. For example, database: test_db collection: test_collection MongoDBSource.builder() .hosts(HOSTNAME) .scheme(SCHEME) .databaseList("test_db") .collectionList("test_db.test_collection") ... Bes

Re: Integrating flink CDC with flink

connector or I can use > flink-connector-mongodb-cdc to process both existing and new data ? > > Thanks > Sachin > > > On Fri, Aug 16, 2024 at 3:46 PM Jiabao Sun wrote: > > > Hi Sachin, > > > > flink-connector-mongodb supports batch reading and writing

Re: Integrating flink CDC with flink

Hi Sachin, flink-connector-mongodb supports batch reading and writing to MongoDB, similar to flink-connector-jdbc, while flink-connector-mongodb-cdc supports streaming MongoDB changes. If you need to stream MongoDB changes, you should use flink-connector-mongodb-cdc. You can refer to the fol

Re: How can I debug Assigned key must not be null error when reading from Mongodb source

Hi Sachin, Could you please check if you have used the keyBy operator and ensure that the keyBy field is not null? Best, Jiabao On 2024/08/05 12:33:27 Sachin Mittal wrote: > So I have an anonymous class implementing MongoDeserializationSchema > > new MongoDeserializationSchema() { > @Overrid

RE: 退订

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Jiabao [1] https://fl

RE: RE: Flink Kafka Sink + Schema Registry + Message Headers

he/flink/connector/kafka/sink/KafkaRecordSerializationSchemaBuilder.java > But this doesn't seem released yet. Can you please point me towards correct > Flink version? > > Also, any help on question 1 regarding Schema Registry? > > Regards, > Kirti Dhar > > -Origin

RE: Flink Kafka Sink + Schema Registry + Message Headers

Hi Kirti, Kafka Sink supports sending messages with headers. You should implement a HeaderProvider to extract headers from input element. KafkaSink sink = KafkaSink.builder() .setBootstrapServers(brokers) .setRecordSerializer(KafkaRecordSerializationSchema.builder()

RE: Re: Request to provide example codes on ElasticsearchSinkFunction updaterequest

> Could you please share examples on how to "*update*" data using > ElasticsearchSink? > > Thanks > > On Mon, Jan 29, 2024 at 9:07 PM Jiabao Sun wrote: > > > Hi Fidea, > > > > I found some examples in the Java documentation, and I hope they can

RE: Request to provide example codes on ElasticsearchSinkFunction updaterequest

Hi Fidea, I found some examples in the Java documentation, and I hope they can be helpful. private static class TestElasticSearchSinkFunction implements ElasticsearchSinkFunction> { public IndexRequest createIndexRequest(Tuple2 element) { Map json = new HashMap<>(); json.pu

RE: Elasticsearch Sink 1.17.2 error message

Hi Tauseef, We cannot directly write POJO types into Elasticsearch. You can try serializing the TopologyDTO into a JSON string like Jackson before writing it. public static void main(String[] args) throws IOException { try (RestHighLevelClient client = new RestHighLevelClient(

RE: 回复：RE: how to get flink accumulated sink record count

ring the metric in Flink tasks. > > > > > -- 原始邮件 ------ > 发件人: "Jiabao Sun" 发送时间: 2024年1月25日(星期四) 下午3:11 > 收件人: "user" 主题: RE: how to get flink accumulated sink record count > > > > > > I guess ge

RE: how to get flink accumulated sink record count

Hi Enric, I guess getting the metrics[1] might be helpful for you. You can query the numRecordsOut metric by Metrics Reporter[2] or REST API[3]. Best, Jiabao [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/metrics/ [2] https://nightlies.apache.org/flink/flink-docs-relea

RE: Re: Python flink statefun

2024/01/19 09:50:21 Alexandre LANGUILLAT wrote: > Thanks Sun I use now the 3.2 version and it works as described in the > README tutorial! I don't see in the code where the port redirection is > handled tho, eg 8090 for PUT and 8091 for GET (they are in the module.yaml > but dont se

RE: Python flink statefun

Hi Alexandre, I couldn't find the image apache/flink-statefun-playground:3.3.0-1.0 in Docker Hub. You can temporarily use the release-3.2 version. Hi Martijn, did we ignore pushing it to the docker registry? Best, Jiabao [1] https://hub.docker.com/r/apache/flink-statefun-playground/tags On 2

RE: Flink Slow Execution

Hi Dulce, MiniCluster is generally used for local testing and is limited by the resources of a single machine. When more tasks are executed, it may not be able to immediately acquire the resources needed to start the MiniCluster, resulting in slower startup times. If running Flink tasks in a

Re: Flink 1.15: How to fill in the timestamp type of jdbc connector property 'scan.partition.lower-bound'

e execution. > I can't figure out whether this is a problem with the flink connector or > iceberg. > > Jiabao Sun mailto:jiabao@xtransfer.cn>> > 于2024年1月10日周三 18:15写道： >> Hi haifang, >> >> lower-bound and upper-bound are defined as long types, and it

RE: 退订这个邮箱

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Jiabao [1] https://fl

RE: Flink 1.15: How to fill in the timestamp type of jdbc connector property 'scan.partition.lower-bound'

Hi haifang, lower-bound and upper-bound are defined as long types, and it seems difficult to fill in the value of timestamp. However, you may use WHERE t > TIMESTAMP '2022-01-01 07:00:01.333', as JDBC supports filter pushdown. Best, Jiabao On 2024/01/10 08:31:23 haifang luo wrote: > Hello~~ >

RE: Rabbitmq connector for Flink v1.18

Hi Charlotta, The latest news about connector releases is here[1]. You can subscribe to the mailing list or follow the jira issue to get the latest updates. Best, Jiabao [1] https://lists.apache.org/thread/r31f988m57rtjy4s75030pzwrlqybpq2 [2] https://flink.apache.org/what-is-flink/community/

RE: Pending records

Hi rania, Does "pending records" specifically refer to the records that have been read from the source but have not been processed yet? If this is the case, FLIP-33[1] introduces some standard metrics for Source, including "pendingRecords," which can be helpful. However, not all Sources suppor

RE: Does Flink on K8s support log4j2 kafka appender? [Flink log] [Flink Kubernetes Operator]

Hi Chosen, Whether kafka appender is supported or not has no relation to the flink-kubernetes-operator. It only depends on whether log4j2 supports kafka appender. From the error message, it appears that the error is caused by the absence of the log4j-layout-template-json[1] plugin. For the cus

RE: Feature flag functionality on flink

Hi, If it is for simplicity, you can also try writing the flag into an external system, such as Redis、Zookeeper or MySQL, and query the flag from the external system when perform data processing. However, Broadcast State is still the mode that I recommend. Perhaps we only need to encapsulate t

RE: Control who can manage Flink jobs

Hi, I don't have much experience with Beam. If you only need to submit Flink tasks, I would recommend StreamPark[1]. Best, Jiabao [1] https://streampark.apache.org/docs/user-guide/Team On 2023/11/30 09:21:50 Поротиков Станислав Вячеславович via user wrote: > Hello! > Is there any way to contro

RE: Socket timeout when report metrics to pushgateway

Hi, The pushgateway uses push mode to report metrics. When deployed on a single machine under high load, there may be some performance issues. A simple solution is to set up multiple pushgateways and push the metrics to different pushgateways based on different task groups. There are other met

RE: Questions about java enum when convert DataStream to Table

Hi haishui, The enum type cannot be mapped as flink table type directly. I think the easiest way is to convert enum to string type first: DataStreamSource> source = env.fromElements( new Tuple2<>("1", TestEnum.A.name()), new Tuple2<>("2", TestEnum.B.name()) ); Or add a map trans

RE: Suggestions for Open Source FLINK SQL editor

Hi Rajat, I think Apache StreamPark(Incubating) or Apache Zeppelin is a good choice. https://streampark.apache.org/ https://zeppelin.apache.org/ Best, Jiabao On 2023/07/19 16:47:43 Rajat Ahuja wrote: > Hi team, > > I have set up

RE: Re: flink configuration in flink kubernetes operator question about password

Hi tian tian, I think we can use podTemplate to mount kubernetes secrets as file or environment variables. Then we can access the secrets in our flink program. Please refers to https://github.com/apache/flink-kubernetes-operator/blob/main/examples/pod-template.yaml

Re:Re: Re: [ANNOUNCE] New PMC member: Dian Fu

gt;Subject:Re: Re: [ANNOUNCE] New PMC member: Dian Fu > >Congratulations Dian ! > > Best > Yun > > >-- >Sender:Marta Paes Moreira >Date:2020/08/27 17:42:34 >Recipient:Yuan Mei >Cc:Xingbo Huang; jincheng

[ANNOUNCE] New PMC member: Dian Fu

Hi all, On behalf of the Flink PMC, I'm happy to announce that Dian Fu is now part of the Apache Flink Project Management Committee (PMC). Dian Fu has been very active on PyFlink component, working on various important features, such as the Python UDF and Pandas integration, and keeps checking an

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

Thank you for your positive feedback Seth ! Would you please vote in the voting mail thread. Thank you! Best, Jincheng Seth Wiesman 于2020年8月10日周一 下午10:34写道： > I think this sounds good. +1 > > On Wed, Aug 5, 2020 at 8:37 PM jincheng sun > wrote: > >> Hi David, Than

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

n a specific way that I >> haven't thought of -- as in this case, where one can use a JDBC sink from >> Python if one thinks to use DDL. >> >> So I think it would be helpful to be explicit about both what is, and >> what is not, supported in PyFlink. And to have s

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

documentation is currently > organized is that it's relatively straightforward to compare the Python API > with the Java and Scala versions. I'm concerned that if the PyFlink docs > are more independent, it will be challenging to respond to questions about > which features from the

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

Jincheng, >>> >> >>> >> Thanks a lot for bringing up this discussion and the proposal. +1 to >>> >> improve the Python API doc. >>> >> >>> >> I have received many feedbacks from PyFlink beginners about >>> >> th

[DISCUSS] FLIP-133: Rework PyFlink Documentation

Hi folks, Since the release of Flink 1.11, users of PyFlink have continued to grow. As far as I know there are many companies have used PyFlink for data analysis, operation and maintenance monitoring business has been put into production(Such as 聚美优品[1](Jumei), 浙江墨芷[2] (Mozhi) etc.). According t

Re: pyflink的table api 能否根据 filed的值进行判断插入到不同的sink表中

您好，jack： Table API 不用 if/else 直接用类似逻辑即可： val t1 = table.filter('x > 2).groupBy(..) val t2 = table.filter('x <= 2).groupBy(..) t1.insert_into("sink1) t2.insert_into("sink2") Best, Jincheng jack 于2020年6月19日周五 上午10:35写道： > > 测试使用如下结构： > table= t_env.from_path("source") > > if table.filter("

Re:[ANNOUNCE] Yu Li is now part of the Flink PMC

Congratulations Yu! Best, Haibo At 2020-06-17 09:15:02, "jincheng sun" wrote: >Hi all, > >On behalf of the Flink PMC, I'm happy to announce that Yu Li is now >part of the Apache Flink Project Management Committee (PMC). > >Yu Li has been very active

[ANNOUNCE] Yu Li is now part of the Flink PMC

Hi all, On behalf of the Flink PMC, I'm happy to announce that Yu Li is now part of the Apache Flink Project Management Committee (PMC). Yu Li has been very active on Flink's Statebackend component, working on various improvements, for example the RocksDB memory management for 1.10. and keeps che

Re: pyflink数据查询

你好 Jack, > pyflink 从source通过sql对数据进行查询聚合等操作 不输出到sink中，而是可以直接作为结果， 我这边可以通过开发web接口直接查询这个结果，不必去sink中进行查询 我理解你上面说的 【直接作为结果】+ 【web接口查询】已经包含了“sink”的动作。只是这个“sink” 是这样的实现而已。对于您的场景： 1. 如果您想直接将结果不落地（不存储）执行推送的 web页面，可以自定义一个Web Socket的Sink。 2. 如果您不是想直接推送到web页面，而是通过查询拉取结果，那么您上面说的 【直接作为结果】这句话就要描述一下，您想怎样作为结果？我

Re: Python UDF from Java

Thanks Flavio and Thanks Marta, That's a good question as many user want to know that! CC to user-zh mailing list :) Best, Jincheng - Twitter: https://twitter.com/sunjincheng121 - Flavio Pompermaier 于2020年5月1日周五 上午7:04写道： > Yes, that's awesome! I think this would be

Re: [ANNOUNCE] Apache Flink 1.9.3 released

Thanks for your great job, Dian! Best, Jincheng Hequn Cheng 于2020年4月25日周六 下午8:30写道： > @Dian, thanks a lot for the release and for being the release manager. > Also thanks to everyone who made this release possible! > > Best, > Hequn > > On Sat, Apr 25, 2020 at 7:57 PM Dian Fu wrote: > >> Hi e

Re: Flink on Kubernetes - Session vs Job cluster mode and storage

Sounds good. Thank you! Hao Sun On Thu, Feb 27, 2020 at 6:52 PM Yang Wang wrote: > Hi Hao Sun, > > I just post the explanation to the user ML so that others could also have > the same problem. > > Gven the job graph is fetched from the jar, do we still need Zookeeper for >

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations Jingsong! Best, Jincheng Zhu Zhu 于2020年2月24日周一 上午11:55写道： > Congratulations Jingsong! > > Thanks, > Zhu Zhu > > Fabian Hueske 于2020年2月22日周六 上午1:30写道： > >> Congrats Jingsong! >> >> Cheers, Fabian >> >> Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong > >: >> >> > Congratula

[ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

Hi everyone, The Apache Flink community is very happy to announce the release of Apache Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI for the Apache Flink Python API 1.9 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, alw

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

gt;>> > >>> > On Tue, Feb 11, 2020 at 12:17 PM Dian Fu >>> wrote: >>> > >>> > > +1 (non-binding) >>> > > >>> > > - Verified the signature and checksum >>> > > - Pip installed the package successfully: pip install

Re:[ANNOUNCE] Apache Flink 1.10.0 released

Thanks Gary & Yu. Great work! Best, Haibo At 2020-02-12 21:31:00, "Yu Li" wrote: >The Apache Flink community is very happy to announce the release of Apache >Flink 1.10.0, which is the latest major release. > >Apache Flink® is an open-source stream processing framework for >distributed, high-p

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

l > and no exception. > - Try a word count example in IDE with Python 2.7.15 and Python 3.7.5, run > well and no exception. > > Best, > Wei > > > 在 2020年2月10日，19:12，jincheng sun 写道： > > Hi everyone, > > Please review and vote on the release candidate #1 f

[VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the PyFlink version 1.9.2, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * the official Apa

[DISCUSS] Upload the Flink Python API 1.9.x to PyPI for user convenience.

Hi folks, I am very happy to receive some user inquiries about the use of Flink Python API (PyFlink) recently. One of the more common questions is whether it is possible to install PyFlink without using source code build. The most convenient and natural way for users is to use `pip install apache-

Re: [ANNOUNCE] Apache Flink 1.9.2 released

Thanks for being the release manager and the great work Hequn :) Also thanks to the community making this release possible! BTW: I have add the 1.9.2 release to report. Best, Jincheng Hequn Cheng 于2020年1月31日周五 下午6:55写道： > Hi everyone, > > The Apache Flink community is very happy to announce t

Re: [ANNOUNCE] Dian Fu becomes a Flink committer

Congrats Dian Fu and welcome on board! Best, Jincheng Shuo Cheng 于2020年1月16日周四 下午6:22写道： > Congratulations! Dian Fu > > > Xingbo Wei Zhong 于2020年1月16日周四 下午6:13写道： >> jincheng sun > 于2020年1月16日周四 下午5:58写道： >

[ANNOUNCE] Dian Fu becomes a Flink committer

Hi everyone, I'm very happy to announce that Dian accepted the offer of the Flink PMC to become a committer of the Flink project. Dian Fu has been contributing to Flink for many years. Dian Fu played an essential role in PyFlink/CEP/SQL/Table API modules. Dian Fu has contributed several major fea

Re: [DISCUSS] What parts of the Python API should we focus on next ?

On Wed, Dec 18, 2019 at 19:05 jincheng sun > wrote: > >> Also CC user-zh. >> >> Best, >> Jincheng >> >> >> jincheng sun 于2019年12月19日周四 上午10:20写道： >> >>> Hi folks, >>> >>> As release-1.10 is under feature-freeze(

Re: [DISCUSS] What parts of the Python API should we focus on next ?

Also CC user-zh. Best, Jincheng jincheng sun 于2019年12月19日周四 上午10:20写道： > Hi folks, > > As release-1.10 is under feature-freeze(The stateless Python UDF is > already supported), it is time for us to plan the features of PyFlink for > the next release. > > To make sure the

[DISCUSS] What parts of the Python API should we focus on next ?

Hi folks, As release-1.10 is under feature-freeze(The stateless Python UDF is already supported), it is time for us to plan the features of PyFlink for the next release. To make sure the features supported in PyFlink are the mostly demanded for the community, we'd like to get more people involved

Re: [ANNOUNCE] Apache Flink 1.8.3 released

Thanks for being the release manager and the great work Hequn :) Also thanks to the community making this release possible! Best, Jincheng Jark Wu 于2019年12月12日周四 下午3:23写道： > Thanks Hequn for helping out this release and being the release manager. > Great work! > > Best, > Jark > > On Thu, 12 De

Re: [DISCUSS] Drop Kafka 0.8/0.9

+1 for drop it, and Thanks for bring up this discussion Chesnay! Best, Jincheng Jark Wu 于2019年12月5日周四 上午10:19写道： > +1 for dropping, also cc'ed user mailing list. > > > Best, > Jark > > On Thu, 5 Dec 2019 at 03:39, Konstantin Knauf > wrote: > > > Hi Chesnay, > > > > +1 for dropping. I have not

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Hi Stephan, bit +1 for adding this great features to Apache Flink. Regarding where we should place it, put it into Flink core repository or create a separate repository? I prefer put it into main repository and looking forward the more detail discussion for this decision. Best, Jincheng Jingso

Re: Flink 1.8.0 S3 Checkpoints fail with java.security.ProviderException: Could not initialize NSS

I saw similar issue when using alpine linux. https://pkgs.alpinelinux.org/package/v3.3/main/x86/nss Installing this package fixed my problem Hao Sun On Thu, Oct 10, 2019 at 3:46 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hi there, > > I'm getting the fol

Re: Challenges Deploying Flink With Savepoints On Kubernetes

Yep I know that option. That's where get me confused as well. In a HA setup, where do I supply this option (allowNonRestoredState)? This option requires a savepoint path when I start a flink job I remember. And HA does not require the path Hao Sun On Thu, Oct 10, 2019 at 11:16 AM Yun

Re: Challenges Deploying Flink With Savepoints On Kubernetes

I think I overlooked it. Good point. I am using Redis to save the path to my savepoint, I might be able to set a TTL to avoid such issue. Hao Sun On Tue, Sep 24, 2019 at 9:54 AM Yuval Itzchakov wrote: > Hi Hao, > > I think he's exactly talking about the usecase where the JM/

Re: Challenges Deploying Flink With Savepoints On Kubernetes

case well, I do not see a need to start from checkpoint after a bug fix. >From what I know, currently you can use checkpoint as a savepoint as well Hao Sun On Tue, Sep 24, 2019 at 7:48 AM Yuval Itzchakov wrote: > AFAIK there's currently nothing implemented to solve this problem, but &

Re: [ANNOUNCE] Apache Flink 1.8.2 released

Thanks for being the release manager and the great work Jark :) Also thanks to the community making this release possible! Best, Jincheng Jark Wu 于2019年9月13日周五 下午10:07写道： > Hi, > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.8.2, which is the second bugf

Re:[ANNOUNCE] Apache Flink 1.9.0 released

Great news! Thanks Gordon and Kurt!Best, Haibo At 2019-08-22 20:03:26, "Tzu-Li (Gordon) Tai" wrote: >The Apache Flink community is very happy to announce the release of Apache >Flink 1.9.0, which is the latest major release. > >Apache Flink® is an open-source stream processing framework for >dist

Re:Re: FlatMap returning Row<> based on ArrayList elements()

n you register the table when the number of elements/columns and data types are both nondeterministic. Correct me if I misunderstood your meaning. Best, Victor From: Andres Angel Date: Wednesday, August 7, 2019 at 9:55 PM To: Haibo Sun Cc: user Subject: Re: FlatMap returning Row

Re:Re: [ANNOUNCE] Hequn becomes a Flink committer

> Best, >> Jeff Yang >> >> >> Original Message >> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer >> From: Piotr Nowojski >> To: JingsongLee >> CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev

[ANNOUNCE] Hequn becomes a Flink committer

Hi everyone, I'm very happy to announce that Hequn accepted the offer of the Flink PMC to become a committer of the Flink project. Hequn has been contributing to Flink for many years, mainly working on SQL/Table API features. He's also frequently helping out on the user mailing lists and helping

Re:FlatMap returning Row<> based on ArrayList elements()

Hi Andres Angel, I guess people don't understand your problem (including me). I don't know if the following sample code is what you want, if not, can you describe the problem more clearly? StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.fromElements

Re:Pramaters in eclipse with Flink

Hi alaa.abutaha, In fact, your problem is not related to Flink, but how to specify program parameters in Eclipse. I think the following document will help you. https://www.cs.colostate.edu/helpdocs/cmd.pdf Best, Haibo At 2019-07-26 22:02:48, "alaa" wrote: >Hallo > I run this example form

Re:Re: How to write value only using flink's SequenceFileWriter?

Hi Liu Bo, If you haven't customize serializations through the configuration item "io.serializations", the default serializer for Writable objects is org.apache.hadoop.io.serializer.WritableSerialization.WritableSerializer. As you said, when WritableSerializer serialize the NullWritable object

Re:StreamingFileSink part file count reset

Hi Sidhartha, Currently, the part counter is never reset to 0, nor is it allowed to customize the part filename. So I don't think there's any way to reset it right now. I guess the reason why it can't be reset to 0 is that it is concerned that the previous parts will be overwritten. Although

Re:sqlQuery split string

Hi Andres Angel, At present, there seems to be no such built-in function, and you need to register a user-defined function to do that. You can look at the following document to see how to do. https://ci.apache.org/projects/flink/flink-docs-master/dev/table/udfs.html#table-functions Best, Ha

Re:Re: Re: Re: Flink and Akka version incompatibility ..

s in Flink ? regards. On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh wrote: Hi Haibo - Thanks for the clarification .. regards. On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun wrote: Hi Debasish Ghosh, I agree that Flink should shade its Akka. Maybe you misunderstood me. I mean, in the

Re:Re: Re: Flink and Akka version incompatibility ..

so that all the projects that use flink won't hit this kind of issue. Haibo Sun 于2019年7月24日周三 下午4:07写道： Hi, Debasish Ghosh I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that. I recommend to shade Akka dependency of your application because

Re:Re: How to handle JDBC connections in a topology

Hi Stephen, I don't think it's possible to use the same connection pool for the entire topology, because the nodes on the topology may run in different JVMs and on different machines. If you want all operators running in the same JVM to use the same connection pool, I think you can implement

Re:Re: Flink and Akka version incompatibility ..

it's recommended to shade Akka version of my application instead of Flink ? regards. On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun wrote: Hi Debasish Ghosh, Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application de

Re:Flink and Akka version incompatibility ..

Hi Debasish Ghosh, Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your ap

Re:Re: MiniClusterResource class not found using AbstractTestBase

Hi, Juan It is dependent on "flink-runtime-*-tests.jar", so build.sbt should be modified as follows: scalaVersion := "2.11.0" val flinkVersion = "1.8.1" libraryDependencies ++= Seq( "org.apache.flink" %% "flink-test-utils" % flinkVersion % Test, "org.apache.flink" %% "flink-runtime

Re:AW: Re:Unable to build Flink1.10 from source

true Best, Haibo At 2019-07-23 04:54:11, "Yebgenya Lazarkhosrouabadi" wrote: Hi, I used the command mvn clean package -DskipTests -Punsafe-mapr-repo , but it didn’t work. I get the same error. Regards Yebgenya Lazar Von: Haibo Sun Gesendet: Montag, 22. Juli 201

Re:Unable to build Flink1.10 from source

Hi, Yebgenya The reason for this problem can be found in this email (http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/NOTICE-SSL-issue-when-building-flink-mapr-fs-td30757.html). The solution is to add the parameter "-Punsafe-mapr-repo" to the maven command, as given in the e-m

Re:Re: Writing Flink logs into specific file

Hi, Soheil Placing the log configuration file in the resource directory of the job's jar will not be used by Flink, because the log configuration is explicitly specified by the script under the bin directory of Flink and the bootstrap code （for example the BootstrapTools class). If you want to

Re:Re: yarn-session vs cluster per job for streaming jobs

to optimise standard situation where I deploy new version of all my jobs. My current impression that job starts faster in yarn-session mode. Thanks, Maxim. On Thu, Jul 18, 2019 at 4:57 AM Haibo Sun wrote: Hi, Maxim For the concern talking on the first point： If HA and checkpointing are enabled,

Re:Re: Job leak in attached mode (batch scenario)

ot; wrote: Thanks Haibo for the response! Is there any community issue or plan to implement heartbeat mechanism between Dispatcher and Client? If not, should I create one? Regards, Qi On Jul 17, 2019, at 10:19 AM, Haibo Sun wrote: Hi, Qi As far as I know, there is no such mechanism now.

Re:yarn-session vs cluster per job for streaming jobs

Hi, Maxim For the concern talking on the first point： If HA and checkpointing are enabled, AM (the application master, that is the job manager you said) will be restarted by YARN after it dies, and then the dispatcher will try to restore all the previously running jobs correctly. Note that th

Re:Job leak in attached mode (batch scenario)

Hi, Qi As far as I know, there is no such mechanism now. To achieve this, I think it may be necessary to add a REST-based heartbeat mechanism between Dispatcher and Client. At present, perhaps you can add a monitoring service to deal with these residual Flink clusters. Best, Haibo At 2019-0

Re:Converting Metrics from a Reporter to a Custom Events mapping

Hi, Vijay Or can you implement a Reporter that transforms the metrics and sends them directly to a Kinesis Stream? Best, Haibo At 2019-07-16 00:01:36, "Vijay Balakrishnan" wrote: Hi, I need to capture the Metrics sent from a Flink app to a Reporter and transform them to an Events API for

Re:Re: Creating a Source function to read data continuously

Hi, Soheil As Caizhi said, to create a source that implements `SourceFunction`, you can first take a closer look at the example in javadoc (https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/streaming/api/functions/source/SourceFunction.html). Although `Input

Re:State incompatible

Hi, Avi Levi I don't think there's any way to solve this problem right now, and Flink documentation clearly shows that this is not supported. “Trying to restore state, which was previously configured without TTL, using TTL enabled descriptor or vice versa will lead to compatibility failure

Re:Re: [ANNOUNCE] Rong Rong becomes a Flink committer

me:2019年7月12日(星期五) 08:09 To:Hao Sun Cc:Xuefu Z ; dev ; Flink ML Subject:Re: [ANNOUNCE] Rong Rong becomes a Flink committer Thank you all for the warm welcome! It's my honor to become an Apache Flink committer. I will continue to work on this great project and contribute more to

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

gt; *发件人:* Rong Rong > *发送时间:* 星期五, 七月 12, 2019 8:09 上午 > *收件人:* Hao Sun > *抄送:* Xuefu Z; dev; Flink ML > *主题:* Re: [ANNOUNCE] Rong Rong becomes a Flink committer > > Thank you all for the warm welcome! > > It's my honor to become an Apache Flink committer. > I will con

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong. On Thu, Jul 11, 2019, 11:39 Xuefu Z wrote: > Congratulations, Rong! > > On Thu, Jul 11, 2019 at 10:59 AM Bowen Li wrote: > >> Congrats, Rong! >> >> >> On Thu, Jul 11, 2019 at 10:48 AM Oytun Tez wrote: >> >> > Congratulations Rong! >> > >> > --- >> > Oytun Tez >> > >> > *M

Re: Graceful Task Manager Termination and Replacement

I have a common interest in this topic. My k8s recycle hosts, and I am facing the same issue. Flink can tolerate this situation, but I am wondering if I can do better On Thu, Jul 11, 2019, 12:39 Aaron Levin wrote: > Hello, > > Is there a way to gracefully terminate a Task Manager beyond just kil

Re: Flink 1.8.1 release tag missing?

Thanks Bekir Oguz and Chesnay! Sorry for that, I forgot push the tag, I've pushed the tag to the repo now. https://github.com/apache/flink/tree/release-1.8.1 Thanks again, and I'm very sorry for my negligence has caused confusion in your use. Thanks, Jincheng Bekir Oguz 于2019年7月10日周三 上午12:50写道

Re:Cannot catch exception throws by kafka consumer with JSONKeyValueDeserializationSchema

Hi, Zhechao Usually, if you can, share your full exception stack and where you are trying to capture exceptions in your code (preferably with posting your relevant code directly ). That will help us understand and locate the issue you encounter. Best, Haibo At 2019-07-08 14:11:22, "Zhecha

Re:Tracking message processing in my application

Hi, Roey > What do you think about that? I would have some concerns about throughput and latency, so I think that the operators should report state data asynchronously and in batches to minimize the impact of monitoring on the normal business processing. In addition, If the amount of busin

Re: [VOTE] How to Deal with Split/Select in DataStream API

Personally I prefer 3) to keep split/select and correct the behavior. I feel side output is kind of overkill for such a primitive function, and I prefer simple APIs like split/select. Hao Sun On Thu, Jul 4, 2019 at 11:20 AM Xingcan Cui wrote: > Hi folks, > > Two weeks ago, I started

  1   2   3   >