Re: [DISCUSS] Reverting sink metric name changes made in 1.15

2022-10-09 Thread Becket Qin
backwards compatibility tests on metrics to avoid unintended breaking changes like this in the future. Thanks, Jiangjie (Becket) Qin On Sun, Oct 9, 2022 at 10:35 AM Qingsheng Ren wrote: > Hi devs and users, > > I’d like to start a discussion about reverting a breaking change about sink

About the Current22 event

2022-06-15 Thread Becket Qin
committee is actively looking for speakers from the Flink community. Please don't hesitate to submit a talk [2] if you are interested! Thanks, Jiangjie (Becket) Qin [1] https://2022.currentevent.io/website/39543/ [2] https://sessionize.com/current-2022/

Re: [ANNOUNCE] Apache Flink Table Store 0.1.0 released

2022-05-11 Thread Becket Qin
Really excited to see the very first release of the flink-table-store! Kudos to everyone who helped with this effort! Cheers, Jiangjie (Becket) Qin On Wed, May 11, 2022 at 1:55 PM Jingsong Lee wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink

Re: [ANNOUNCE] Apache Flink Table Store 0.1.0 released

2022-05-11 Thread Becket Qin
Really excited to see the very first release of the flink-table-store! Kudos to everyone who helped with this effort! Cheers, Jiangjie (Becket) Qin On Wed, May 11, 2022 at 1:55 PM Jingsong Lee wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink

Re: [Discuss] Creating an Apache Flink slack workspace

2022-05-06 Thread Becket Qin
doing this and come back to where we are right now. Thanks, Jiangjie (Becket) Qin On Fri, May 6, 2022 at 8:55 PM Martijn Visser wrote: > Hi everyone, > > While I see Slack having a major downside (the results are not indexed by > external search engines, you can't link directly to S

Re: flink1.12.2 "Failed to execute job"

2021-04-14 Thread Becket Qin
Hi, Piotr is correct. The cause of this issue is likely because the instantiation of the SplitEnumerator is done in the JM main thread. FLINK-22282 has been created to address this issue. Thanks, Jiangjie (Becket) Qin On Wed, Apr 14, 2021 at 10:32 PM Piotr Nowojski wrote: > Hi, > >

Re: FlinkKafkaProducer Fails with "Topic not present in metadata"

2020-12-10 Thread Becket Qin
Hi Joseph, Thanks for the thorough information. Do you happen to have the trace level logging available? If so, do you mind puttng it somewhere so we can take a look? Thanks, Jiangjie (Becket) Qin On Thu, Dec 3, 2020 at 8:55 PM Joseph Lorenzini wrote: > Hi all, > > I have a

Re: State of Machine Learning with Flink and especially FLIP-39

2020-11-19 Thread Becket Qin
be contributed to Flink. You may also take a look at them. Alink - A machine learning library. https://github.com/alibaba/alink Flink-AI-Extended - A project helps running TF / PyTorch on top of Flink. https://github.com/alibaba/flink-ai-extended Thanks, Jiangjie (Becket) Qin On Fri, Nov 20, 2020

Re: What does Kafka Error sending fetch request mean for the Kafka source?

2020-10-29 Thread Becket Qin
, Jiangjie (Becket) Qin On Thu, Oct 22, 2020 at 11:38 PM John Smith wrote: > Any thoughts this doesn't seem to create duplicates all the time or maybe > it's unrelated as we are still seeing the message and there is no > duplicates... > > On Wed., Oct. 21, 2020, 12:09 p.m. John

Re: Status of a job when a kafka source dies

2020-08-14 Thread Becket Qin
you elaborate a little bit on why you would like to have an exception thrown in your Flink app when Kafka is down, rather than let it run until Kafka is up again? Thanks, JIangjie (Becket) Qin On Fri, Aug 14, 2020 at 4:28 PM Piotr Nowojski wrote: > Hey, > > But do you know what API

Re: ML/DL via Flink

2020-04-28 Thread Becket Qin
earning workflow, with datasets, models and metrics managed. I had a talk about it at the recent Flink Forward virtual event. The videos should be available soon. But feel free to reach out to me for more details. Thanks, Jiangjie (Becket) Qin On Wed, Apr 29, 2020 at 1:12 AM Timo Walther

Re: New kafka producer on each checkpoint

2020-04-13 Thread Becket Qin
A slightly more common case that may cause the producer to be not reusable is when there is no data for long time, the producer won't send any request to the broker and the tansactional.id may also expire on the broker side. On Tue, Apr 14, 2020 at 8:44 AM Becket Qin wrote: > Hi Ma

Re: New kafka producer on each checkpoint

2020-04-13 Thread Becket Qin
have expired on the broker side and the producer may not be reusable anymore. But that should be a rare case. @Piotr Nowojski might know some more reasons that the producers are not reused when it was initially implemented. Thanks, JIangjie (Becket) Qin On Mon, Apr 13, 2020 at 4:59 PM Maxim

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

2020-03-19 Thread Becket Qin
with the same client.id, their metrics will collide with each other. This is why the exception was reported. Thanks, Jiangjie (Becket) Qin On Thu, Mar 19, 2020 at 12:04 AM Rong Rong wrote: > Hi Becket/Till, > > Thanks for the detail explanation. Just to confirm: > the issue in FLINK

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

2020-03-17 Thread Becket Qin
Actually it might be better to create another ticket, FLINK-8093 was mainly complaining about the JMX bean collision when there are multiple tasks running in the same TM. Jiangjie (Becket) Qin On Tue, Mar 17, 2020 at 6:33 PM Becket Qin wrote: > Hi Till, > > It looks FLINK-80

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

2020-03-17 Thread Becket Qin
Hi Till, It looks FLINK-8093 <https://issues.apache.org/jira/browse/FLINK-8093> reports the same issue, although the reported information is not exactly correct, as this should not cause the producer to fail. I'll take care of the ticket. Thanks, Jiangjie (Becket) Qin On Tue, Mar 17, 202

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

2020-03-15 Thread Becket Qin
(Becket) Qin On Mon, Mar 16, 2020 at 12:44 AM Rong Rong wrote: > We also had seen this issue before running Flink apps in a shared cluster > environment. > > Basically, Kafka is trying to register a JMX MBean[1] for application > monitoring. > This is only a WARN suggesting that y

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

2020-02-11 Thread Becket Qin
+1 (binding) - verified signature - Ran word count example successfully. Thanks, Jiangjie (Becket) Qin On Wed, Feb 12, 2020 at 1:29 PM Jark Wu wrote: > +1 > > - checked/verified signatures and hashes > - Pip installed the package successfully: pip install > apache-flink-1.9.2

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

2020-02-11 Thread Becket Qin
+1 (binding) - verified signature - Ran word count example successfully. Thanks, Jiangjie (Becket) Qin On Wed, Feb 12, 2020 at 1:29 PM Jark Wu wrote: > +1 > > - checked/verified signatures and hashes > - Pip installed the package successfully: pip install > apache-flink-1.9.2

Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2

2019-11-04 Thread Becket Qin
quite a huge change by itself. I think there will be at least 10K lines of additions. Because of that, it might worth confine the scope a little bit. So I do not plan to implement event time alignment in FLIP-27. But we did think about it and I think it can be implemented easily afterwards. Thanks, J

Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2

2019-11-04 Thread Becket Qin
by this Sunday. Then I'll update FLIP-27 wiki page. Thanks, Jiangjie (Becket) Qin On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise wrote: > Is there any activity on FLIP-27 that would make it relevant for 1.10 > release? > > Thanks Gary for the update, it provides excellent visibility on curren

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

2019-10-14 Thread Becket Qin
problems of maintaining multiple different repositories. Thanks, Jiangjie (Becket) Qin On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng wrote: > Hi Stephan, > > Big +1 for adding this to Apache Flink! > > As for the problem of whether this should be added to the Flink main > repositor

Re: [DISCUSS] Drop older versions of Kafka Connectors (0.9, 0.10) for Flink 1.10

2019-09-15 Thread Becket Qin
.x. Thanks, Jiangjie (Becket) Qin On Wed, Sep 11, 2019 at 4:26 PM Wesley Peng wrote: > > > on 2019/9/11 16:17, Stephan Ewen wrote: > > We still maintain connectors for Kafka 0.8 and 0.9 in Flink. > > I would suggest to drop those with Flink 1.10 and start supporting only &

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

2019-09-11 Thread Becket Qin
Congrats, Zili! On Thu, Sep 12, 2019 at 9:39 AM Paul Lam wrote: > Congratulations Zili! > > Best, > Paul Lam > > 在 2019年9月12日,09:34,Rong Rong 写道: > > Congratulations Zili! > > -- > Rong > > On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng wrote: > >> Congratulations! >> >> Best, Hequn >> >> On

Re: Kafka consumer behavior with same group Id, 2 instances of apps in separate cluster?

2019-09-03 Thread Becket Qin
, Jiangjie (Becket) Qin On Wed, Sep 4, 2019 at 11:00 AM Ashish Pokharel wrote: > Thanks Becket, > > Sorry for delayed response. That’s what I thought as well. I built a hacky > custom source today directly using Kafka client which was able to join > consumer group etc. which work

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

2019-09-02 Thread Becket Qin
bly reproducible, is it possible to turn on debug level logging on kafka.coordinator.transaction.TransactionCoordinator to see what does the broker say? Thanks, Jiangjie (Becket) Qin On Thu, Aug 29, 2019 at 3:55 PM Tony Wei wrote: > Hi, > > Has anyone run into the same problem? I have updated

Re: Kafka consumer behavior with same group Id, 2 instances of apps in separate cluster?

2019-08-29 Thread Becket Qin
if Flink meets you requirement in another way. Thanks, Jiangjie (Becket) Qin On Thu, Aug 29, 2019 at 9:19 PM ashish pok wrote: > Looks like Flink is using “assign” partitions instead of “subscribe” which > will not allow participating in a group if I read the code correctly. > > Has a

Re: [ANNOUNCE] Apache Flink 1.9.0 released

2019-08-22 Thread Becket Qin
Cheers!! Thanks Gordon and Kurt for driving the release! On Thu, Aug 22, 2019 at 5:36 PM 不常用邮箱 wrote: > Good news! > > Best. > -- > Louis > Email: xu_soft39211...@163.com > > On Aug 22, 2019, at 22:10, Yang Wang wrote: > > Glad to hear that. > Thanks Gordon, Kurt and everyone who had made

Re: [ANNOUNCE] Hequn becomes a Flink committer

2019-08-07 Thread Becket Qin
Congrats, Hequn! Well deserved! On Wed, Aug 7, 2019 at 11:16 AM Zili Chen wrote: > Congrats Hequn! > > Best, > tison. > > > Jeff Zhang 于2019年8月7日周三 下午5:14写道: > >> Congrats Hequn! >> >> Paul Lam 于2019年8月7日周三 下午5:08写道: >> >>> Congrats Hequn! Well deserved! >>> >>> Best, >>> Paul Lam >>> >>> 在

Re: [DISCUSS] Create a Flink ecosystem website

2019-07-19 Thread Becket Qin
[Sorry for the incomplete message. Clicked send by mistake...] I agree with Marta that it might be good to have multi-language support as a mid-term goal. Jiangjie (Becket) Qin On Sat, Jul 20, 2019 at 11:22 AM Becket Qin wrote: > The website is awesome! I really like its conciseness and

Re: [DISCUSS] Create a Flink ecosystem website

2019-07-19 Thread Becket Qin
s? Thanks Robert and Daryl for the great effort. Looking forward to seeing this get published soon!! I agree with Marta that Jiangjie (Becket) Qin On Sat, Jul 20, 2019 at 1:34 AM Marta Paes Moreira wrote: > Hey, Robert. > > I will keep an eye on the overall progress and get start

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

2019-07-11 Thread Becket Qin
Congrats, Rong! On Fri, Jul 12, 2019 at 1:13 AM Xingcan Cui wrote: > Congrats Rong! > > Best, > Xingcan > > On Jul 11, 2019, at 1:08 PM, Shuyi Chen wrote: > > Congratulations, Rong! > > On Thu, Jul 11, 2019 at 8:26 AM Yu Li wrote: > >> Congratulations Rong! >> >> Best Regards, >> Yu >> >> >>

Re: [DISCUSS] Deprecate previous Python APIs

2019-06-11 Thread Becket Qin
+1 on deprecating the old Python API in 1.9 release. Thanks, Jiangjie (Becket) Qin On Wed, Jun 12, 2019 at 9:07 AM Dian Fu wrote: > +1 for this proposal. > > Regards, > Dian > > 在 2019年6月12日,上午8:24,jincheng sun 写道: > > big +1 for the proposal. > > We will so

Re: [SURVEY] Usage of flink-ml and [DISCUSS] Delete flink-ml

2019-05-27 Thread Becket Qin
+1 for removal. Personally I'd prefer marking it as deprecated and remove the module in the next release, just to follow the established procedure. And +1 on removing the `flink-libraries/flink-ml-uber` as well. Thanks, Jiangjie (Becket) Qin On Mon, May 27, 2019 at 5:07 PM jincheng sun wrote

Re: flink-kafka 消费组问题

2019-04-30 Thread Becket Qin
Flink 的Kafka source没有使用 Kafka 本身的Consumer Group管理机制。所以不同Flink job 即使使用同样的 group id 也会消费到同样的消息。 On Mon, Apr 22, 2019 at 1:24 PM 13341000780 <13341000...@163.com> wrote: > hi, 各位大牛好! > kafka作为数据源时,在 Properties > 中设置了group.id,信息,但是同一个group下的consumer中能消费到同一个主题下想同一条消息。 > 另外我单独使用 kafka

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

2019-04-25 Thread Becket Qin
to the listener method? Otherwise users may not be able to easily take action. Thanks, Jiangjie (Becket) Qin On Wed, Apr 24, 2019 at 2:43 PM Jeff Zhang wrote: > Hi Till, > > IMHO, allow adding hooks involves 2 steps. > 1. Provide hook interface, and call these hook in flink (C

Re: [DISCUSS] Create a Flink ecosystem website

2019-04-24 Thread Becket Qin
ly like it! >> >> Becket: Our plan forward is that Congxian is implementing the backend for >> the website. He has already started with the work, but needs at least one >> more week. >> >> >> [Re-sending this email because the first one was blocked on dev@f.

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-21 Thread Becket Qin
y at 2pm CET, > 9pm > > China time about the design of the ecosystem page (see: > > https://github.com/rmetzger/flink-community-tools/issues/4) > > Please let me know if others want to join as well, I can add them to the > > invite. > > > > On Wed, Mar 20, 2

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-19 Thread Becket Qin
> I propose to make the website english-only, but maybe consider allowing > comments in different languages. > If we would make it multi-language, then we might have problems with > people submitting packages in non-english languages. > > > > On Tue, Mar 19, 2019 at 2:42 AM Becke

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-18 Thread Becket Qin
Can you accept and review my proposal for the website? > > > On Sat, Mar 16, 2019 at 3:47 PM Becket Qin wrote: > >> > >> > I have a very capable and motivated frontend developer who would be >> > willing to implement what I've mocked in my proposal. >

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-16 Thread Becket Qin
lso ready. Thanks, Jiangjie (Becket) Qin [1] https://issues.apache.org/jira/browse/INFRA-18010 On Fri, Mar 15, 2019 at 5:39 PM Robert Metzger wrote: > Thank you for reaching out to Infra and the ember client. > When I first saw the Ember repository, I thought it is the whole th

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-14 Thread Becket Qin
. I'll reach out to the author to see if they have any concern over our usage. Apache Infra has not replied to my email regarding some details about the VM. I'll open an infra Jira ticket tomorrow if there is still no response. Thanks, Jiangjie (Becket) Qin [1] https://emberobserver.com/ [2

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-07 Thread Becket Qin
Absolutely! Thanks for the pointer. I'll submit a PR to update the ecosystem page and the navigation. Thanks, Jiangjie (Becket) Qin On Thu, Mar 7, 2019 at 8:47 PM Robert Metzger wrote: > Okay. I will reach out to spark-packages.org and see if they are willing > to share. > >

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-07 Thread Becket Qin
anyways, how about we first do that while working on the dynamic website? Thanks, Jiangjie (Becket) Qin On Thu, Mar 7, 2019 at 4:59 AM Ufuk Celebi wrote: > I like Shaoxuan's idea to keep this a static site first. We could then > iterate on this and make it a dynamic thing. Of course, if w

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-05 Thread Becket Qin
Forgot to provide the link... [1] https://www.apache.org/dev/services.html#blogs (Apache infra services) [2] https://www.apache.org/dev/freebsd-jails (FreeBSD Jail provided by Apache Infra) On Wed, Mar 6, 2019 at 10:46 AM Becket Qin wrote: > Hi Robert, > > Thanks for the

Re: [DISCUSS] Create a Flink ecosystem website

2019-03-05 Thread Becket Qin
ould start with while we are working on the dynamic pages. Thanks, Jiangjie (Becket) Qin On Wed, Mar 6, 2019 at 10:40 AM Shaoxuan Wang wrote: > Hi Becket and Robert, > > I like this idea! Let us roll this out with Flink connectors at the first > beginning. We can start wi

[DISCUSS] Create a Flink ecosystem website

2019-03-05 Thread Becket Qin
/12oCItoLbKrLGuwEUFcCfigezIR2hW3925j1hh3kGp4A/edit# Thanks, Jiangjie (Becket) Qin

Re: Blink对是否对迭代进行了优化?

2019-02-27 Thread Becket Qin
Blink 目前没有对迭代进行特别优化。不过Blink是基于社区1.5.1版本的,建议你用Flink 1.5.1版本测试一下看看是不是还是有区别。 On Thu, Feb 28, 2019 at 10:17 AM Dreamer <1762882...@qq.com> wrote: > 各位,我想请教一下,Blink是否对Flink的迭代进行了优化?如果优化了的话是对批处理进行的优化还是流计算进行的优化?主要采用了哪些优化技术? > >

Re: Re: Re: [Blink]sql client kafka sink 失败

2019-02-24 Thread Becket Qin
ink-1.5.1/opt/connectors/flink-connector-hadoop-compatibility_2.11-1.5.1.jar > > --jar > > /bigdata/flink-1.5.1/opt/connectors/flink-connector-hive_2.11-1.5.1.jar > > --jar /bigdata/flink-1.5.1/opt/sql-client/flink-sql-client-1.5.1.jar > > > > > > > > > >

Re: Re: [Blink]sql client kafka sink 失败

2019-02-22 Thread Becket Qin
能不能看一下运行sql-client.sh的运行参数。具体做法是: 运行sql-client.sh ps | grep sql-client 查看一下其中是不是有这个 flink-connector-kafka-0.11 的 jar. Jiangjie (Becket) Qin On Fri, Feb 22, 2019 at 6:54 PM 张洪涛 wrote: > > > 是包含这个类的 > > > jar -tf flink-connector-kafka-0.11_2.11-*.jar | grep Crc32C > > o

Re: [DISCUSS] Towards a leaner flink-dist

2019-01-27 Thread Becket Qin
, such as connectors and other pluggables, to the same place (maybe a separate category), so the community can share the commonly used libraries as well. Thanks, Jiangjie (Becket) Qin On Sat, Jan 26, 2019 at 2:49 PM Hequn Cheng wrote: > Hi Chesnay, > > Thanks a lot for the proposal! +1 for a lea