Hey Steven,
Your conclusion at this point sounds reasonable to me. That being said, I
think we need to consider a bit more about the extensibility of Flink in
the future. I would be happy to drive some efforts in that direction. So
later on, the timestamp alignment of Iceberg may be able to
退订
| |
zh_ha...@163.com
|
|
邮箱：zh_ha...@163.com
|
回复的原邮件
| 发件人 | Great Info |
| 日期 | 2022年05月07日 13:21 |
| 收件人 | dev@flink.apache.org、user |
| 抄送至 | |
| 主题 | flink Job is throwing depdnecy issue when submitted to clusrer |
I have one flink job which reads files from s3 and
I have one flink job which reads files from s3 and processes them.
Currently, it is running on flink 1.9.0, I need to upgrade my cluster to
1.13.5, so I have done the changes in my job pom and brought up the flink
cluster using 1.13.5 dist.
when I submit my application I am getting the below
I am interested in this, can you share your DDL first? At the same time,
there is no standard to describe hybrid Source DDL.
I also hope that the community can discuss together and share their ideas.
Ran Tao 于2022年5月7日周六 11:59写道：
> HybridSource is a good feature, but now release version did
Congrats Yang and well Deserved!
Best,
Jacky Lau
Yun Gao 于2022年5月7日周六 10:44写道：
> Congratulations Yang!
>
> Best,
> Yun Gao
>
>
>
> --Original Mail --
> Sender:David Morávek
> Send Date:Sat May 7 01:05:41 2022
> Recipients:Dev
> Subject:Re: [ANNOUNCE] New
Hi Xuyang,
Thanks for starting this discussion. Join Hint is a long-time requested
feature.
I have briefly gone through the design doc. Join Hint is a public API for
SQL syntax.
It should work for both streaming and batch SQL. I understand some special
hints
may only work for batch SQL. Could you
Hi Shengkai,
Thanks for driving the proposal, I've been paying attention to the
ververica/flink-sql-gateway[1] project recently. Because the Flink version the
company is currently using is 1.14, so try to upgrade the gateway to
Flink-1.14, because of the changes in flink-table api makes the
HybridSource is a good feature, but now release version did not support
table & sql api, i wonder which time it will be ready to end-users.
And i have implemented a inner version of my company and it works well now.
the implementation of table & sql api may involve some core questions, e.g.
Thanks Lijie and ZhuZhu for the explanation.
I just overlooked the "MARK_BLOCKLISTED". For tasks level, it is indeed
some functionalities the external tools(e.g. kubectl taint) could not
support.
Best,
Yang
Lijie Wang 于2022年5月6日周五 22:18写道：
> Thanks for your feedback, Jiangang and Martijn.
>
might be the same as => might NOT be the same as
On Fri, May 6, 2022 at 8:13 PM Steven Wu wrote:
> The conclusion of this discussion could be that we don't see much value in
> leveraging FLIP-182 with Iceberg source. That would totally be fine.
>
> For me, one big sticking point is the
The conclusion of this discussion could be that we don't see much value in
leveraging FLIP-182 with Iceberg source. That would totally be fine.
For me, one big sticking point is the alignment timestamp for the (Iceberg)
source might be the same as the Flink application watermark.
On Thu, May 5,
Very thanks everyone for all the help on
releasing 1.15.0!
And the doc links are now also fixed, thanks
Peter for pointing out the issues!
Best,
Yun Gao
--
From:Johannes Moser
Send Time:2022 May 6 (Fri.) 17:42
To:dev
Jingsong Lee created FLINK-27538:
Summary: Change flink.version 1.15-SNAPSHOT to 1.15.0 in table
store
Key: FLINK-27538
URL: https://issues.apache.org/jira/browse/FLINK-27538
Project: Flink
Congratulations Yang!
Best,
Yun Gao
--Original Mail --
Sender:David Morávek
Send Date:Sat May 7 01:05:41 2022
Recipients:Dev
Subject:Re: [ANNOUNCE] New Flink PMC member: Yang Wang
Nice! Congrats Yang, well deserved! ;)
On Fri 6. 5. 2022 at 17:53, Peter Huang
Hi Tian,
I am not sure whether you want to get such SDK or just want to refer some other
existing repos to try to implement your own.
For the former purpose, you can find several non-official projects in
github[1], which has not been tested widely.
If you just want to refer to some other
Hi Jeremy,
Could you add a link to the previous discussion?
And you would need to first create an account at
https://cwiki.apache.org/confluence .
Thank you~
Xintong Song
On Fri, May 6, 2022 at 9:26 PM Ber, Jeremy wrote:
> Hello,
>
> I require Confluence Edit Permissions in order to
Not exactly what you asked for, but... Have a look at the Apache Beam
project.
Their goal is quite literally any language on any runner (including Flink).
They recently released an initial version in JavaScript "for fun" and it
took about two weeks to develop. To give you an idea of the effort
Hi folks
Flink is a very awesome project ! ,I want to develop C# SDK which is similar
Java python SDK . Is there any protocol between flink with client SDK ?
Most of the open source communities I know have set up their slack
channels, such as Apache Iceberg [1], Apache Druid [2], etc.
So I think slack can be worth trying.
David is right, there are some cases that need to communicate back and
forth, slack communication will be more effective.
But back
Nice! Congrats Yang, well deserved! ;)
On Fri 6. 5. 2022 at 17:53, Peter Huang wrote:
> Congrats, Yang!
>
>
>
> Best Regards
> Peter Huang
>
> On Fri, May 6, 2022 at 8:46 AM Yu Li wrote:
>
> > Congrats and welcome, Yang!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Fri, 6 May 2022 at 14:48,
Zichen Liu created FLINK-27537:
--
Summary: Remove requirement for Async Sink's RequestEntryT to be
serializable
Key: FLINK-27537
URL: https://issues.apache.org/jira/browse/FLINK-27537
Project: Flink
Congrats, Yang!
Best Regards
Peter Huang
On Fri, May 6, 2022 at 8:46 AM Yu Li wrote:
> Congrats and welcome, Yang!
>
> Best Regards,
> Yu
>
>
> On Fri, 6 May 2022 at 14:48, Paul Lam wrote:
>
> > Congrats, Yang! Well Deserved!
> >
> > Best,
> > Paul Lam
> >
> > > 2022年5月6日 14:38，Yun Tang
Congrats and welcome, Yang!
Best Regards,
Yu
On Fri, 6 May 2022 at 14:48, Paul Lam wrote:
> Congrats, Yang! Well Deserved!
>
> Best,
> Paul Lam
>
> > 2022年5月6日 14:38，Yun Tang 写道：
> >
> > Congratulations, Yang!
> >
> > Best
> > Yun Tang
> >
> > From: Jing Ge
Thanks for your feedback, Jiangang and Martijn.
@Jiangang
> For auto-detecting, I wonder how to make the strategy and mark a node
blocked?
In fact, we currently plan to not support auto-detection in this FLIP. The
part about auto-detection may be continued in a separate FLIP in the
future.
I have mixed feelings about this.
I have been rather visible on stack overflow, and as a result I get a lot
of DMs asking for help. I enjoy helping, but want to do it on a platform
where the responses can be searched and shared.
It is currently the case that good questions on stack overflow
Zichen Liu created FLINK-27536:
--
Summary: Rename method parameter in AsyncSinkWriter
Key: FLINK-27536
URL: https://issues.apache.org/jira/browse/FLINK-27536
Project: Flink
Issue Type:
Hello,
I require Confluence Edit Permissions in order to create a Flink Connector
Template page as discussed via e-mail.
Jeremy
+1 to generic interface for sorted state and Binary***State.
Very happy to be able to go one step further and thank you for your discussion.
Best,
Jingsong
On Fri, May 6, 2022 at 8:37 PM Nico Kruber wrote:
>
> While working a bit more on this, David A and I noticed a couple of things
> that
Thanks for the proposal, Xintong.
While I share the same concerns as those mentioned in the previous
discussion thread, admittedly there are benefits of having a slack channel
as a supplementary way to discuss Flink. The fact that this topic is raised
once a while indicates lasting interests.
Hi everyone,
While I see Slack having a major downside (the results are not indexed by
external search engines, you can't link directly to Slack content unless
you've signed up), I do think that the open source space has progressed and
that Slack is considered as something that's invaluable to
Dong Lin created FLINK-27535:
Summary: Optimize the unit test execution time
Key: FLINK-27535
URL: https://issues.apache.org/jira/browse/FLINK-27535
Project: Flink
Issue Type: Improvement
Timo Walther created FLINK-27534:
Summary: Apply scalafmt to 1.15 branch
Key: FLINK-27534
URL: https://issues.apache.org/jira/browse/FLINK-27534
Project: Flink
Issue Type: Improvement
While working a bit more on this, David A and I noticed a couple of things
that were not matching each other in the proposed APIs:
a) the proposed BinarySortedMultiMapState class didn't actually have getters
that return multiple items per key, and
b) while having a single multi-map like
Hi Xintong,
I'm not sure if slack is the right tool for the job. IMO it works great as
an adhoc tool for discussion between developers, but it's not searchable
and it's not persistent. Between devs, it works fine, as long as the result
of the ad hoc discussions is backported to JIRA/mailing
@Till,
Thanks for volunteering.
@Konstantin,
>From my experience, the effort that requires DockerHub access in the main
project release process is quite limited. I helped Yun Gao on releasing the
1.15.0 images, and what I did was just check out the `flink-docker` repo
and run the release
hi Nicholas,
Thanks for bringing this discussion, we also think it's a useful feature.
Some fine-grained timeout pattern matching can be implemented in CEP which
makes Flink CEP more powerful
Nicholas 于2022年5月5日周四 14:28写道：
> Hi everyone,
>
>
>
>
> Pattern#withIn interface in CEP defines the
hi Nicholas ,
Nicholas 于2022年5月5日周四 14:28写道：
> Hi everyone,
>
>
>
>
> Pattern#withIn interface in CEP defines the maximum time interval in which
> a matching pattern has to be completed in order to be considered valid,
> which interval corresponds to the maximum time gap between first and the
Hi Santhosh,
There's currently an ongoing discussion on this topic in the Dev mailing
list, see https://lists.apache.org/thread/dqw5jw4hmyct47j7m13vdfqcdnbgq0lw
Best regards,
Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser
On Fri, 6 May 2022 at 04:31,
Hi Paul,
Great that you could find something in the SQL standard! I'll try to read
the FLIP once more completely next week to see if I have any more concerns.
Best regards,
Martijn
On Fri, 6 May 2022 at 08:21, Paul Lam wrote:
> I had a look at SQL-2016 that Martijn mentioned, and found that
Hi Shengkai,
Thanks for clarifying.
Best regards,
Martijn
On Fri, 6 May 2022 at 08:40, Shengkai Fang wrote:
> Hi Martijn.
>
> > So this implementation would not rely in any way on Hive, only on Thrift?
>
> Yes. The dependency is light. We also can just copy the iface file from
> the Hive
Hi Vishist,
For future posts, please don't cross-post to both Stackoverflow and the
mailing list. This only causes extra work for volunteers who try to help.
Best regards,
Martijn
On Fri, 6 May 2022 at 09:41, Vishist Bhoopalam
wrote:
> HI
>
> I have explained the problem in the depth in
> If we only support to block nodes manually, then I could not see
the obvious advantages compared with current SRE's approach(via *yarn
rmadmin or kubectl taint*).
I agree with Yang Wang on this.
> To me this sounds yet again like one of those magical mechanisms that
will rarely work just
Chesnay Schepler created FLINK-27533:
Summary: Unstable
AdaptiveSchedulerSimpleITCase#testJobCancellationWhileRestartingSucceeds
Key: FLINK-27533
URL: https://issues.apache.org/jira/browse/FLINK-27533
Hi everyone,
Happy to see that this discussion is very much active. Couple of comments:
> It's not about internal interfaces. Flink itself doesn't provide backward
compatibility for public APIs.
Is that so? In FLIP-196 [1] is explicitly stated "What we guarantee in
terms of stability is that a
Hi Xintong,
it is a pity that we can only have 5 maintainers. Every (patch) release of
flink, flink-statefun, the flink-kubernetes-operator requires a maintainer
to publish the image then, if I am not mistaken. As its mostly different
groups managing the sub-projects, this is quite the
Hi Martijn, Shengkai
>I don't think that the Gateway is a 'core'
>function of Flink which should be included with Flink.
I have a different viewpoint. For SQL users, it provides out-of-box experience.
Just like SQL-client, users can enjoy variable sql experiences after
downloading/building a
Thanks for the valuable design. The auto-detecting can decrease great work
for us. We have implemented the similar feature in our inner flink version.
Below is something that I care about:
1. For auto-detecting, I wonder how to make the strategy and mark a node
blocked? Sometimes the
Hi everyone,
I would like to send out a final reminder. We have already received some
great submissions for FlinkForward San Francisco 2022. Nevertheless, we
decided to extend the deadline by another week to give people a second
chance to work on their abstracts and presentation ideas.
This
Yes!
Thanks to the whole community.
All this involvement keeps impressing me.
> On 06.05.2022, at 01:42, Thomas Weise wrote:
>
> Thank you to all who contributed for making this release happen!
>
> Thomas
>
> On Thu, May 5, 2022 at 7:41 AM Zhu Zhu wrote:
>>
>> Thanks Yun, Till and Joe
Chesnay Schepler created FLINK-27532:
Summary: Drop flink-clients test-jar
Key: FLINK-27532
URL: https://issues.apache.org/jira/browse/FLINK-27532
Project: Flink
Issue Type: Technical
Thank you~
Xintong Song
-- Forwarded message -
From: Xintong Song
Date: Fri, May 6, 2022 at 5:07 PM
Subject: Re: [Discuss] Creating an Apache Flink slack workspace
To: private
Cc: Chesnay Schepler
Hi Chesnay,
Correct me if I'm wrong, I don't find this is *repeatedly*
Thanks, Shengkai for driving. And all for your discussion.
> intergate the Gateway into the Flink code base
After I talk with Shengkai offline and read the topic `Powering HTAP at
ByteDance with Apache Flink` of Flink Forward Asia. I think it is better to
integrate Gateway code into the Flink
Hi
I created the FLINK-27530[1] as the parent ticket. And I
updated it to FLIP.
[1] https://issues.apache.org/jira/browse/FLINK-27530
Thanks
fanrui
On Fri, May 6, 2022 at 4:27 PM Piotr Nowojski wrote:
> Hi,
>
> I'm not sure. Maybe 5 will be fine? Anton, Dawid, what do you think?
>
> Can you
Yun Tang created FLINK-27531:
Summary: Add explict doc on data consistency problem of RocksDB's
map state iterator
Key: FLINK-27531
URL: https://issues.apache.org/jira/browse/FLINK-27531
Project: Flink
fanrui created FLINK-27530:
--
Summary: Support overdraft buffer
Key: FLINK-27530
URL: https://issues.apache.org/jira/browse/FLINK-27530
Project: Flink
Issue Type: Improvement
Components:
Hi,
I'm not sure. Maybe 5 will be fine? Anton, Dawid, what do you think?
Can you create a parent ticket for the whole FLIP to group all of the
issues together?
Also FLIP should be officially voted first.
Best,
Piotrek
pt., 6 maj 2022 o 09:08 rui fan <1996fan...@gmail.com> napisał(a):
> Hi
Ran Tao created FLINK-27529:
---
Summary: HybridSourceSplitEnumerator sourceIndex use error Integer
check
Key: FLINK-27529
URL: https://issues.apache.org/jira/browse/FLINK-27529
Project: Flink
Issue
Jane Chan created FLINK-27528:
-
Summary: Introduce a new configuration option key 'compact-rescale'
Key: FLINK-27528
URL: https://issues.apache.org/jira/browse/FLINK-27528
Project: Flink
Issue
HI
I have explained the problem in the depth in stack Overflow hoping for quick
response since have a dead line to catch.
https://stackoverflow.com/questions/72137036/how-to-deseralize-avro-response-getting-from-datastream-scala-apache-flink
Thanks and Regards
BR Vishist
Alexander Preuss created FLINK-27527:
Summary: Create a file-based Upsert sink for testing internal
components
Key: FLINK-27527
URL: https://issues.apache.org/jira/browse/FLINK-27527
Project:
Jane Chan created FLINK-27526:
-
Summary: Support scaling bucket number for FileStore
Key: FLINK-27526
URL: https://issues.apache.org/jira/browse/FLINK-27526
Project: Flink
Issue Type: New
Hi Anton, Piotrek and Dawid,
Thanks for your help.
I created FLINK-27522[1] as the first task. And I will finish it asap.
@Piotrek, for the default value, do you think it should be less
than 5? What do you think about 3? Actually, I think 5 isn't big.
It's 1 or 3 or 5 that doesn't matter much,
Hi everyone,
thanks for starting this discussion Xintong. I would volunteer as a
maintainer of the flink-statefun Docker repository if you need one.
Cheers,
Till
On Fri, May 6, 2022 at 6:22 AM Xintong Song wrote:
> It seems to me we at least don't have a consensus on dropping the use of
>
luoyuxia created FLINK-27525:
Summary: Add a demo for using kafka sink via Flink SQL
Key: FLINK-27525
URL: https://issues.apache.org/jira/browse/FLINK-27525
Project: Flink
Issue Type:
Congrats, Yang! Well Deserved!
Best,
Paul Lam
> 2022年5月6日 14:38，Yun Tang 写道：
>
> Congratulations, Yang!
>
> Best
> Yun Tang
>
> From: Jing Ge
> Sent: Friday, May 6, 2022 14:24
> To: dev
> Subject: Re: [ANNOUNCE] New Flink PMC member: Yang Wang
>
> Congrats
Xuannan Su created FLINK-27524:
--
Summary: Introduce cache API to DataStream
Key: FLINK-27524
URL: https://issues.apache.org/jira/browse/FLINK-27524
Project: Flink
Issue Type: Sub-task
Hi Martijn.
> So this implementation would not rely in any way on Hive, only on Thrift?
Yes. The dependency is light. We also can just copy the iface file from
the Hive repo and maintain by ourselves.
Best,
Shengkai
Martijn Visser 于2022年5月4日周三 21:44写道：
> Hi Shengkai,
>
> > Actually we will
Xuannan Su created FLINK-27523:
--
Summary: Runtime supports producing and consuming cached
intermediate result
Key: FLINK-27523
URL: https://issues.apache.org/jira/browse/FLINK-27523
Project: Flink
Congratulations, Yang!
Best
Yun Tang
From: Jing Ge
Sent: Friday, May 6, 2022 14:24
To: dev
Subject: Re: [ANNOUNCE] New Flink PMC member: Yang Wang
Congrats Yang and well Deserved!
Best regards,
Jing
On Fri, May 6, 2022 at 7:38 AM Lincoln Lee wrote:
>
fanrui created FLINK-27522:
--
Summary: Ignore max buffers per channel when allocate buffer
Key: FLINK-27522
URL: https://issues.apache.org/jira/browse/FLINK-27522
Project: Flink
Issue Type:
Thanks Martijn, Nicholas, Godfrey, Jark and Jingsong feedback
> I would like to understand why it's complicated to make the upgrades
> problematic
I aggree with Jark's point. The API is not very stable in the Flink
actually. For example, the Gateway relies on the planner. But in
release-1.14
Xuannan Su created FLINK-27521:
--
Summary: FLIP-205: Support Cache in DataStream for Batch Processing
Key: FLINK-27521
URL: https://issues.apache.org/jira/browse/FLINK-27521
Project: Flink
Issue
Congrats Yang and well Deserved!
Best regards,
Jing
On Fri, May 6, 2022 at 7:38 AM Lincoln Lee wrote:
> Congratulations Yang!
>
> Best,
> Lincoln Lee
>
>
> Őrhidi Mátyás 于2022年5月6日周五 12:46写道：
>
> > Congrats Yang! Well deserved!
> > Best,
> > Matyas
> >
> > On Fri, May 6, 2022 at 5:30 AM
I had a look at SQL-2016 that Martijn mentioned, and found that
maybe we could follow the transaction savepoint syntax.
SAVEPOINT
RELEASE SAVEPOINT
These savepoint statements are supported in lots of databases, like
Oracle[1], PG[2], MariaDB[3].
They’re usually used in the middle of a SQL
74 matches
Mail list logo
