Re: Question around Flink's AdaptiveBatchScheduler

Thanks for the reminder. I will review it soon during my free time. > > Venkatakrishnan Sowrirajan 于2024年5月4日周六 10:10写道： > > > Jinrui and Xia > > > > Gentle ping for reviews. > > > > On Mon, Apr 29, 2024, 8:28 PM Venkatakrishnan Sowrirajan < > vsowr...@a

Re: Question around Flink's AdaptiveBatchScheduler

Jinrui and Xia Gentle ping for reviews. On Mon, Apr 29, 2024, 8:28 PM Venkatakrishnan Sowrirajan wrote: > Hi Xia and Jinrui, > > Filed https://github.com/apache/flink/pull/24736 to address the above > described issue. Please take a look whenever you can. > > Thanks > Ven

Re: Question around Flink's AdaptiveBatchScheduler

Hi Xia and Jinrui, Filed https://github.com/apache/flink/pull/24736 to address the above described issue. Please take a look whenever you can. Thanks Venkat On Thu, Apr 18, 2024 at 12:16 PM Venkatakrishnan Sowrirajan < vsowr...@asu.edu> wrote: > Filed https://issues.apache.org/ji

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

Hi Xia, +1 on introducing dynamic parallelism inference for HiveSource. Orthogonal to this discussion, curious, how commonly HiveSource is used these days in the industry given the popularity of table formats/sources like Iceberg, Hudi and Delta lake? Thanks Venkat On Wed, Apr 24, 2024, 7:41 

Re: Question around Flink's AdaptiveBatchScheduler

ink your > example illustrates the significance of this proposal very well. Please > feel free go ahead and address the concerns. > > Best, > Junrui > > Venkatakrishnan Sowrirajan 于2024年4月16日周二 07:01写道： > > > Thanks for adding your thoughts to this discussion. > &

Re: Question around Flink's AdaptiveBatchScheduler

cheduler should support that the parallelism of source > vertices can exceed the job's global max parallelism. > > Best, > Xia > > Venkatakrishnan Sowrirajan 于2024年4月14日周日 10:31写道： > > > Let me state why I think "*jobmanager.adaptive-batch-sche* > >

Re: [DISCUSS] FLIP-438: Make Flink's Hadoop and YARN configuration probing consistent

lexity. > > Regarding the FLIP, in the "YARN configuration override example" > part, I think you should present an example that works correctly > at the moment: "flink.yarn.application.classpath" -> > "yarn.application.classpath". > > Best, >

Re: Question around Flink's AdaptiveBatchScheduler

s the > source vertices are part of the global execution graph, they should also > respect this rule to ensure consistent application of parallelism > constraints. > > Best, > Junrui > > Venkatakrishnan Sowrirajan 于2024年4月12日周五 02:10写道： > > > Gentle bump on this questio

Re: Question around Flink's AdaptiveBatchScheduler

Gentle bump on this question. cc @Becket Qin as well. Regards Venkata krishnan On Tue, Mar 12, 2024 at 10:11 PM Venkatakrishnan Sowrirajan < vsowr...@asu.edu> wrote: > Thanks for the response Lijie and Junrui. Sorry for the late reply. Few > follow up questions. > > >

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

Ron and Lincoln, Great proposal and interesting discussion for adding support for dynamic tables within Flink. At LinkedIn, we are also trying to solve compute/storage convergence for similar problems discussed as part of this FLIP, specifically periodic backfill, bootstrap + nearline update use

[DISCUSS] FLIP-438: Make Flink's Hadoop and YARN configuration probing consistent

Hi Flink devs, I would like to start a discussion on FLIP-XXX: Make Flink's Hadoop and YARN configuration probing consistent . This stems from an earlier discussion thread here

Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

Rui, I assume the current proposal would also handle the case of mixed mode (BATCH + STREAMING within the same app) in the future, right? Regards Venkat On Wed, Mar 27, 2024 at 10:15 AM Venkatakrishnan Sowrirajan < vsowr...@asu.edu> wrote: > This will be a very useful addition to

Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

This will be a very useful addition to Flink UI. Thanks Rui for starting a FLIP for this improvement. Regards Venkata krishnan On Wed, Mar 27, 2024 at 4:49 AM Muhammet Orazov wrote: > Hello Rui, > > Thanks for the proposal! It looks good! > > I have minor clarification from my side: > > The

Re: Flink's treatment to "hadoop" and "yarn" configuration overrides seems unintuitive

IPsSkEDcwzDHxrhej3L0i2YI$ > > > On Tuesday, February 27th, 2024 at 07:08, Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> wrote: > > > > > > > Thanks for sharing your thoughts, Ferenc. > > > > Just my 2 cents, coming from Spark background that use

Re: Question around Flink's AdaptiveBatchScheduler

imit should be effective, otherwise some downstream > > tasks will have no data to process. Source can actually ignore this limit > > because it has no upstream, but this will lead to semantic inconsistency. > > > > Best, > > Lijie > > > > Venkatakrishna

Question around Flink's AdaptiveBatchScheduler

Hi Flink devs, With Flink's AdaptiveBatchScheduler (Note: this is different from AdaptiveScheduler

Re: Flink's treatment to "hadoop" and "yarn" configuration overrides seems unintuitive

ior. > > It would make sense to align these in Flink 2.0 IMO, but I would > be curious about other opinions. > > > > > On Saturday, February 24th, 2024 at 07:36, Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> wrote: > > > > > > > Gentle ping on the

Re: Flink's treatment to "hadoop" and "yarn" configuration overrides seems unintuitive

Gentle ping on the ^^ question to surface this back up again. Any thoughts? Regards Venkata krishnan On Fri, Feb 16, 2024 at 7:32 PM Venkatakrishnan Sowrirajan wrote: > Hi Flink devs, > > Flink supports overriding "hadoop" and "yarn" configuration. As part of &g

Flink's treatment to "hadoop" and "yarn" configuration overrides seems unintuitive

Hi Flink devs, Flink supports overriding "hadoop" and "yarn" configuration. As part of the override mechanism, users have to prefix `hadoop` configs with " flink.hadoop." and the prefix will be removed, while with `yarn` configs users have to prefix it with "flink.yarn." but "flink." only is

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks for the proposal, Jiabao. I agree with Becket if a *Source* is implementing the *SupportsXXXPushDown* (in this case *SupportsFilterPushdown*) interface, then the *Source* (in your FLIP example which is a database) is designed to support filter pushdown. The corresponding Source can have

Re: FW: RE: Close orphaned/stale PRs

Sorry for the delay. I filed https://issues.apache.org/jira/projects/FLINK/issues/FLINK-33343 to track and address the proposal here. Regards Venkata krishnan On Tue, Oct 17, 2023 at 7:49 PM Venkatakrishnan Sowrirajan wrote: > Thanks Martijn, David, Ryan and others for contribut

Re: FW: RE: Close orphaned/stale PRs

the merging and > review work has not been taken on by anyone. I notice spelling mistake prs > that have not been merged (there are 8 with this query > https://urldefense.com/v3/__https://github.com/apache/flink/pulls?q=is*3Apr*is*3Aopen*spelling__;JSslKw!!IKRxdwAv5BmarQ!ecidqKPH8p4_x35QVQoYRVAFoVKPVkG

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

Congratulations Jane! Regards Venkata krishnan On Tue, Oct 17, 2023 at 6:45 AM Matthias Pohl wrote: > Congratulations :) > > On Mon, Oct 16, 2023 at 1:04 PM Jing Ge > wrote: > > > Congrats! > > > > Best regards, > > Jing > > > > On Mon, Oct 16, 2023 at 10:33 AM Yuepeng Pan > > wrote: > > >

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

Congrats Ron! On Mon, Oct 16, 2023, 9:34 AM David Radley wrote: > Congratulations Ron! > > From: Jark Wu > Date: Sunday, 15 October 2023 at 18:57 > To: dev > Cc: ron9@gmail.com > Subject: [EXTERNAL] [ANNOUNCE] New Apache Flink Committer - Ron Liu > Hi, everyone > > On behalf of the PMC,

Re: Close orphaned/stale PRs

Gentle ping to surface this up for more discussions. Regards Venkata krishnan On Tue, Sep 26, 2023 at 4:59 PM Venkatakrishnan Sowrirajan wrote: > Hi Martijn, > > Agree with your point that closing a PR without any review feedback even > after 'X' days is discouraging to a new co

Re: Close orphaned/stale PRs

passed > b) Don't follow the code contribution guide (like commit naming > conventions) > c) Have changes requested but aren't being followed-up by the contributor > > I'm -1 for automatically closing PRs where no maintainers have taken a > review for the reasons I've

Re: [VOTE] FLIP-327: Support switching from batch to stream mode to improve throughput when processing backlog data

+1 (non-binding) On Sun, Sep 24, 2023, 6:49 PM Xintong Song wrote: > +1 (binding) > > Best, > > Xintong > > > > On Sat, Sep 23, 2023 at 10:16 PM Yuepeng Pan > wrote: > > > +1(non-binding), thank you for driving this proposal. > > > > Best, > > Yuepeng Pan. > > At 2023-09-22 14:07:45, "Dong

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Thanks Martijn. Regards Venkata krishnan On Fri, Sep 22, 2023 at 1:51 AM Martijn Visser wrote: > Sure thing, I've made the necessary changes. Thnx for clarifying > > On Thu, Sep 21, 2023 at 8:24 PM Venkatakrishnan Sowrirajan > wrote: > > > > Got it, Martijn. > &

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

3x40wtjb6cp9ky6__;!!IKRxdwAv5BmarQ!f6gg5kDGU-9tqM2rGpbK81L_7sOscG4aVKSjp0RchK1pevsMz_YrhlStS1tXduiLHoxjMP8_BjpJYrQc28-X111i$ > > > > On Wed, Sep 20, 2023 at 9:29 PM Martijn Visser > > > wrote: > > > > > For clarity purposes, this FLIP is being abando

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

LIP-95? > > On Thu, Sep 7, 2023 at 3:01 AM Venkatakrishnan Sowrirajan > wrote: > > > > Hi everyone, > > > > Posted a PR ( > https://urldefense.com/v3/__https://github.com/apache/flink/pull/23313__;!!IKRxdwAv5BmarQ!bKDafb_iRxZNnrFgbAXlTZnZVaYGrfO5tgH2Ppe_aWDCOq

Re: Close orphaned/stale PRs

ly move to dedicated Github Action Runners instead of the current > setup with Azure, but that's primarily blocked by ASF Infra. > > I'm curious what others in the community think. > > Best regards, > > Martijn > > On Tue, Sep 19, 2023 at 10:33 PM Venkatakrishnan Sowrirajan

Close orphaned/stale PRs

Hi Flink devs, There are currently over 1,000 open pull requests (PRs) in the Apache Flink repository, with only 162 having been updated in the last two months

Re: [VOTE] FLIP-361: Improve GC Metrics

+1 (non-binding) On Wed, Sep 13, 2023, 6:55 PM Matt Wang wrote: > +1 (non-binding) > > > Thanks for driving this FLIP > > > > > -- > > Best, > Matt Wang > > > Replied Message > | From | Xintong Song | > | Date | 09/14/2023 09:54 | > | To | | > | Subject | Re: [VOTE] FLIP-361: Improve

Re: [DISCUSS] FLIP-361: Improve GC Metrics

Hi Gyula, Thanks for driving this FLIP. The proposal looks good to me. Only one minor suggestion I have is, can we also include the % GC time spent wrt the overall CPU time especially useful in the cases of TM which helps in easily identifying issues related to GC. Thoughts? Regards Venkata

Re: [VOTE] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

Thanks for driving this FLIP, Archit. +1 (non-binding) Regards Venkata krishnan On Wed, Sep 13, 2023 at 10:47 AM Archit Goyal wrote: > Hi everyone, > > Thanks for reviewing the FLIP-355 Add parent dir of files to classpath > using yarn.provided.lib.dirs : > >

Re: [Discuss] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

年9月13日周三 08:00写道： > > > Hi All, > > > > If there are no further concerns about this FLIP, I will start a vote > > thread tomorrow. > > > > Thanks, > > Archit Goyal > > > > > > From: Venkatakrishnan Sowrirajan > > Date: Thursday, August 24, 2023 at

Re: [VOTE] FLIP-323: Support Attached Execution on Flink Application Completion for Batch Jobs

Thanks for driving this FLIP, Allison. +1 (non-binding Regards Venkata krishnan On Thu, Sep 7, 2023 at 1:20 PM Allison Chang wrote: > Hi everyone, > > Would like to start the VOTE for FLIP-323< >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi everyone, Posted a PR (https://github.com/apache/flink/pull/23313) to add nested fields filter pushdown. Please review. Thanks. Regards Venkata krishnan On Tue, Sep 5, 2023 at 10:04 PM Venkatakrishnan Sowrirajan wrote: > Based on an offline discussion with Becket Qin, I ad

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

gt; > > Cheers, > > > > Jiangjie (Becket) Qin > > > > On Wed, Sep 6, 2023 at 1:12 PM Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> > > wrote: > > > > > Based on the recent discussions in the thread [DISCUSS] FLIP-356: > Suppo

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

(non-binding) > > >> > > >> Yuepeng Pan 于2023年9月1日周五 15:43写道： > > >> > > >>> +1 (non-binding) > > >>> > > >>> Best, > > >>> Yuepeng > > >>> > > >>> > > >>> &

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

o why ProjectionPushDown would work by only relying on the indexes, not > the field names. So I think the field index array is a must have here in > the NestedFieldReferenceExpression. > > Thanks, > > Jiangjie (Becket) Qin > > On Fri, Sep 1, 2023 at 8:12 AM Venkatakrishnan Sowrirajan

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Gentle ping on the vote for FLIP-356: Support Nested fields filter pushdown <https://www.mail-archive.com/dev@flink.apache.org/msg69289.html>. Regards Venkata krishnan On Tue, Aug 29, 2023 at 9:18 PM Venkatakrishnan Sowrirajan wrote: > Sure, will reference this discussion to resume

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

ld you summarize the design of this part > in the future part of the FLIP? This can be easier to get started with in > the future. > > > Best, > Jark > > > On Wed, 30 Aug 2023 at 02:45, Venkatakrishnan Sowrirajan > > wrote: > > > Thanks Jark. Sounds good. > >

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

because the field index should represent > index of the field in the underlying row type. Field references shouldn't > be > aware of the number of inputs. > > Best, > Jark > > > On Tue, 29 Aug 2023 at 02:24, Venkatakrishnan Sowrirajan > > wro

[VOTE] FLIP-356: Support Nested Fields Filter Pushdown

Hi everyone, Thank you all for your feedback on FLIP-356. I'd like to start a vote. Discussion thread: https://lists.apache.org/thread/686bhgwrrb4xmbfzlk60szwxos4z64t7 FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-356%3A+Support+Nested+Fields+Filter+Pushdown Regards Venkata

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

t; it is only for Join operators, and it is only for SQL optimization. It > > looks like there is no requirement for Nested. > > > > Best, > > Jingsong > > > > On Mon, Aug 28, 2023 at 1:13 PM Venkatakrishnan Sowrirajan > > wrote: > > > > &

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

terPushDown and SupportsProjectionPushDown (or others) > from > > the perspective of interface implementers. A stable API can reduce user > > development and change costs, if the current API can fully meet the > > functional requirements at the framework level, I personal suggest

Re: [Discuss] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

Thanks Yang Wang. In that case, whenever you get a chance could you please review the PR? On Wed, Aug 23, 2023, 8:15 PM Yang Wang wrote: > +1 for this FLIP > > Maybe a FLIP is an overkill for this enhancement. > > > Best, > Yang > > Venkatakrishnan Sowrir

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

* without nested fields. Regards Venkata krishnan On Thu, Aug 24, 2023 at 10:13 AM Venkatakrishnan Sowrirajan < vsowr...@asu.edu> wrote: > Jark, > > How about having a separate NestedFieldReferenceExpression, and >> abstracting a common base class

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

uot; for > NestedFieldReferenceExpression and FieldReferenceExpression? This makes > unifying expressions in > "SupportsProjectionPushdown#applyProjections(List > ...)" > possible. > > Best, > Jark > > On Thu, 24 Aug 2023 at 07:00, Venkatakrishnan Sowriraja

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

; > add > > > a > > > > flag for each new filter, > > > > the interface will be filled with lots of flags (e.g., > supportsBetween, > > > > supportsIN). > > > > > > > > 2. NestedFieldReferenceExpression#nestedFieldName should be an array > of > >

Re: [Discuss] FLIP-355: Add parent dir of files to classpath using yarn.provided.lib.dirs

Thanks for the FLIP, Archit. This is definitely quite a useful addition to the *yarn.provided.lib.dirs* . +1. IMO, except for the fact that *yarn.provided.lib.dirs* (platform specific jars can be cached) takes only directories whereas *yarn.ship-files* (user files) takes both files and dirs, the

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

estedProjection() methods from the SupportsFilterPushDown / > SupportsProjectionPushDown interfaces. > > Personally I prefer this alternative. It takes longer to finish the work, > but the API eventually becomes clean and consistent. But I can live with > the current proposal. > > Thank

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Gentle ping for reviews/feedback. On Tue, Aug 15, 2023, 5:37 PM Venkatakrishnan Sowrirajan wrote: > Hi All, > > I am opening this thread to discuss FLIP-356: Support Nested Fields > Filter Pushdown. The FLIP can be found at > https://cwiki.apache.org/confluence/display/FLINK/FLIP

Re: FLINK-20767 - Support for nested fields filter push down

Gentle ping On Wed, Aug 16, 2023, 11:56 AM Venkatakrishnan Sowrirajan wrote: > Forgot to share the link - > https://lists.apache.org/thread/686bhgwrrb4xmbfzlk60szwxos4z64t7 in the > last email. > > Regards > Venkata krishnan > > > On Wed, Aug 16, 2023 at 11:55 AM

Re: FLINK-20767 - Support for nested fields filter push down

Forgot to share the link - https://lists.apache.org/thread/686bhgwrrb4xmbfzlk60szwxos4z64t7 in the last email. Regards Venkata krishnan On Wed, Aug 16, 2023 at 11:55 AM Venkatakrishnan Sowrirajan < vsowr...@asu.edu> wrote: > Btw, this is the FLIP proposal discussion thread. Please s

Re: FLINK-20767 - Support for nested fields filter push down

Btw, this is the FLIP proposal discussion thread. Please share your thoughts. Thanks. Regards Venkata krishnan On Sun, Aug 13, 2023 at 6:35 AM liu ron wrote: > Hi, Venkata krishnan > > Thanks for driving this work, look forward to your FLIP. > > Best, > Ron > > Venka

[DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi All, I am opening this thread to discuss FLIP-356: Support Nested Fields Filter Pushdown. The FLIP can be found at https://cwiki.apache.org/confluence/display/FLINK/FLIP-356%3A+Support+Nested+Fields+Filter+Pushdown This FLIP adds support for pushing down nested fields filters to the

Re: FLINK-20767 - Support for nested fields filter push down

Regards, > Yunhong Zheng (Swuferhong) > > Venkatakrishnan Sowrirajan 于2023年8月7日周一 13:30写道： > > > (Sorry, I pressed send too early) > > > > Thanks for the help @zhengyunhon...@gmail.com. > > > > Agree on not changing the API as much as possible as well as wrt

Re: [DISCUSS] FLIP-323: Support Attached Execution on Flink Application Completion for Batch Jobs

This is definitely a useful feature especially for the flink batch execution workloads using flow orchestrators like Airflow, Azkaban, Oozie etc. Thanks for reviving this issue and starting a FLIP. Regards Venkata krishnan On Mon, Aug 7, 2023 at 4:09 PM Allison Chang wrote: > Hi all, > > I am

Re: FLINK-20767 - Support for nested fields filter push down

. If you have some pointers for that, please let me know. Thanks. Regards Venkata krishnan On Sun, Aug 6, 2023 at 10:23 PM Venkatakrishnan Sowrirajan wrote: > Thanks @zhengyunhon...@gmail.com > Regards > Venkata krishnan > > > On Sun, Aug 6, 2023 at 6:16 PM yh z wrote: > >>

Re: FLINK-20767 - Support for nested fields filter push down

jectedFields`, > > > users have to convert the `List` back to > > int[][] > > > which is an overhead for users. > > > Field indexes (int[][]) is required to project schemas with the > > > utility org.apache.flink.table.connector.Projection. > > >

Re: FLINK-20767 - Support for nested fields filter push down

gt; > This will need a FLIP. > > Thanks, > > Jiangjie (Becket) Qin > > On Tue, Aug 1, 2023 at 11:42 PM Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> > wrote: > > > Thanks for the response. Looking forward to your pointers. In the > > meanwhile,

Re: FLINK-20767 - Support for nested fields filter push down

her into reasonable solutions, which we'll discuss > further later on. > > Best, > Ron > > > Venkatakrishnan Sowrirajan 于2023年7月29日周六 03:31写道： > > > Hi all, > > > > Currently, I am working on adding support for nested fields filter push > > down. In

FLINK-20767 - Support for nested fields filter push down

Hi all, Currently, I am working on adding support for nested fields filter push down. In our use case running Flink on Batch, we found nested fields filter push down is key - without it, it is significantly slow. Note: Spark SQL supports nested fields filter push down. While debugging the code

Re: [VOTE] FLIP-312: Add Yarn ACLs to Flink Containers

Thanks for starting the vote on this one, Archit. +1 (non-binding) Regards Venkata krishnan On Mon, Jun 5, 2023 at 9:55 AM Archit Goyal wrote: > Hi everyone, > > Thanks for all the feedback for FLIP-312: Add Yarn ACLs to Flink > Containers< >

Re: [DISCUSS] FLIP-312: Add Yarn ACLs to Flink Containers

Thanks for the FLIP, Archit. +1 from me as well. This would be very useful for us and others in the community given the same issue was raised earlier as well. Regards Venkata krishnan On Fri, May 12, 2023 at 4:03 PM Becket Qin wrote: > Thanks for the FLIP, Archit. > > The motivation sounds