Re: FLINK-20767 - Support for nested fields filter push down

Gentle ping On Wed, Aug 16, 2023, 11:56 AM Venkatakrishnan Sowrirajan wrote: > Forgot to share the link - > https://lists.apache.org/thread/686bhgwrrb4xmbfzlk60szwxos4z64t7 in the > last email. > > Regards > Venkata krishnan > > > On Wed, Aug 16, 2023 at 11:55 AM Venkatakrishnan Sowrirajan < >

Re: FLINK-20767 - Support for nested fields filter push down

Forgot to share the link - https://lists.apache.org/thread/686bhgwrrb4xmbfzlk60szwxos4z64t7 in the last email. Regards Venkata krishnan On Wed, Aug 16, 2023 at 11:55 AM Venkatakrishnan Sowrirajan < vsowr...@asu.edu> wrote: > Btw, this is the FLIP proposal discussion thread. Please share your >

Re: FLINK-20767 - Support for nested fields filter push down

Btw, this is the FLIP proposal discussion thread. Please share your thoughts. Thanks. Regards Venkata krishnan On Sun, Aug 13, 2023 at 6:35 AM liu ron wrote: > Hi, Venkata krishnan > > Thanks for driving this work, look forward to your FLIP. > > Best, > Ron > > Venkatakrishnan Sowrirajan

Re: FLINK-20767 - Support for nested fields filter push down

Hi, Venkata krishnan Thanks for driving this work, look forward to your FLIP. Best, Ron Venkatakrishnan Sowrirajan 于2023年8月13日周日 14:34写道： > Thanks Yunhong. That's correct. I am able to make it work locally. > Currently, in the process of writing a FLIP for the necessary changes to > the

Re: FLINK-20767 - Support for nested fields filter push down

Thanks Yunhong. That's correct. I am able to make it work locally. Currently, in the process of writing a FLIP for the necessary changes to the SupportsFilterPushDown API to support nested fields filter push down. Regards Venkata krishnan On Mon, Aug 7, 2023 at 8:28 PM yh z wrote: > Hi

Re: FLINK-20767 - Support for nested fields filter push down

Hi Venkatakrishnan, Sorry for the late reply. I have looked at the code and feel like you need to modify the logic of the ExpressionConverter.visit(FieldReferenceExpression expression) method to support nested types, which are not currently supported in currently code. Regards, Yunhong Zheng

Re: FLINK-20767 - Support for nested fields filter push down

(Sorry, I pressed send too early) Thanks for the help @zhengyunhon...@gmail.com. Agree on not changing the API as much as possible as well as wrt simplifying Projection pushdown with nested fields eventually as well. In terms of the code itself, currently I am trying to leverage the

Re: FLINK-20767 - Support for nested fields filter push down

Thanks @zhengyunhon...@gmail.com Regards Venkata krishnan On Sun, Aug 6, 2023 at 6:16 PM yh z wrote: > Hi, Venkatakrishnan, > I think this is a very useful feature. I have been focusing on the > development of the flink-table-planner module recently, so if you need some > help, I can assist

Re: FLINK-20767 - Support for nested fields filter push down

Hi, Venkatakrishnan, I think this is a very useful feature. I have been focusing on the development of the flink-table-planner module recently, so if you need some help, I can assist you in completing the development of some sub-tasks or code review. Returning to the design itself, I think it's

Re: FLINK-20767 - Support for nested fields filter push down

Hi Jark, If the FieldReferenceExpression contains an int[] to support a nested field reference, List (or FieldReferenceExpression[]) and int[][] are actually equivalent. If we are designing this from scratch, personally I prefer using List for consistency, i.e. always resolving everything to

Re: FLINK-20767 - Support for nested fields filter push down

Hi, I agree with Becket that we may need to extend FieldReferenceExpression to support nested field access (or maybe a new NestedFieldReferenceExpression). But I have some concerns about evolving the SupportsProjectionPushDown.applyProjection. A projection is much simpler than Filter Expression

Re: FLINK-20767 - Support for nested fields filter push down

Thanks Becket for the suggestion. That makes sense. Let me try it out and get back to you. Regards Venkata krishnan On Tue, Aug 1, 2023 at 9:04 AM Becket Qin wrote: > This is a very useful feature in practice. > > It looks to me that the key issue here is that Flink ResolvedExpression > does

Re: FLINK-20767 - Support for nested fields filter push down

This is a very useful feature in practice. It looks to me that the key issue here is that Flink ResolvedExpression does not have necessary abstraction for nested field access. So the Calcite RexFieldAccess does not have a counterpart in the ResolvedExpression. The FieldReferenceExpression only

Re: FLINK-20767 - Support for nested fields filter push down

Thanks for the response. Looking forward to your pointers. In the meanwhile, let me figure out how we can implement it. Will keep you posted. On Mon, Jul 31, 2023, 11:43 PM liu ron wrote: > Hi, Venkata > > Thanks for reporting this issue. Currently, Flink doesn't support nested > filter

Re: FLINK-20767 - Support for nested fields filter push down

Hi, Venkata Thanks for reporting this issue. Currently, Flink doesn't support nested filter pushdown. I also think that this optimization would be useful, especially for jobs, which may need to read a lot of data from the parquet or orc file. We didn't move forward with this for some priority

FLINK-20767 - Support for nested fields filter push down

Hi all, Currently, I am working on adding support for nested fields filter push down. In our use case running Flink on Batch, we found nested fields filter push down is key - without it, it is significantly slow. Note: Spark SQL supports nested fields filter push down. While debugging the code