Hi Hongshun, Thanks for your response!
The big win here is the improved resiliency. We're using the `debezium-avro-confluent` format internally and currently, ANY breaking schema change will break ALL queries. With projection pushdown, queries only break if they use the changed fields. This is a huge win for production resiliency. > the pushdown only be applied in kafka source. Although we can't push projections down into the Kafka brokers, there should still be _some_ performance benefit. If the underlying `DecodingFormat` supports projection pushdown, it can avoid a bunch of work if certain fields aren't needed. I agree it's not as good as storage-layer pushdown but it could still be meaningful for wide tables. > It will make the connector more complex, will it improve a lot? I think the increase in complexity is justified by the improved resiliency. The performance gains are a nice bonus. I'm willing to work with reviewers to see if there's ways to simplify the PR. Also, the PR isn't as big as it looks; `src/main` has only ~600 changed lines, the rest of the ~1200 lines changed are all under `src/test`. Let me know your thoughts. Thanks, Farooq On Mon, Jul 21, 2025 at 10:14 PM Hongshun Wang <loserwang1...@gmail.com> wrote: > Hi Farooq, > Projection pushdown is often used to pushdown projection from flink to > storage server. However, Kafka is a row format, and kafka broker not > support it. Thus, the pushdown only be applied in kafka source. It will > make the connector more complex, will it improve a lot? > > Best > Hongshun > > On Tue, May 27, 2025 at 12:09 AM Farooq Qaiser > <farooq.qai...@shopify.com.invalid> wrote: > > > Hey folks, > > > > Just wanted to give this thread a quick bump for visibility. > > > > Thanks, > > Farooq > > > > On Fri, May 9, 2025 at 12:04 PM Farooq Qaiser <farooq.qai...@shopify.com > > > > wrote: > > > > > Hi folks, > > > > > > Just wanted to share a PR I've been working on to add support for > > > projection pushdown to the Flink Kafka Table connector: > > > https://github.com/apache/flink-connector-kafka/pull/174 > > > > > > I think this could improve performance and even resiliency in some > cases > > > (see the PR description for more details). > > > > > > Would love to hear thoughts from the community on the proposed change > and > > > if there are people who are interested in helping review this. > > > > > > Thanks, > > > Farooq > > > > > >
