Hello. Thanks for sharing this with us. I think it's not easy work to support Deduplicate in streaming mode. For example, in the keep first-row case, we need to memorize all records during the running. Because the first row may be deleted at some point. One idea to work around is to use window deduplication to give the engine a hint[1] about how long to keep all msg in the memory.
Best, Shengkai [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/window-deduplication/ Qing Lim <q....@mwam.com> 于2023年3月4日周六 01:00写道： > Hi Flink User group, > > > > I bumped into the following error when using FlinkSQL, version 1.15.3 > > > > - StreamPhysicalDeduplicate doesn't support consuming update and > delete changes which is produced by node ChangelogNormalize > > > > Essentially I was trying to dedup over an upsert kafka topic. > > > > I wonder if there’s workaround? I also don’t fully understand why it is > not supported, is it just a missing feature or is there some underlying > fundamental reason making this hard/impossible? > > > > Kind regards > > > > *Qing Lim *| Marshall Wace LLP, George House, 131 Sloane Street, London | > E-mail: > q....@mwam.com | Tel: +44 207 925 4865 > > > > > > This e-mail and any attachments are confidential to the addressee(s) and > may contain information that is legally privileged and/or confidential. If > you are not the intended recipient of this e-mail you are hereby notified > that any dissemination, distribution, or copying of its content is strictly > prohibited. If you have received this message in error, please notify the > sender by return e-mail and destroy the message and all copies in your > possession. > > > To find out more details about how we may collect, use and share your > personal information, please see https://www.mwam.com/privacy-policy. > This includes details of how calls you make to us may be recorded in order > for us to comply with our legal and regulatory obligations. > > > To the extent that the contents of this email constitutes a financial > promotion, please note that it is issued only to and/or directed only at > persons who are professional clients or eligible counterparties as defined > in the FCA Rules. Any investment products or services described in this > email are available only to professional clients and eligible > counterparties. Persons who are not professional clients or eligible > counterparties should not rely or act on the contents of this email. > > > Marshall Wace LLP is authorised and regulated by the Financial Conduct > Authority. Marshall Wace LLP is a limited liability partnership registered > in England and Wales with registered number OC302228 and registered office > at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving > this e-mail as a client, or an investor in an investment vehicle, managed > or advised by Marshall Wace North America L.P., the sender of this e-mail > is communicating with you in the sender's capacity as an associated or > related person of Marshall Wace North America L.P. ("MWNA"), which is > registered with the US Securities and Exchange Commission ("SEC") as an > investment adviser. Registration with the SEC does not imply that MWNA or > its employees possess a certain level of skill or training. >
