I'm not sure that's feasible in this case, but I'll have a look! Thanks, Ben
-----Original Message----- From: Liam Clarke-Hutchinson <liam.cla...@adscale.co.nz> Sent: 06 May 2020 19:47 To: users@kafka.apache.org Subject: EXTERNAL: Re: Separate Kafka partitioning from key compaction Could you deploy a Kafka Streams app that implemented your desired partitioning? Obviously this would require a duplication in topics between those produced to initially, and those partitioned the way you'd like, but it would solve the issue you're having. On Wed, 6 May 2020, 10:25 pm Young, Ben, <ben.yo...@fisglobal.com.invalid> wrote: > Hi, > > We have a use case where we'd like the partition a key is hashed to, > to be a subset of the keys that are used for compaction. It would be > really cool if there was a built in hashing strategy that could help > us as we're potentially using Kafka from multiple languages and it > could be hard to standardise. > > For instance, large message processing. We have keys like "msg1:1/3", > "msg1:2/3" and "msg:3/3". We'd like all messages to be retained by > compaction, but all of these messages to go to the same partition... > There's lots of similar use cases where we'd like compaction to keep > more than we'd use for partitioning. > > Obviously we could write our own hashing etc but that's hard when our > main producers and consumers are in C# and we want to integration with KSQL > etc. > > My desert island solution would be to have the partition key > optionally called out by something like braces in Redis ( > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fredi > slabs.com%2Fblog%2Fredis-clustering-best-practices-with-keys%2F&da > ta=02%7C01%7CBen.Young%40fisglobal.com%7Cafc29b19d3474c65e40608d7f1ed777c%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C637243874538623413&sdata=k9pAZ%2BY5wGU28Krg5ZJhC9f11AoY3r1g9wR5QOglitE%3D&reserved=0) > , but the whole key used for log compaction. This wouldn't be backwards > compatible so I guess it would have to be a new strategy... > > Does anyone else have requirements like this? How have they solved them? > > Thanks, > Ben Young > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) > delete the message and all copies; (ii) do not disclose, distribute or > use the message in any manner; and (iii) notify the sender > immediately. In addition, please be aware that any message addressed > to our domain is subject to archiving and review by persons other than > the intended recipient. FIS is a trading name of the following companies: > Advanced Portfolio Technologies Ltd (No: > 6312142) | Clear2Pay Limited (No: 5792457) | Decalog (UK) Limited (No: > 2567370) | FIS Apex (International) Limited (No: 2999960) | FIS Apex > (UK) Limited (No. 3573008) | FIS Consulting Services (UK) Limited (No: > 2486794) > | FIS Derivatives Utility Services (UK) Limited (No: 9398140) | FIS > | Energy > Solutions Limited (No: 1889028) | FIS Global Execution Services > Limited (No. 3127109) | FIS Global Trading (UK) Limited (No: 2523114) > | FIS Investment Systems (UK) Limited (No: 1366010) | FIS Sherwood > Systems Group Limited (No: 982833) | FIS Systems Limited (No: 1937159) > | FIS Treasury Systems (Europe) Limited (No: 2624209) | FIS Treasury > Systems (UK) Limited > (No: 2893376) | GL Settle Limited (No: 2396127) | Integrity Treasury > Solutions Europe Limited (No: 3289271) | Monis Software Limited (No: > 2333925) | Reech Capital Limited (No: 3649490) | Solutions Plus > Consulting Services Limited (No: 3839487) | Valuelink Information > Services Limited > (No: 3827424) all registered in England & Wales with their registered > office at 25 Canada Square, London E14 5LQ | FIS Global Execution > Services Limited is authorised and regulated by the Financial Conduct > Authority | Certegy Card Services Limited (No: 3517639) | Certegy France > Limited (No: > 2557650) | eFunds International Limited (No: 1930117) | Fidelity > Information Services Limited (No: 2225203) | FIS Payments (UK) Limited (No: > 4215488) | Metavante Technologies Limited (No: 2659326) all registered > in England & Wales with their registered office at 1st Floor Tricorn > House, > 51-53 Hagley Road, Edgbaston, Birmingham, West Midlands, B16 8TU, > United Kingdom | FIS Payments (UK) Limited is authorised and regulated > by the Financial Conduct Authority; some services are covered by the > Financial Ombudsman Service (in the UK). Clear2Pay Limited, Registered > in Scotland (No SC157659), Registered Office: Clear2Pay House, > Pitreavie Court, Pitreavie Business Park Queensferry Rd, Dunfermline, > Fife, SS, KY11 8UU, Scotland | FIS eProcess Intelligence LLC (UK > Branch), UK Establishment Registered in England & Wales (No: > FC16527/Branch No. BR000355), Registered Branch Office: 25 Canada > Square, London, E14 5LQ; FIS eProcess Intelligence LLC is a limited > liability company formed in the USA registered on file with the Office > of the Delaware Secretary of State, Division of Corporations (File No. > 2032143), Head Office: 601 Riverside Avenue, Jacksonville Florida, > FL32204, USA | FIS Investment Systems LLC, UK Establishment Registered in > England & Wales (No: FC033836/Branch No. > BR018923), Registered Branch Office: 25 Canada Square, London, E14 > 5LQ; FIS Investment Systems LLC is a limited liability company formed > in the USA registered on file with the Office of the Delaware > Secretary of State, Division of Corporations (File No. 0881255), Head Office: > 377 E. > Butterfield Road, Suite 800, Lombard, IL 60148, USA | Calls to and > from the companies may be recorded for quality purposes. | All of the > named companies are part of FIS (Fidelity National Information Services, > Inc.). > The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. FIS is a trading name of the following companies: Advanced Portfolio Technologies Ltd (No: 6312142) | Clear2Pay Limited (No: 5792457) | Decalog (UK) Limited (No: 2567370) | FIS Apex (International) Limited (No: 2999960) | FIS Apex (UK) Limited (No. 3573008) | FIS Consulting Services (UK) Limited (No: 2486794) | FIS Derivatives Utility Services (UK) Limited (No: 9398140) | FIS Energy Solutions Limited (No: 1889028) | FIS Global Execution Services Limited (No. 3127109) | FIS Global Trading (UK) Limited (No: 2523114) | FIS Investment Systems (UK) Limited (No: 1366010) | FIS Sherwood Systems Group Limited (No: 982833) | FIS Systems Limited (No: 1937159) | FIS Treasury Systems (Europe) Limited (No: 2624209) | FIS Treasury Systems (UK) Limited (No: 2893376) | GL Settle Limited (No: 2396127) | Integrity Treasury Solutions Europe Limited (No: 3289271) | Monis Software Limited (No: 2333925) | Reech Capital Limited (No: 3649490) | Solutions Plus Consulting Services Limited (No: 3839487) | Valuelink Information Services Limited (No: 3827424) all registered in England & Wales with their registered office at 25 Canada Square, London E14 5LQ | FIS Global Execution Services Limited is authorised and regulated by the Financial Conduct Authority | Certegy Card Services Limited (No: 3517639) | Certegy France Limited (No: 2557650) | eFunds International Limited (No: 1930117) | Fidelity Information Services Limited (No: 2225203) | FIS Payments (UK) Limited (No: 4215488) | Metavante Technologies Limited (No: 2659326) all registered in England & Wales with their registered office at 1st Floor Tricorn House, 51-53 Hagley Road, Edgbaston, Birmingham, West Midlands, B16 8TU, United Kingdom | FIS Payments (UK) Limited is authorised and regulated by the Financial Conduct Authority; some services are covered by the Financial Ombudsman Service (in the UK). Clear2Pay Limited, Registered in Scotland (No SC157659), Registered Office: Clear2Pay House, Pitreavie Court, Pitreavie Business Park Queensferry Rd, Dunfermline, Fife, SS, KY11 8UU, Scotland | FIS eProcess Intelligence LLC (UK Branch), UK Establishment Registered in England & Wales (No: FC16527/Branch No. BR000355), Registered Branch Office: 25 Canada Square, London, E14 5LQ; FIS eProcess Intelligence LLC is a limited liability company formed in the USA registered on file with the Office of the Delaware Secretary of State, Division of Corporations (File No. 2032143), Head Office: 601 Riverside Avenue, Jacksonville Florida, FL32204, USA | FIS Investment Systems LLC, UK Establishment Registered in England & Wales (No: FC033836/Branch No. BR018923), Registered Branch Office: 25 Canada Square, London, E14 5LQ; FIS Investment Systems LLC is a limited liability company formed in the USA registered on file with the Office of the Delaware Secretary of State, Division of Corporations (File No. 0881255), Head Office: 377 E. Butterfield Road, Suite 800, Lombard, IL 60148, USA | Calls to and from the companies may be recorded for quality purposes. | All of the named companies are part of FIS (Fidelity National Information Services, Inc.).
