The Ingress component uses a custom partitioner which does already compute a hash on the GTS id:
partitioner.class = io.warp10.continuum.KafkaPartitioner On Friday, December 11, 2020 at 11:38:04 AM UTC+1 [email protected] wrote: > > Hey everyone, > > Because of an issue we experienced, I found out that kafka message keys > written by the Ingress are not hashed. They are basically a concatenation > of the class and the labels. This means that all points related to a class > go into the same partition / group of partitions. > > Of course, this is fine in most cases. > > In the case where data of a specific class is significantly bigger than > the rest of the data, this can be an issue (big strings versus int / > floats) where some store threads will have lots of work to do and others > not so much. > > Besides performance concerns, do you think having the key hashed (which > would spread out messages of any class over all partitions but still keep > the order guarantee for a given GTS) could cause problems? Maybe this could > be an option? > > Thanks for your input! > -- You received this message because you are subscribed to the Google Groups "Warp 10 users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/warp10-users/c3a84962-b73d-4bf6-9b63-834bcfb0ceadn%40googlegroups.com.
