[
https://issues.apache.org/jira/browse/KAFKA-20744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-20744:
------------------------------------
Description:
KIP-1017 ([KIP-1071: Streams Rebalance
Protocol|https://cwiki.apache.org/confluence/spaces/KAFKA/pages/311627992/KIP-1071+Streams+Rebalance+Protocol])
proposed to deprecate and remove the Kafka Streams config
`rack.aware.assignment.tags`. This is correct by itself, but incomplete,
because we need this config broker side.
The KIP was updated and corrected now. This ticket is about closing this broker
side gap, and add `rack.aware.assignment.tags` as broker and group-level config.
was:
I was digging into client-tags a little bit more, and while these are sent to
the broker, and passed into the TaskAssignor I think we have a problem...
client-tags are only one part of the equation. We also need
RACK_AWARE_ASSIGNMENT_TAGS_CONFIG which KIP-1071 lists as "future deprecation".
Not sure why -- from my understanding of
[KIP-708|https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+aware+StandbyTask+assignment+for+Kafka+Streams]
we need this other config too (and we need it inside the TaskAssignor).
Or there is something in KIP-1071 I don't know about? In the end, I did not pay
too close attention to the details of the KIP, and did not even VOTE it... So
it can easily be that everything is correct, and I am off here.
The problem I see is, that in existing code, we verify if
RACK_AWARE_ASSIGNMENT_TAGS_CONFIG matches to configured client.tag.<mytag>
config keys, and if it does not match, we throw an exception. -- However, if
RACK_AWARE_ASSIGNMENT_TAGS_CONFIG is a client config we need to make a RPC
change to be able to send it to the broker. Making it a broker side
group-config (what would actually be better, as we pull all these config into
the broker) is also difficult, because we cannot do the verification between
RACK_AWARE_ASSIGNMENT_TAGS_CONFIG and set client.tag.<mytag> any longer; at
least not client side. We could still verify on the broker when we receive a
heartbeat, but we would need a new error code if the verification fails? In any
case, it seems to be a larger change, that we cannot get into AK 4.3.
While digging around, I also improved some verifications and added a few new
tests about this: [https://github.com/apache/kafka/pull/21724] (I think we want
to be a little bit more strict client side to begin with, to avoid broker side
issue [of course, we might want to verify sent client-tags, broker side, too]--
If you look into the test as whole, you can see what verifications between both
configs we do.
Long story short, I think we cannot do rack-aware standby task assignment any
time soon. We need to figure out these question, and can only add rack-aware
standby-task assignment a-la KIP-708 after we fixed the broker... Not sure what
backward / cross-version compatibility questions will arise.
Maybe we could still do rack-aware standby-task assignment based on client.rack?
All this doesn't really affect what we need on the client, and we can still
start to send offset/lag information (and get this into AK 4.3) and get started
with warmup task support I believe (if I don't find anything else).
> Add back `rack.aware.assignment.tags` config
> --------------------------------------------
>
> Key: KAFKA-20744
> URL: https://issues.apache.org/jira/browse/KAFKA-20744
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Ziyun Fu
> Assignee: Ziyun Fu
> Priority: Major
> Labels: kip
>
> KIP-1017 ([KIP-1071: Streams Rebalance
> Protocol|https://cwiki.apache.org/confluence/spaces/KAFKA/pages/311627992/KIP-1071+Streams+Rebalance+Protocol])
> proposed to deprecate and remove the Kafka Streams config
> `rack.aware.assignment.tags`. This is correct by itself, but incomplete,
> because we need this config broker side.
> The KIP was updated and corrected now. This ticket is about closing this
> broker side gap, and add `rack.aware.assignment.tags` as broker and
> group-level config.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
