Ziyun Fu created KAFKA-20744:
--------------------------------
Summary: Add back `rack.aware.assignment.tags` config
Key: KAFKA-20744
URL: https://issues.apache.org/jira/browse/KAFKA-20744
Project: Kafka
Issue Type: Improvement
Reporter: Ziyun Fu
I was digging into client-tags a little bit more, and while these are sent to
the broker, and passed into the TaskAssignor I think we have a problem...
client-tags are only one part of the equation. We also need
RACK_AWARE_ASSIGNMENT_TAGS_CONFIG which KIP-1071 lists as "future deprecation".
Not sure why -- from my understanding of
[KIP-708|https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+aware+StandbyTask+assignment+for+Kafka+Streams]
we need this other config too (and we need it inside the TaskAssignor).
Or there is something in KIP-1071 I don't know about? In the end, I did not pay
too close attention to the details of the KIP, and did not even VOTE it... So
it can easily be that everything is correct, and I am off here.
The problem I see is, that in existing code, we verify if
RACK_AWARE_ASSIGNMENT_TAGS_CONFIG matches to configured client.tag.<mytag>
config keys, and if it does not match, we throw an exception. -- However, if
RACK_AWARE_ASSIGNMENT_TAGS_CONFIG is a client config we need to make a RPC
change to be able to send it to the broker. Making it a broker side
group-config (what would actually be better, as we pull all these config into
the broker) is also difficult, because we cannot do the verification between
RACK_AWARE_ASSIGNMENT_TAGS_CONFIG and set client.tag.<mytag> any longer; at
least not client side. We could still verify on the broker when we receive a
heartbeat, but we would need a new error code if the verification fails? In any
case, it seems to be a larger change, that we cannot get into AK 4.3.
While digging around, I also improved some verifications and added a few new
tests about this: [https://github.com/apache/kafka/pull/21724] (I think we want
to be a little bit more strict client side to begin with, to avoid broker side
issue [of course, we might want to verify sent client-tags, broker side, too]--
If you look into the test as whole, you can see what verifications between both
configs we do.
Long story short, I think we cannot do rack-aware standby task assignment any
time soon. We need to figure out these question, and can only add rack-aware
standby-task assignment a-la KIP-708 after we fixed the broker... Not sure what
backward / cross-version compatibility questions will arise.
Maybe we could still do rack-aware standby-task assignment based on client.rack?
All this doesn't really affect what we need on the client, and we can still
start to send offset/lag information (and get this into AK 4.3) and get started
with warmup task support I believe (if I don't find anything else).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
