[
https://issues.apache.org/jira/browse/KAFKA-20744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyun Fu reassigned KAFKA-20744:
--------------------------------
Assignee: Ziyun Fu
> Add back `rack.aware.assignment.tags` config
> --------------------------------------------
>
> Key: KAFKA-20744
> URL: https://issues.apache.org/jira/browse/KAFKA-20744
> Project: Kafka
> Issue Type: Improvement
> Reporter: Ziyun Fu
> Assignee: Ziyun Fu
> Priority: Major
>
> I was digging into client-tags a little bit more, and while these are sent to
> the broker, and passed into the TaskAssignor I think we have a problem...
> client-tags are only one part of the equation. We also need
> RACK_AWARE_ASSIGNMENT_TAGS_CONFIG which KIP-1071 lists as "future
> deprecation". Not sure why -- from my understanding of
> [KIP-708|https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+aware+StandbyTask+assignment+for+Kafka+Streams]
> we need this other config too (and we need it inside the TaskAssignor).
> Or there is something in KIP-1071 I don't know about? In the end, I did not
> pay too close attention to the details of the KIP, and did not even VOTE
> it... So it can easily be that everything is correct, and I am off here.
> The problem I see is, that in existing code, we verify if
> RACK_AWARE_ASSIGNMENT_TAGS_CONFIG matches to configured client.tag.<mytag>
> config keys, and if it does not match, we throw an exception. -- However, if
> RACK_AWARE_ASSIGNMENT_TAGS_CONFIG is a client config we need to make a RPC
> change to be able to send it to the broker. Making it a broker side
> group-config (what would actually be better, as we pull all these config into
> the broker) is also difficult, because we cannot do the verification between
> RACK_AWARE_ASSIGNMENT_TAGS_CONFIG and set client.tag.<mytag> any longer; at
> least not client side. We could still verify on the broker when we receive a
> heartbeat, but we would need a new error code if the verification fails? In
> any case, it seems to be a larger change, that we cannot get into AK 4.3.
> While digging around, I also improved some verifications and added a few new
> tests about this: [https://github.com/apache/kafka/pull/21724] (I think we
> want to be a little bit more strict client side to begin with, to avoid
> broker side issue [of course, we might want to verify sent client-tags,
> broker side, too]-- If you look into the test as whole, you can see what
> verifications between both configs we do.
> Long story short, I think we cannot do rack-aware standby task assignment any
> time soon. We need to figure out these question, and can only add rack-aware
> standby-task assignment a-la KIP-708 after we fixed the broker... Not sure
> what backward / cross-version compatibility questions will arise.
> Maybe we could still do rack-aware standby-task assignment based on
> client.rack?
> All this doesn't really affect what we need on the client, and we can still
> start to send offset/lag information (and get this into AK 4.3) and get
> started with warmup task support I believe (if I don't find anything else).
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
