[
https://issues.apache.org/jira/browse/KAFKA-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178234#comment-17178234
]
feyman edited comment on KAFKA-10307 at 8/15/20, 11:40 AM:
-----------------------------------------------------------
Hi, [~vvcephei] I just verified with the latest trunk branch and found that the
results of
*setRepartitionTopicMetadataNumberOfPartitions*, specifically the populated
`*repartitionTopicMetadata*` is dependent on the order of *topicGroups*, and
the results of `*repartitionTopicMetadata*` is as I mentioned previously,
Topic22 =Topic30 = Topic9 = 7, Topic17 =5 .
And later on, after the line:
{code:java}
ensureCopartitioning(taskManager.builder().copartitionGroups(),
repartitionTopicMetadata, metadata);{code}
The results will be as you said:
_"So, Topic30 should be resolved to 3, Topic9 should be resolved to 5, and
Topic22 should be resolved to 7."_
So it turns out in the current version, the
`*setRepartitionTopicMetadataNumberOfPartitions*` randomly go over the
topology, and roughly populate the `*repartitionTopicMetadata*`, which then
must be “trimmed” by below function to be finished.
{code:java}
ensureCopartitioning(taskManager.builder().copartitionGroups(),
repartitionTopicMetadata, metadata);{code}
Is it by design to be like this? Thanks!
was (Author: feyman):
Hi, [~vvcephei] I just verified with the latest trunk branch, and found that
the results of `
setRepartitionTopicMetadataNumberOfPartitions`, specifically the populated
`repartitionTopicMetadata` is dependent on the order of `topicGroups`, and the
results of `repartitionTopicMetadata` is as I mentioned previously,
Topic22 =Topic30 = Topic9 = 7, Topic17 =5 .
And later on, after the line:
```
ensureCopartitioning(taskManager.builder().copartitionGroups(),
repartitionTopicMetadata, metadata);
```
The results will be as you said:
"So, Topic30 should be resolved to 3, Topic9 should be resolved to 5, and
Topic22 should be resolved to 7."
So it turns out in the current version, the
`setRepartitionTopicMetadataNumberOfPartitions` randomly go over the topology,
and roughly populate the `repartitionTopicMetadata`, which then must be
“trimmed” by below function to be finished.
```
ensureCopartitioning(taskManager.builder().copartitionGroups(),
repartitionTopicMetadata, metadata);
```
Is it by design to be like this? Thanks!
> Topology cycles in
> KTableKTableForeignKeyInnerJoinMultiIntegrationTest#shouldInnerJoinMultiPartitionQueryable
> -------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-10307
> URL: https://issues.apache.org/jira/browse/KAFKA-10307
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.4.0, 2.5.0, 2.6.0
> Reporter: Boyang Chen
> Priority: Major
> Attachments: repartition_calc.jpg
>
>
> We have spotted a cycled topology for the foreign-key join test
> *shouldInnerJoinMultiPartitionQueryable*, not sure yet whether this is a bug
> in the algorithm or the test only. Used
> [https://zz85.github.io/kafka-streams-viz/] to visualize:
> {code:java}
> Sub-topology: 0
> Source: KTABLE-SOURCE-0000000019 (topics:
> [KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-0000000017-topic])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000020
> Source: KTABLE-SOURCE-0000000032 (topics:
> [KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-0000000030-topic])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000033
> Source: KSTREAM-SOURCE-0000000001 (topics: [table1])
> --> KTABLE-SOURCE-0000000002
> Processor:
> KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000020 (stores:
> [table1-STATE-STORE-0000000000])
> --> KTABLE-FK-JOIN-OUTPUT-0000000021
> <-- KTABLE-SOURCE-0000000019
> Processor:
> KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000033 (stores:
> [INNER-store1])
> --> KTABLE-FK-JOIN-OUTPUT-0000000034
> <-- KTABLE-SOURCE-0000000032
> Processor: KTABLE-FK-JOIN-OUTPUT-0000000021 (stores: [INNER-store1])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000023
> <-- KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000020
> Processor: KTABLE-FK-JOIN-OUTPUT-0000000034 (stores: [INNER-store2])
> --> KTABLE-TOSTREAM-0000000035
> <-- KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000033
> Processor: KTABLE-SOURCE-0000000002 (stores:
> [table1-STATE-STORE-0000000000])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000010
> <-- KSTREAM-SOURCE-0000000001
> Processor: KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000010 (stores:
> [])
> --> KTABLE-SINK-0000000011
> <-- KTABLE-SOURCE-0000000002
> Processor: KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000023 (stores:
> [])
> --> KTABLE-SINK-0000000024
> <-- KTABLE-FK-JOIN-OUTPUT-0000000021
> Processor: KTABLE-TOSTREAM-0000000035 (stores: [])
> --> KSTREAM-SINK-0000000036
> <-- KTABLE-FK-JOIN-OUTPUT-0000000034
> Sink: KSTREAM-SINK-0000000036 (topic: output-)
> <-- KTABLE-TOSTREAM-0000000035
> Sink: KTABLE-SINK-0000000011 (topic:
> KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000009-topic)
> <-- KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000010
> Sink: KTABLE-SINK-0000000024 (topic:
> KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000022-topic)
> <-- KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000023 Sub-topology: 1
> Source: KSTREAM-SOURCE-0000000004 (topics: [table2])
> --> KTABLE-SOURCE-0000000005
> Source: KTABLE-SOURCE-0000000012 (topics:
> [KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000009-topic])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000014
> Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000014 (stores:
> [KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-0000000013])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000015
> <-- KTABLE-SOURCE-0000000012
> Processor: KTABLE-SOURCE-0000000005 (stores:
> [table2-STATE-STORE-0000000003])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000016
> <-- KSTREAM-SOURCE-0000000004
> Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000015 (stores:
> [table2-STATE-STORE-0000000003])
> --> KTABLE-SINK-0000000018
> <-- KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000014
> Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000016 (stores:
> [KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-0000000013])
> --> KTABLE-SINK-0000000018
> <-- KTABLE-SOURCE-0000000005
> Sink: KTABLE-SINK-0000000018 (topic:
> KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-0000000017-topic)
> <-- KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000015,
> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000016 Sub-topology: 2
> Source: KSTREAM-SOURCE-0000000007 (topics: [table3])
> --> KTABLE-SOURCE-0000000008
> Source: KTABLE-SOURCE-0000000025 (topics:
> [KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000022-topic])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000027
> Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000027 (stores:
> [KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-0000000026])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000028
> <-- KTABLE-SOURCE-0000000025
> Processor: KTABLE-SOURCE-0000000008 (stores:
> [table3-STATE-STORE-0000000006])
> --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000029
> <-- KSTREAM-SOURCE-0000000007
> Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000028 (stores:
> [table3-STATE-STORE-0000000006])
> --> KTABLE-SINK-0000000031
> <-- KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000027
> Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000029 (stores:
> [KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-0000000026])
> --> KTABLE-SINK-0000000031
> <-- KTABLE-SOURCE-0000000008
> Sink: KTABLE-SINK-0000000031 (topic:
> KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-0000000030-topic)
> <-- KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000028,
> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000029
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
