[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse
[
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14070933#comment-14070933
]
Jun Rao commented on KAFKA-1549:
Thanks for the patch. Could we define inReadLock and inWriteLock in the same
way as inLock? Since we have code like
inlock(lock) {
...
}
It would be consistent if we can just do
inReadLock(readWriteLock) {
...
}
> dead brokers coming in the TopicMetadataResponse
>
>
> Key: KAFKA-1549
> URL: https://issues.apache.org/jira/browse/KAFKA-1549
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.8.2
> Environment: trunk
>Reporter: nicu marasoiu
>Assignee: Jun Rao
> Attachments:
> kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch,
>
> kafka-1549__without_implicit__only_last_seen_alive_brokers_to_be_responded_part_of_the_top.patch
>
>
> JunRao confirming my observation that brokers are only added to the
> metadataCache, never removed: "The way that we update liveBrokers in
> MetadataCache.updateCache() doesn't seem right. We only add newly received
> live brokers to the list. However, there could be existing brokers in that
> list that are now dead. Those dead brokers shouldn't be returned to the
> clients. We should probably just take the new live broker list and cache it."
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14070751#comment-14070751 ] nicu marasoiu commented on KAFKA-1549: -- [~jkreps] [~junrao] Hi, can you help me choose from the patches? Thank you, Nicu > dead brokers coming in the TopicMetadataResponse > > > Key: KAFKA-1549 > URL: https://issues.apache.org/jira/browse/KAFKA-1549 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.2 > Environment: trunk >Reporter: nicu marasoiu >Assignee: Jun Rao > Attachments: > kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch, > > kafka-1549__without_implicit__only_last_seen_alive_brokers_to_be_responded_part_of_the_top.patch > > > JunRao confirming my observation that brokers are only added to the > metadataCache, never removed: "The way that we update liveBrokers in > MetadataCache.updateCache() doesn't seem right. We only add newly received > live brokers to the list. However, there could be existing brokers in that > list that are now dead. Those dead brokers shouldn't be returned to the > clients. We should probably just take the new live broker list and cache it." -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069189#comment-14069189 ] nicu marasoiu commented on KAFKA-1549: -- added an implementation without the implicit: this required a pair of function definitions in each client class (to my current scala knowleadge) > dead brokers coming in the TopicMetadataResponse > > > Key: KAFKA-1549 > URL: https://issues.apache.org/jira/browse/KAFKA-1549 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.2 > Environment: trunk >Reporter: nicu marasoiu >Assignee: Jun Rao > Attachments: > kafka-1549__only_last_seen_alive_brokers_to_be_responded_part_of_the_topic_metadata_refres.patch, > > kafka-1549__without_implicit__only_last_seen_alive_brokers_to_be_responded_part_of_the_top.patch > > > JunRao confirming my observation that brokers are only added to the > metadataCache, never removed: "The way that we update liveBrokers in > MetadataCache.updateCache() doesn't seem right. We only add newly received > live brokers to the list. However, there could be existing brokers in that > list that are now dead. Those dead brokers shouldn't be returned to the > clients. We should probably just take the new live broker list and cache it." -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068442#comment-14068442 ] nicu marasoiu commented on KAFKA-1549: -- 2. Indeed, to avoid that overhead and make it atomical (i.e. brokers and topic metadata from the same snapshot of metadata). However the atomicity is not required by the functionality. > dead brokers coming in the TopicMetadataResponse > > > Key: KAFKA-1549 > URL: https://issues.apache.org/jira/browse/KAFKA-1549 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.2 > Environment: trunk >Reporter: nicu marasoiu >Assignee: Jun Rao > Attachments: > KAFKA-1549__dead_brokers_coming_in_the_TopicMetadataResponse_.patch > > > JunRao confirming my observation that brokers are only added to the > metadataCache, never removed: "The way that we update liveBrokers in > MetadataCache.updateCache() doesn't seem right. We only add newly received > live brokers to the list. However, there could be existing brokers in that > list that are now dead. Those dead brokers shouldn't be returned to the > clients. We should probably just take the new live broker list and cache it." -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse
[
https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068165#comment-14068165
]
Jun Rao commented on KAFKA-1549:
Thanks for the patch. A couple of comments.
1. inReadLock seems like a generally useful util. We probably can add the
following to the Utils file where inLock lives, and change all existing usage
of readWriteLock to use those utils.
def inReadLock[T](lock: ReadWriteLock)(fun: => T): T = {
def inWriteLock[T](lock: ReadWriteLock)(fun: => T): T = {
2. Not sure what the purpose of the change in KafkaApis is. Is that to avoid
the overhead of an extra readLock? It's better to limit the usage of the lock
inside the MetadataCache.
> dead brokers coming in the TopicMetadataResponse
>
>
> Key: KAFKA-1549
> URL: https://issues.apache.org/jira/browse/KAFKA-1549
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.8.2
> Environment: trunk
>Reporter: nicu marasoiu
>Assignee: Jun Rao
> Attachments:
> KAFKA-1549__dead_brokers_coming_in_the_TopicMetadataResponse_.patch
>
>
> JunRao confirming my observation that brokers are only added to the
> metadataCache, never removed: "The way that we update liveBrokers in
> MetadataCache.updateCache() doesn't seem right. We only add newly received
> live brokers to the list. However, there could be existing brokers in that
> list that are now dead. Those dead brokers shouldn't be returned to the
> clients. We should probably just take the new live broker list and cache it."
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066784#comment-14066784 ] nicu marasoiu commented on KAFKA-1549: -- esepntially i will replace the list/map of brokers every update. this will allow for the structure to be immutable likely. > dead brokers coming in the TopicMetadataResponse > > > Key: KAFKA-1549 > URL: https://issues.apache.org/jira/browse/KAFKA-1549 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.2 > Environment: trunk >Reporter: nicu marasoiu >Assignee: nicu marasoiu > > JunRao confirming my observation that brokers are only added to the > metadataCache, never removed: "The way that we update liveBrokers in > MetadataCache.updateCache() doesn't seem right. We only add newly received > live brokers to the list. However, there could be existing brokers in that > list that are now dead. Those dead brokers shouldn't be returned to the > clients. We should probably just take the new live broker list and cache it." -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066548#comment-14066548 ] nicu marasoiu commented on KAFKA-1549: -- test if i receive mails on this jira task > dead brokers coming in the TopicMetadataResponse > > > Key: KAFKA-1549 > URL: https://issues.apache.org/jira/browse/KAFKA-1549 > Project: Kafka > Issue Type: Bug > Environment: trunk >Reporter: nicu marasoiu >Assignee: nicu marasoiu > > The way that we update liveBrokers in MetadataCache.updateCache() doesn't > seem right. We only add newly received live brokers to the list. However, > there could be existing brokers in that list that are now dead. Those dead > brokers shouldn't be returned to the clients. We should probably just take > the new live broker list and cache it. -- This message was sent by Atlassian JIRA (v6.2#6252)
