[jira] [Commented] (KAFKA-7206) Enable batching in FindCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839059#comment-16839059 ] Yishun Guan commented on KAFKA-7206: Hi [~sagarrao], feel free to take a look at this: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-347%3A++Enable+batching+in+FindCoordinatorRequest], turns out the backward compatibility is a little tricky here, so it is a little complex for a minor improvement. > Enable batching in FindCoordinator > -- > > Key: KAFKA-7206 > URL: https://issues.apache.org/jira/browse/KAFKA-7206 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Yishun Guan >Assignee: Yishun Guan >Priority: Critical > Labels: needs-discussion, needs-kip, newbie++ > > To quote [~guozhang] : > "The proposal is that, we extend FindCoordinatorRequest to have multiple > consumer ids: today each FindCoordinatorRequest only contains a single > consumer id, so in our scenario we need to send N request for N consumer > groups still. If we can request for coordinators in a single request, then > the workflow could be simplified to: > # send a single FindCoordinatorRequest to a broker asking for coordinators > of all consumer groups. > 1.a) note that the response may still succeed in finding some coordinators > while error on others, and we need to handle them on that granularity (see > below). > # and then for the collected coordinator, group them by coordinator id and > send one request per coordinator destination. > Note that this change would require the version to be bumped up, to > FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE > version should be bumped up in order to include multiple coordinators." > A KIP is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7206) Enable batching in FindCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850051#comment-16850051 ] Yishun Guan commented on KAFKA-7206: [~sagarrao] yes, that's basically the reason. > Enable batching in FindCoordinator > -- > > Key: KAFKA-7206 > URL: https://issues.apache.org/jira/browse/KAFKA-7206 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Yishun Guan >Assignee: Yishun Guan >Priority: Critical > Labels: needs-discussion, needs-kip, newbie++ > > To quote [~guozhang] : > "The proposal is that, we extend FindCoordinatorRequest to have multiple > consumer ids: today each FindCoordinatorRequest only contains a single > consumer id, so in our scenario we need to send N request for N consumer > groups still. If we can request for coordinators in a single request, then > the workflow could be simplified to: > # send a single FindCoordinatorRequest to a broker asking for coordinators > of all consumer groups. > 1.a) note that the response may still succeed in finding some coordinators > while error on others, and we need to handle them on that granularity (see > below). > # and then for the collected coordinator, group them by coordinator id and > send one request per coordinator destination. > Note that this change would require the version to be bumped up, to > FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE > version should be bumped up in order to include multiple coordinators." > A KIP is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7699) Improve wall-clock time punctuations
[ https://issues.apache.org/jira/browse/KAFKA-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736364#comment-16736364 ] Yishun Guan edited comment on KAFKA-7699 at 1/7/19 9:16 PM: Hi, just curious (I am not an expert in this), why won't _ScheduledExecutorService_ works in this case? thanks! was (Author: shung): Hi, just curious (I am not an expert in this), why won't _ScheduledExecutorService_ works __ in this case? thanks! > Improve wall-clock time punctuations > > > Key: KAFKA-7699 > URL: https://issues.apache.org/jira/browse/KAFKA-7699 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Matthias J. Sax >Priority: Major > Labels: needs-kip > > Currently, wall-clock time punctuation allow to schedule periodic call backs > based on wall-clock time progress. The punctuation time starts, when the > punctuation is scheduled, thus, it's non-deterministic what is desired for > many use cases (I want a call-back in 5 minutes from "now"). > It would be a nice improvement, to allow users to "anchor" wall-clock > punctation, too, similar to a cron job: Thus, a punctuation would be > triggered at "fixed" times like the beginning of the next hour, independent > when the punctuation was registered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7699) Improve wall-clock time punctuations
[ https://issues.apache.org/jira/browse/KAFKA-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736364#comment-16736364 ] Yishun Guan commented on KAFKA-7699: Hi, just curious (I am not an expert in this), why won't _ScheduledExecutorService_ works __ in this case? thanks! > Improve wall-clock time punctuations > > > Key: KAFKA-7699 > URL: https://issues.apache.org/jira/browse/KAFKA-7699 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Matthias J. Sax >Priority: Major > Labels: needs-kip > > Currently, wall-clock time punctuation allow to schedule periodic call backs > based on wall-clock time progress. The punctuation time starts, when the > punctuation is scheduled, thus, it's non-deterministic what is desired for > many use cases (I want a call-back in 5 minutes from "now"). > It would be a nice improvement, to allow users to "anchor" wall-clock > punctation, too, similar to a cron job: Thus, a punctuation would be > triggered at "fixed" times like the beginning of the next hour, independent > when the punctuation was registered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6460) Add mocks for state stores used in Streams unit testing
[
https://issues.apache.org/jira/browse/KAFKA-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736402#comment-16736402
]
Yishun Guan commented on KAFKA-6460:
Hi [~guozhang] , I am still interested if nobody else is looking into this.
Will take some time since I am not familiar with the test packages, but I can
definitely do some researches and take a stab at it.
> Add mocks for state stores used in Streams unit testing
> ---
>
> Key: KAFKA-6460
> URL: https://issues.apache.org/jira/browse/KAFKA-6460
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Guozhang Wang
>Priority: Major
> Labels: newbie++
>
> We'd like to use mocks for different types of state stores: kv, window,
> session that can be used to record the number of expected put / get calls
> used in the DSL operator unit testing. This involves implementing the two
> interfaces {{StoreSupplier}} and {{StoreBuilder}} that can return a object
> created from, say, EasyMock, and the object can then be set up with the
> expected calls.
> In addition, we should also add a mock record collector which can be returned
> from the mock processor context so that with logging enabled store, users can
> also validate if the changes have been forwarded to the changelog as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7699) Improve wall-clock time punctuations
[ https://issues.apache.org/jira/browse/KAFKA-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744402#comment-16744402 ] Yishun Guan commented on KAFKA-7699: [~mjsax] Oh I see now I have more context. Thanks! > Improve wall-clock time punctuations > > > Key: KAFKA-7699 > URL: https://issues.apache.org/jira/browse/KAFKA-7699 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Matthias J. Sax >Priority: Major > Labels: needs-kip > > Currently, wall-clock time punctuation allow to schedule periodic call backs > based on wall-clock time progress. The punctuation time starts, when the > punctuation is scheduled, thus, it's non-deterministic what is desired for > many use cases (I want a call-back in 5 minutes from "now"). > It would be a nice improvement, to allow users to "anchor" wall-clock > punctation, too, similar to a cron job: Thus, a punctuation would be > triggered at "fixed" times like the beginning of the next hour, independent > when the punctuation was registered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-6460) Add mocks for state stores used in Streams unit testing
[
https://issues.apache.org/jira/browse/KAFKA-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yishun Guan reassigned KAFKA-6460:
--
Assignee: Yishun Guan
> Add mocks for state stores used in Streams unit testing
> ---
>
> Key: KAFKA-6460
> URL: https://issues.apache.org/jira/browse/KAFKA-6460
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Guozhang Wang
>Assignee: Yishun Guan
>Priority: Major
> Labels: newbie++
>
> We'd like to use mocks for different types of state stores: kv, window,
> session that can be used to record the number of expected put / get calls
> used in the DSL operator unit testing. This involves implementing the two
> interfaces {{StoreSupplier}} and {{StoreBuilder}} that can return a object
> created from, say, EasyMock, and the object can then be set up with the
> expected calls.
> In addition, we should also add a mock record collector which can be returned
> from the mock processor context so that with logging enabled store, users can
> also validate if the changes have been forwarded to the changelog as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6460) Add mocks for state stores used in Streams unit testing
[
https://issues.apache.org/jira/browse/KAFKA-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757803#comment-16757803
]
Yishun Guan commented on KAFKA-6460:
Hi [~guozhang] , I still not quite understand - so we should implement
_StoreTestDriver_ class that will be providing _MockStoreSupplier_ and
_MockStoreBuilder_ classes_?_ Or StoreSupplier and StoreBuilder doesn't have to
be mock, but this _StoreTestDriver_ will contain these two as variables and
wrap around those __ and provide auxiliary methods (like createKeyValueStore or
getSizeOfStore) for testing purpose_?_ Is a KIP needed? thanks
> Add mocks for state stores used in Streams unit testing
> ---
>
> Key: KAFKA-6460
> URL: https://issues.apache.org/jira/browse/KAFKA-6460
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Guozhang Wang
>Assignee: Yishun Guan
>Priority: Major
> Labels: newbie++
>
> We'd like to use mocks for different types of state stores: kv, window,
> session that can be used to record the number of expected put / get calls
> used in the DSL operator unit testing. This involves implementing the two
> interfaces {{StoreSupplier}} and {{StoreBuilder}} that can return a object
> created from, say, EasyMock, and the object can then be set up with the
> expected calls.
> In addition, we should also add a mock record collector which can be returned
> from the mock processor context so that with logging enabled store, users can
> also validate if the changes have been forwarded to the changelog as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6460) Add mocks for state stores used in Streams unit testing
[
https://issues.apache.org/jira/browse/KAFKA-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757803#comment-16757803
]
Yishun Guan edited comment on KAFKA-6460 at 1/31/19 11:32 PM:
--
Hi [~guozhang] , I still not quite understand - so we should implement
_StoreTestDriver_ class that will be providing _MockStoreSupplier_ and
_MockStoreBuilder_ classes? _Or StoreSupplier and StoreBuilder doesn't have to
be mock, but this StoreTestDriver_ will contain these two as variables and wrap
around those _and provide auxiliary methods (like createKeyValueStore or
getSizeOfStore) for testing purpose_? Is a KIP needed? thanks
was (Author: shung):
Hi [~guozhang] , I still not quite understand - so we should implement
_StoreTestDriver_ class that will be providing _MockStoreSupplier_ and
_MockStoreBuilder_ classes_?_ Or StoreSupplier and StoreBuilder doesn't have to
be mock, but this _StoreTestDriver_ will contain these two as variables and
wrap around those __ and provide auxiliary methods (like createKeyValueStore or
getSizeOfStore) for testing purpose_?_ Is a KIP needed? thanks
> Add mocks for state stores used in Streams unit testing
> ---
>
> Key: KAFKA-6460
> URL: https://issues.apache.org/jira/browse/KAFKA-6460
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Guozhang Wang
>Assignee: Yishun Guan
>Priority: Major
> Labels: newbie++
>
> We'd like to use mocks for different types of state stores: kv, window,
> session that can be used to record the number of expected put / get calls
> used in the DSL operator unit testing. This involves implementing the two
> interfaces {{StoreSupplier}} and {{StoreBuilder}} that can return a object
> created from, say, EasyMock, and the object can then be set up with the
> expected calls.
> In addition, we should also add a mock record collector which can be returned
> from the mock processor context so that with logging enabled store, users can
> also validate if the changes have been forwarded to the changelog as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6460) Add mocks for state stores used in Streams unit testing
[
https://issues.apache.org/jira/browse/KAFKA-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769862#comment-16769862
]
Yishun Guan commented on KAFKA-6460:
Sounds good to me [~mjsax] and [~guozhang], the design scope and all the use
cases are a little bit out of reach to me already. So if I am continuing with
this story, a KIP should be the way to go, so I can gather more opinions and
suggestions.
>From what I am seeing, what we want is to implement mock stores. So we will be
>implementing a `MockStoreFactory` that generates 'MockxxxStoreBuilder' that
>generates 'MockxxxStoreSupplier', so we are implementing all these layers?
>Which ones should be EasyMock? 'MockxxxStoreSupplier' should be tracking it's
>function calls, what kind of detail levels do we want? So it be similar to
>KeyValueStoreTestDriver's structure?
These will be mainly used with the `TopologyTestDriver` (the other usage is to
replace the vanilla store test drivers like KeyValueStoreTestDriver), so use
the following as an example:
{code:java}
@Test
public void shouldDriveGlobalStore() {
final String storeName = "my-store";
final String global = "global";
final String topic = "topic";
topology.addGlobalStore(Stores.keyValueStoreBuilder(Stores.inMemoryKeyValueStore(storeName),
Serdes.String(), Serdes.String()).withLoggingDisabled(),
global, STRING_DESERIALIZER, STRING_DESERIALIZER, topic, "processor",
define(new StatefulProcessor(storeName)));
driver = new TopologyTestDriver(topology, props);
final KeyValueStore globalStore =
driver.getKeyValueStore(storeName);
driver.pipeInput(recordFactory.create(topic, "key1", "value1"));
driver.pipeInput(recordFactory.create(topic, "key2", "value2"));
assertEquals("value1", globalStore.get("key1"));
assertEquals("value2", globalStore.get("key2"));
}{code}
>From what Guozhang is saying, even though `Topology` takes an
>'inMemoryKeyValueStore', we need to have something under the hood of
>`TopologyTestDriver`, that will iterate through the stores in 'Topology' and
>replace them with a 'MockStoreFactory.mockedKeyValueStore(...', is that true?
I agree, more could be discussed through a KIP, I will prepare one soon.
> Add mocks for state stores used in Streams unit testing
> ---
>
> Key: KAFKA-6460
> URL: https://issues.apache.org/jira/browse/KAFKA-6460
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Guozhang Wang
>Assignee: Yishun Guan
>Priority: Major
> Labels: newbie++
>
> We'd like to use mocks for different types of state stores: kv, window,
> session that can be used to record the number of expected put / get calls
> used in the DSL operator unit testing. This involves implementing the two
> interfaces {{StoreSupplier}} and {{StoreBuilder}} that can return a object
> created from, say, EasyMock, and the object can then be set up with the
> expected calls.
> In addition, we should also add a mock record collector which can be returned
> from the mock processor context so that with logging enabled store, users can
> also validate if the changes have been forwarded to the changelog as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528306#comment-16528306
]
Yishun Guan commented on KAFKA-6755:
[~rhauch] [~jingc]Hi, is anyone working on this story? If no, I can pick this
up. Thanks.
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yishun Guan reassigned KAFKA-6755:
--
Assignee: Yishun Guan
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Yishun Guan
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yishun Guan reassigned KAFKA-6755:
--
Assignee: (was: Yishun Guan)
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6788) Grouping consumer requests per consumer coordinator in admin client
[ https://issues.apache.org/jira/browse/KAFKA-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530469#comment-16530469 ] Yishun Guan commented on KAFKA-6788: Does it have a retry mechanism now? I am looking at KAKFA-6789 (https://issues.apache.org/jira/browse/KAFKA-6789) where it talks about a possible implementation of a retry mechanism. > Grouping consumer requests per consumer coordinator in admin client > --- > > Key: KAFKA-6788 > URL: https://issues.apache.org/jira/browse/KAFKA-6788 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Guozhang Wang >Assignee: Yishun Guan >Priority: Major > Labels: newbie++ > > In KafkaAdminClient, for some requests like describeGroup and deleteGroup, we > will first try to get the coordinator for each requested group id, and then > send the corresponding request for that group id. However, different group > ids could be hosted on the same coordinator, and these requests do support > multi group ids be sent within the same request. So we can consider optimize > it by grouping the requests per coordinator destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6788) Grouping consumer requests per consumer coordinator in admin client
[ https://issues.apache.org/jira/browse/KAFKA-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16535308#comment-16535308 ] Yishun Guan commented on KAFKA-6788: Hi, I will work on this issue from scratch then? > Grouping consumer requests per consumer coordinator in admin client > --- > > Key: KAFKA-6788 > URL: https://issues.apache.org/jira/browse/KAFKA-6788 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Guozhang Wang >Assignee: Yishun Guan >Priority: Major > Labels: newbie++ > > In KafkaAdminClient, for some requests like describeGroup and deleteGroup, we > will first try to get the coordinator for each requested group id, and then > send the corresponding request for that group id. However, different group > ids could be hosted on the same coordinator, and these requests do support > multi group ids be sent within the same request. So we can consider optimize > it by grouping the requests per coordinator destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7146) Grouping consumer requests per consumer coordinator in admin client in describeConsumerGroups
Yishun Guan created KAFKA-7146: -- Summary: Grouping consumer requests per consumer coordinator in admin client in describeConsumerGroups Key: KAFKA-7146 URL: https://issues.apache.org/jira/browse/KAFKA-7146 Project: Kafka Issue Type: Sub-task Reporter: Yishun Guan Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7146) Grouping consumer requests per consumer coordinator in admin client in describeConsumerGroups
[ https://issues.apache.org/jira/browse/KAFKA-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan updated KAFKA-7146: --- Description: Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups(). According to KAFKA-6788: " In KafkaAdminClient, for some requests like describeGroup and deleteGroup, we will first try to get the coordinator for each requested group id, and then send the corresponding request for that group id. However, different group ids could be hosted on the same coordinator, and these requests do support multi group ids be sent within the same request. So we can consider optimize it by grouping the requests per coordinator destination. " was:Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups(). > Grouping consumer requests per consumer coordinator in admin client in > describeConsumerGroups > - > > Key: KAFKA-7146 > URL: https://issues.apache.org/jira/browse/KAFKA-7146 > Project: Kafka > Issue Type: Sub-task >Reporter: Yishun Guan >Priority: Critical > > Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups(). > > According to KAFKA-6788: > > " > In KafkaAdminClient, for some requests like describeGroup and deleteGroup, we > will first try to get the coordinator for each requested group id, and then > send the corresponding request for that group id. However, different group > ids could be hosted on the same coordinator, and these requests do support > multi group ids be sent within the same request. So we can consider optimize > it by grouping the requests per coordinator destination. > " -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7146) Grouping consumer requests per consumer coordinator in admin client in describeConsumerGroups
[ https://issues.apache.org/jira/browse/KAFKA-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan reassigned KAFKA-7146: -- Assignee: Yishun Guan > Grouping consumer requests per consumer coordinator in admin client in > describeConsumerGroups > - > > Key: KAFKA-7146 > URL: https://issues.apache.org/jira/browse/KAFKA-7146 > Project: Kafka > Issue Type: Sub-task >Reporter: Yishun Guan >Assignee: Yishun Guan >Priority: Critical > > Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups(). > > According to KAFKA-6788: > > "In KafkaAdminClient, for some requests like describeGroup and deleteGroup, > we will first try to get the coordinator for each requested group id, and > then send the corresponding request for that group id. However, different > group ids could be hosted on the same coordinator, and these requests do > support multi group ids be sent within the same request. So we can consider > optimize it by grouping the requests per coordinator destination." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7146) Grouping consumer requests per consumer coordinator in admin client in describeConsumerGroups
[ https://issues.apache.org/jira/browse/KAFKA-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan updated KAFKA-7146: --- Description: Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups(). According to KAFKA-6788: "In KafkaAdminClient, for some requests like describeGroup and deleteGroup, we will first try to get the coordinator for each requested group id, and then send the corresponding request for that group id. However, different group ids could be hosted on the same coordinator, and these requests do support multi group ids be sent within the same request. So we can consider optimize it by grouping the requests per coordinator destination." was: Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups(). According to KAFKA-6788: " In KafkaAdminClient, for some requests like describeGroup and deleteGroup, we will first try to get the coordinator for each requested group id, and then send the corresponding request for that group id. However, different group ids could be hosted on the same coordinator, and these requests do support multi group ids be sent within the same request. So we can consider optimize it by grouping the requests per coordinator destination. " > Grouping consumer requests per consumer coordinator in admin client in > describeConsumerGroups > - > > Key: KAFKA-7146 > URL: https://issues.apache.org/jira/browse/KAFKA-7146 > Project: Kafka > Issue Type: Sub-task >Reporter: Yishun Guan >Priority: Critical > > Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups(). > > According to KAFKA-6788: > > "In KafkaAdminClient, for some requests like describeGroup and deleteGroup, > we will first try to get the coordinator for each requested group id, and > then send the corresponding request for that group id. However, different > group ids could be hosted on the same coordinator, and these requests do > support multi group ids be sent within the same request. So we can consider > optimize it by grouping the requests per coordinator destination." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-5447) AdminClient metrics
[ https://issues.apache.org/jira/browse/KAFKA-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan reassigned KAFKA-5447: -- Assignee: Yishun Guan > AdminClient metrics > --- > > Key: KAFKA-5447 > URL: https://issues.apache.org/jira/browse/KAFKA-5447 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Yishun Guan >Priority: Major > Labels: needs-kip > > We should consider if we should expose AdminClient-specific metrics. It > currently only exposes metrics from the underlying Selector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-5447) AdminClient metrics
[ https://issues.apache.org/jira/browse/KAFKA-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan resolved KAFKA-5447. Resolution: Fixed https://cwiki.apache.org/confluence/display/KAFKA/KIP-324%3A+Add+method+to+get+metrics%28%29+in+AdminClient > AdminClient metrics > --- > > Key: KAFKA-5447 > URL: https://issues.apache.org/jira/browse/KAFKA-5447 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Yishun Guan >Priority: Major > Labels: needs-kip > > We should consider if we should expose AdminClient-specific metrics. It > currently only exposes metrics from the underlying Selector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5447) AdminClient metrics
[ https://issues.apache.org/jira/browse/KAFKA-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544308#comment-16544308 ] Yishun Guan commented on KAFKA-5447: This is resolved with https://cwiki.apache.org/confluence/display/KAFKA/KIP-324%3A+Add+method+to+get+metrics%28%29+in+AdminClient > AdminClient metrics > --- > > Key: KAFKA-5447 > URL: https://issues.apache.org/jira/browse/KAFKA-5447 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Yishun Guan >Priority: Major > Labels: needs-kip > > We should consider if we should expose AdminClient-specific metrics. It > currently only exposes metrics from the underlying Selector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7206) Enable batching in FindCoordinator
Yishun Guan created KAFKA-7206:
--
Summary: Enable batching in FindCoordinator
Key: KAFKA-7206
URL: https://issues.apache.org/jira/browse/KAFKA-7206
Project: Kafka
Issue Type: Improvement
Components: admin
Reporter: Yishun Guan
To quote [~guozhang]
"The proposal is that, we extend FindCoordinatorRequest to have multiple
consumer ids: today each FindCoordinatorRequest only contains a single consumer
id, so in our scenario we need to send N request for N consumer groups still.
If we can request for coordinators in a single request, then the workflow could
be simplified to:
# send a single FindCoordinatorRequest to a broker asking for coordinators of
all consumer groups.
1.a) note that the response may still succeed in finding some coordinators
while error on others, and we need to handle them on that granularity (see
below).
# and then for the collected coordinator, group them by coordinator id and
send one request per coordinator destination.
Note that this change would require the version to be bumped up, to
{{FIND_COORDINATOR_REQUEST_V3}}for such protocol changes, also the RESPONSE
version should be bumped up in order to include multiple coordinators."
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7206) Enable batching in FindCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yishun Guan updated KAFKA-7206:
---
Description:
To quote [~guozhang] :
"The proposal is that, we extend FindCoordinatorRequest to have multiple
consumer ids: today each FindCoordinatorRequest only contains a single consumer
id, so in our scenario we need to send N request for N consumer groups still.
If we can request for coordinators in a single request, then the workflow could
be simplified to:
# send a single FindCoordinatorRequest to a broker asking for coordinators of
all consumer groups.
1.a) note that the response may still succeed in finding some coordinators
while error on others, and we need to handle them on that granularity (see
below).
# and then for the collected coordinator, group them by coordinator id and
send one request per coordinator destination.
Note that this change would require the version to be bumped up, to
\{{FIND_COORDINATOR_REQUEST_V3}}for such protocol changes, also the RESPONSE
version should be bumped up in order to include multiple coordinators."
was:
To quote [~guozhang]
"The proposal is that, we extend FindCoordinatorRequest to have multiple
consumer ids: today each FindCoordinatorRequest only contains a single consumer
id, so in our scenario we need to send N request for N consumer groups still.
If we can request for coordinators in a single request, then the workflow could
be simplified to:
# send a single FindCoordinatorRequest to a broker asking for coordinators of
all consumer groups.
1.a) note that the response may still succeed in finding some coordinators
while error on others, and we need to handle them on that granularity (see
below).
# and then for the collected coordinator, group them by coordinator id and
send one request per coordinator destination.
Note that this change would require the version to be bumped up, to
{{FIND_COORDINATOR_REQUEST_V3}}for such protocol changes, also the RESPONSE
version should be bumped up in order to include multiple coordinators."
> Enable batching in FindCoordinator
> --
>
> Key: KAFKA-7206
> URL: https://issues.apache.org/jira/browse/KAFKA-7206
> Project: Kafka
> Issue Type: Improvement
> Components: admin
>Reporter: Yishun Guan
>Priority: Critical
> Labels: newbie++
>
> To quote [~guozhang] :
> "The proposal is that, we extend FindCoordinatorRequest to have multiple
> consumer ids: today each FindCoordinatorRequest only contains a single
> consumer id, so in our scenario we need to send N request for N consumer
> groups still. If we can request for coordinators in a single request, then
> the workflow could be simplified to:
> # send a single FindCoordinatorRequest to a broker asking for coordinators
> of all consumer groups.
> 1.a) note that the response may still succeed in finding some coordinators
> while error on others, and we need to handle them on that granularity (see
> below).
> # and then for the collected coordinator, group them by coordinator id and
> send one request per coordinator destination.
> Note that this change would require the version to be bumped up, to
> \{{FIND_COORDINATOR_REQUEST_V3}}for such protocol changes, also the RESPONSE
> version should be bumped up in order to include multiple coordinators."
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7206) Enable batching in FindCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yishun Guan updated KAFKA-7206:
---
Description:
To quote [~guozhang] :
"The proposal is that, we extend FindCoordinatorRequest to have multiple
consumer ids: today each FindCoordinatorRequest only contains a single consumer
id, so in our scenario we need to send N request for N consumer groups still.
If we can request for coordinators in a single request, then the workflow could
be simplified to:
# send a single FindCoordinatorRequest to a broker asking for coordinators of
all consumer groups.
1.a) note that the response may still succeed in finding some coordinators
while error on others, and we need to handle them on that granularity (see
below).
# and then for the collected coordinator, group them by coordinator id and
send one request per coordinator destination.
Note that this change would require the version to be bumped up, to
FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE
version should be bumped up in order to include multiple coordinators."
was:
To quote [~guozhang] :
"The proposal is that, we extend FindCoordinatorRequest to have multiple
consumer ids: today each FindCoordinatorRequest only contains a single consumer
id, so in our scenario we need to send N request for N consumer groups still.
If we can request for coordinators in a single request, then the workflow could
be simplified to:
# send a single FindCoordinatorRequest to a broker asking for coordinators of
all consumer groups.
1.a) note that the response may still succeed in finding some coordinators
while error on others, and we need to handle them on that granularity (see
below).
# and then for the collected coordinator, group them by coordinator id and
send one request per coordinator destination.
Note that this change would require the version to be bumped up, to
\{{FIND_COORDINATOR_REQUEST_V3}}for such protocol changes, also the RESPONSE
version should be bumped up in order to include multiple coordinators."
> Enable batching in FindCoordinator
> --
>
> Key: KAFKA-7206
> URL: https://issues.apache.org/jira/browse/KAFKA-7206
> Project: Kafka
> Issue Type: Improvement
> Components: admin
>Reporter: Yishun Guan
>Priority: Critical
> Labels: newbie++
>
> To quote [~guozhang] :
> "The proposal is that, we extend FindCoordinatorRequest to have multiple
> consumer ids: today each FindCoordinatorRequest only contains a single
> consumer id, so in our scenario we need to send N request for N consumer
> groups still. If we can request for coordinators in a single request, then
> the workflow could be simplified to:
> # send a single FindCoordinatorRequest to a broker asking for coordinators
> of all consumer groups.
> 1.a) note that the response may still succeed in finding some coordinators
> while error on others, and we need to handle them on that granularity (see
> below).
> # and then for the collected coordinator, group them by coordinator id and
> send one request per coordinator destination.
> Note that this change would require the version to be bumped up, to
> FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE
> version should be bumped up in order to include multiple coordinators."
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7206) Enable batching in FindCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yishun Guan updated KAFKA-7206:
---
Labels: newbie++ (was: )
> Enable batching in FindCoordinator
> --
>
> Key: KAFKA-7206
> URL: https://issues.apache.org/jira/browse/KAFKA-7206
> Project: Kafka
> Issue Type: Improvement
> Components: admin
>Reporter: Yishun Guan
>Priority: Critical
> Labels: newbie++
>
> To quote [~guozhang]
> "The proposal is that, we extend FindCoordinatorRequest to have multiple
> consumer ids: today each FindCoordinatorRequest only contains a single
> consumer id, so in our scenario we need to send N request for N consumer
> groups still. If we can request for coordinators in a single request, then
> the workflow could be simplified to:
> # send a single FindCoordinatorRequest to a broker asking for coordinators
> of all consumer groups.
> 1.a) note that the response may still succeed in finding some coordinators
> while error on others, and we need to handle them on that granularity (see
> below).
> # and then for the collected coordinator, group them by coordinator id and
> send one request per coordinator destination.
> Note that this change would require the version to be bumped up, to
> {{FIND_COORDINATOR_REQUEST_V3}}for such protocol changes, also the RESPONSE
> version should be bumped up in order to include multiple coordinators."
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7206) Enable batching in FindCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan updated KAFKA-7206: --- Labels: needs-discussion needs-kip newbie++ (was: newbie++) > Enable batching in FindCoordinator > -- > > Key: KAFKA-7206 > URL: https://issues.apache.org/jira/browse/KAFKA-7206 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Yishun Guan >Priority: Critical > Labels: needs-discussion, needs-kip, newbie++ > > To quote [~guozhang] : > "The proposal is that, we extend FindCoordinatorRequest to have multiple > consumer ids: today each FindCoordinatorRequest only contains a single > consumer id, so in our scenario we need to send N request for N consumer > groups still. If we can request for coordinators in a single request, then > the workflow could be simplified to: > # send a single FindCoordinatorRequest to a broker asking for coordinators > of all consumer groups. > 1.a) note that the response may still succeed in finding some coordinators > while error on others, and we need to handle them on that granularity (see > below). > # and then for the collected coordinator, group them by coordinator id and > send one request per coordinator destination. > Note that this change would require the version to be bumped up, to > FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE > version should be bumped up in order to include multiple coordinators." > A KIP is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7206) Enable batching in FindCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan updated KAFKA-7206: --- Description: To quote [~guozhang] : "The proposal is that, we extend FindCoordinatorRequest to have multiple consumer ids: today each FindCoordinatorRequest only contains a single consumer id, so in our scenario we need to send N request for N consumer groups still. If we can request for coordinators in a single request, then the workflow could be simplified to: # send a single FindCoordinatorRequest to a broker asking for coordinators of all consumer groups. 1.a) note that the response may still succeed in finding some coordinators while error on others, and we need to handle them on that granularity (see below). # and then for the collected coordinator, group them by coordinator id and send one request per coordinator destination. Note that this change would require the version to be bumped up, to FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE version should be bumped up in order to include multiple coordinators." A KIP is needed. was: To quote [~guozhang] : "The proposal is that, we extend FindCoordinatorRequest to have multiple consumer ids: today each FindCoordinatorRequest only contains a single consumer id, so in our scenario we need to send N request for N consumer groups still. If we can request for coordinators in a single request, then the workflow could be simplified to: # send a single FindCoordinatorRequest to a broker asking for coordinators of all consumer groups. 1.a) note that the response may still succeed in finding some coordinators while error on others, and we need to handle them on that granularity (see below). # and then for the collected coordinator, group them by coordinator id and send one request per coordinator destination. Note that this change would require the version to be bumped up, to FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE version should be bumped up in order to include multiple coordinators." > Enable batching in FindCoordinator > -- > > Key: KAFKA-7206 > URL: https://issues.apache.org/jira/browse/KAFKA-7206 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Yishun Guan >Priority: Critical > Labels: needs-discussion, needs-kip, newbie++ > > To quote [~guozhang] : > "The proposal is that, we extend FindCoordinatorRequest to have multiple > consumer ids: today each FindCoordinatorRequest only contains a single > consumer id, so in our scenario we need to send N request for N consumer > groups still. If we can request for coordinators in a single request, then > the workflow could be simplified to: > # send a single FindCoordinatorRequest to a broker asking for coordinators > of all consumer groups. > 1.a) note that the response may still succeed in finding some coordinators > while error on others, and we need to handle them on that granularity (see > below). > # and then for the collected coordinator, group them by coordinator id and > send one request per coordinator destination. > Note that this change would require the version to be bumped up, to > FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE > version should be bumped up in order to include multiple coordinators." > A KIP is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7206) Enable batching in FindCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan reassigned KAFKA-7206: -- Assignee: Yishun Guan > Enable batching in FindCoordinator > -- > > Key: KAFKA-7206 > URL: https://issues.apache.org/jira/browse/KAFKA-7206 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Yishun Guan >Assignee: Yishun Guan >Priority: Critical > Labels: needs-discussion, needs-kip, newbie++ > > To quote [~guozhang] : > "The proposal is that, we extend FindCoordinatorRequest to have multiple > consumer ids: today each FindCoordinatorRequest only contains a single > consumer id, so in our scenario we need to send N request for N consumer > groups still. If we can request for coordinators in a single request, then > the workflow could be simplified to: > # send a single FindCoordinatorRequest to a broker asking for coordinators > of all consumer groups. > 1.a) note that the response may still succeed in finding some coordinators > while error on others, and we need to handle them on that granularity (see > below). > # and then for the collected coordinator, group them by coordinator id and > send one request per coordinator destination. > Note that this change would require the version to be bumped up, to > FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE > version should be bumped up in order to include multiple coordinators." > A KIP is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7402) Kafka Streams should implement AutoCloseable where appropriate
[ https://issues.apache.org/jira/browse/KAFKA-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619541#comment-16619541 ] Yishun Guan commented on KAFKA-7402: Hi [~vvcephei], should we create an KIP for KafkaStream first or we should create a universal one for all the applicable classes? > Kafka Streams should implement AutoCloseable where appropriate > -- > > Key: KAFKA-7402 > URL: https://issues.apache.org/jira/browse/KAFKA-7402 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Priority: Minor > Labels: needs-kip, newbie > > Various components in Streams have close methods but do not implement > AutoCloseable. This means that they can't be used in try-with-resources > blocks. > Remedying that would simplify our tests and make life easier for users as > well. > KafkaStreams itself is a notable example of this, but we can take the > opportunity to look for other components that make sense as AutoCloseable as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7402) Kafka Streams should implement AutoCloseable where appropriate
[ https://issues.apache.org/jira/browse/KAFKA-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishun Guan reassigned KAFKA-7402: -- Assignee: Yishun Guan > Kafka Streams should implement AutoCloseable where appropriate > -- > > Key: KAFKA-7402 > URL: https://issues.apache.org/jira/browse/KAFKA-7402 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Assignee: Yishun Guan >Priority: Minor > Labels: needs-kip, newbie > > Various components in Streams have close methods but do not implement > AutoCloseable. This means that they can't be used in try-with-resources > blocks. > Remedying that would simplify our tests and make life easier for users as > well. > KafkaStreams itself is a notable example of this, but we can take the > opportunity to look for other components that make sense as AutoCloseable as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
