[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14317786#comment-14317786 ] Hongliang Su commented on KAFKA-1760: - [~jkreps] do you have any docs about how to use the new consumer client? Thank you. > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, > KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch, > KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[
https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14317711#comment-14317711
]
Hongliang Su commented on KAFKA-1760:
-
Does the new consumer client work ?
I have error as follows, and I'm curious about the consumer which connects
directly to the kafka server, not using the zookeeper clusters?
logs:
[2015-02-12 14:25:30,664] INFO ConsumerConfig values:
rebalance.callback.class = class
org.apache.kafka.clients.consumer.internals.NoOpConsumerRebalanceCallback
check.crcs = true
retry.backoff.ms = 100
metrics.sample.window.ms = 3
metadata.max.age.ms = 30
receive.buffer.bytes = 32768
auto.commit.interval.ms = 5000
session.timeout.ms = 3
group.id = henrytest
enable.auto.commit = true
metric.reporters = []
bootstrap.servers = [10.53.132.52:9092, 10.53.132.55:9092]
client.id =
fetch.max.wait.ms = 500
fetch.min.bytes = 1024
send.buffer.bytes = 131072
reconnect.backoff.ms = 50
key.deserializer = class
org.apache.kafka.common.serialization.StringDeserializer
auto.offset.reset = latest
metrics.num.samples = 2
value.deserializer = class
org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
partition.assignment.strategy = blah
(org.apache.kafka.clients.consumer.ConsumerConfig:113)
[2015-02-12 14:25:30,665] DEBUG Starting the Kafka consumer
(org.apache.kafka.clients.consumer.KafkaConsumer:512)
[2015-02-12 14:25:39,681] DEBUG Updated cluster metadata version 1 to
Cluster(nodes = [Node(-2, 10.53.132.55, 9092), Node(-1, 10.53.132.52, 9092)],
partitions = []) (org.apache.kafka.clients.Metadata:147)
[2015-02-12 14:25:39,729] DEBUG Kafka consumer created
(org.apache.kafka.clients.consumer.KafkaConsumer:581)
[2015-02-12 14:25:39,730] DEBUG Subscribed to topic(s):
(org.apache.kafka.clients.consumer.KafkaConsumer:612)
[2015-02-12 14:25:39,730] DEBUG Joining group henrytest
(org.apache.kafka.clients.consumer.KafkaConsumer:964)
[2015-02-12 14:25:39,735] DEBUG Issuing co-ordinator request: JOIN_GROUP:
{group_id=henrytest,session_timeout=3,topics=[javatest],consumer_id=,partition_assignment_strategy=blah}
(org.apache.kafka.clients.consumer.KafkaConsumer:1188)
[2015-02-12 14:25:39,735] DEBUG No consumer co-ordinator known, attempting to
discover one. (org.apache.kafka.clients.consumer.KafkaConsumer:1251)
[2015-02-12 14:25:39,736] DEBUG Initiating connection to node -1 at
10.53.132.52:9092. (org.apache.kafka.clients.NetworkClient:483)
[2015-02-12 14:25:39,738] DEBUG Completed connection to node -1
(org.apache.kafka.clients.NetworkClient:419)
[2015-02-12 14:25:39,838] DEBUG Sending metadata request
ClientRequest(expectResponse=true, callback=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=0,client_id=consumer-1},
body={topics=[javatest]})) to node -1
(org.apache.kafka.clients.NetworkClient:460)
[2015-02-12 14:25:39,839] DEBUG Initiating connection to node -2 at
10.53.132.55:9092. (org.apache.kafka.clients.NetworkClient:483)
[2015-02-12 14:25:39,841] DEBUG Completed connection to node -2
(org.apache.kafka.clients.NetworkClient:419)
[2015-02-12 14:25:39,849] DEBUG Found consumer co-ordinator: Node(2147483646,
10.53.132.52, 9092) (org.apache.kafka.clients.consumer.KafkaConsumer:1258)
[2015-02-12 14:25:39,849] DEBUG Initiating connection to node 2147483646 at
10.53.132.52:9092. (org.apache.kafka.clients.NetworkClient:483)
[2015-02-12 14:25:39,849] DEBUG No connection to co-ordinator, attempting to
connect. (org.apache.kafka.clients.consumer.KafkaConsumer:1236)
[2015-02-12 14:25:39,855] DEBUG Updated cluster metadata version 2 to
Cluster(nodes = [Node(0, 10.53.132.55, 9092), Node(1, 10.53.132.52, 9092)],
partitions = [Partition(topic = javatest, partition = 0, leader = 0, replicas =
[0,], isr = [0,]]) (org.apache.kafka.clients.Metadata:147)
[2015-02-12 14:25:39,855] DEBUG Completed connection to node 2147483646
(org.apache.kafka.clients.NetworkClient:419)
[2015-02-12 14:25:39,858] DEBUG Joined group:
{error_code=-1,group_generation_id=-1,consumer_id=,assigned_partitions=[]}
(org.apache.kafka.clients.consumer.KafkaConsumer:985)
Exception in thread "Thread-0"
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
> Implement new consumer client
> -
>
> Key: KAFKA-1760
> URL: https://issues.apache.org/jira/browse/KAFKA-1760
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Fix For: 0.8.3
>
> Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch,
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[
https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306358#comment-14306358
]
Guozhang Wang commented on KAFKA-1760:
--
[~jkreps], I think this is a bug, because later on when it tries to send any
coordinator requests it will use just the node id, not the host/port:
{code}
RequestSend send = new RequestSend(this.consumerCoordinator.id(), header,
request);
{code}
and like I mentioned before, when coordinator migrates, the updated coordinator
info ended up not getting to consumer's network client, and hence it still
tries to talk to the old guy.
> Implement new consumer client
> -
>
> Key: KAFKA-1760
> URL: https://issues.apache.org/jira/browse/KAFKA-1760
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Fix For: 0.8.3
>
> Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch,
> KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch,
> KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch,
> KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch
>
>
> Implement a consumer client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[
https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306332#comment-14306332
]
Jay Kreps commented on KAFKA-1760:
--
Actually shameful to say but I'm not sure this is a bug but rather a "feature".
We need to give the client it's own connection to the co-ordinator, but the
existing client maintains a single connection per node id. My hack for this is
to give the coordinator connection the node id -MAX_INT:
{code}
if (response.errorCode() == Errors.NONE.code())
return new Node(Integer.MIN_VALUE, response.node().host(),
response.node().port());
{code}
I would agree this is somewhat egregious, but I really am not sure about how we
would change the client to model connections and nodes differently for a
non-hack to work and still be usable.
> Implement new consumer client
> -
>
> Key: KAFKA-1760
> URL: https://issues.apache.org/jira/browse/KAFKA-1760
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Fix For: 0.8.3
>
> Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch,
> KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch,
> KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch,
> KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch
>
>
> Implement a consumer client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[
https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306324#comment-14306324
]
Guozhang Wang commented on KAFKA-1760:
--
OK, the first one observed so far is that the consumer metadata responses does
not actually help update the metadata due to some protocol bug:
In the below example, although the returned node id is 1, it is read as
-2147483648 as always (and Node.toString did not print the id since it is < 0),
and later responses with node id 2 will not change consumer's coordinator
destination.
{code}
[2015-02-04 12:43:46,251] TRACE [KafkaApi-1] Sending consumer metadata
ConsumerMetadataResponse(Some(id:1,host:localhost,port:46148),0,27) for
correlation id 27 to client consumer-1. (kafka.server.KafkaApis:36)
[2015-02-04 12:43:46,252] TRACE Processor id 1 selection time = 4558827 ns
(kafka.network.Processor:36)
[2015-02-04 12:43:46,252] TRACE Socket server received response to send,
registering for write:
Response(1,Request(1,sun.nio.ch.SelectionKeyImpl@4bd15081,null,1423082626245,/127.0.0.1:51454),kafka.network.BoundedByteBufferSend@40aaecb5,SendAction)
(kafka.network.Processor:36)
[2015-02-04 12:43:46,252] TRACE Processor id 1 selection time = 18275 ns
(kafka.network.Processor:36)
[2015-02-04 12:43:46,252] TRACE 29 bytes written to /127.0.0.1:51454 using
key sun.nio.ch.SelectionKeyImpl@4bd15081 (kafka.network.Processor:36)
[2015-02-04 12:43:46,253] TRACE Completed request:Name:
ConsumerMetadataRequest; Version: 0; CorrelationId: 27; ClientId: consumer-1;
Group: my-test from client
/127.0.0.1:51454;totalTime:7,requestQueueTime:2,localTime:5,remoteTime:0,responseQueueTime:0,sendTime:0
(kafka.request.logger:85)
[2015-02-04 12:43:46,253] TRACE Finished writing, registering for read on
connection /127.0.0.1:51454 (kafka.network.Processor:36)
[2015-02-04 12:43:46,253] DEBUG Found consumer co-ordinator:
Node(localhost, 46148) (org.apache.kafka.clients.consumer.KafkaConsumer:1259)
[2015-02-04 12:43:46,253] DEBUG Initiating connection to node -2147483648
at localhost:46148. (org.apache.kafka.clients.NetworkClient:484)
{code}
> Implement new consumer client
> -
>
> Key: KAFKA-1760
> URL: https://issues.apache.org/jira/browse/KAFKA-1760
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Fix For: 0.8.3
>
> Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch,
> KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch,
> KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch,
> KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch
>
>
> Implement a consumer client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299027#comment-14299027 ] Guozhang Wang commented on KAFKA-1760: -- Has committed to trunk, closing this ticket. > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, > KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch, > KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297847#comment-14297847 ] Guozhang Wang commented on KAFKA-1760: -- LGTM, let's commit to trunk now. > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, > KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch, > KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296710#comment-14296710 ] Jay Kreps commented on KAFKA-1760: -- Updated reviewboard https://reviews.apache.org/r/27799/diff/ against branch trunk > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, > KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch, > KAFKA-1760_2015-01-23_13:13:00.patch, KAFKA-1760_2015-01-29_03:20:20.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289987#comment-14289987 ] Jay Kreps commented on KAFKA-1760: -- Updated reviewboard https://reviews.apache.org/r/27799/diff/ against branch trunk > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, > KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch, > KAFKA-1760_2015-01-23_13:13:00.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288721#comment-14288721 ] Jay Kreps commented on KAFKA-1760: -- Updated reviewboard https://reviews.apache.org/r/27799/diff/ against branch trunk > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, > KAFKA-1760_2015-01-22_10:03:26.patch, KAFKA-1760_2015-01-22_20:21:56.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287869#comment-14287869 ] Jay Kreps commented on KAFKA-1760: -- Updated reviewboard https://reviews.apache.org/r/27799/diff/ against branch trunk > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch, > KAFKA-1760_2015-01-22_10:03:26.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285851#comment-14285851 ] Jay Kreps commented on KAFKA-1760: -- Updated reviewboard https://reviews.apache.org/r/27799/diff/ against branch trunk > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch, KAFKA-1760_2015-01-21_08:42:20.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282101#comment-14282101 ] Jay Kreps commented on KAFKA-1760: -- Posted an updated patch rebased against trunk. > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282100#comment-14282100 ] Jay Kreps commented on KAFKA-1760: -- Updated reviewboard https://reviews.apache.org/r/27799/diff/ against branch trunk > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch, > KAFKA-1760_2015-01-18_19:10:13.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275597#comment-14275597 ] Guozhang Wang commented on KAFKA-1760: -- Thanks Jay, I will take a look at the patch. > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8.3 > > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273128#comment-14273128 ] Jay Kreps commented on KAFKA-1760: -- I updated this patch and consider it more or less complete. I think we should review and check in as soon as 0.8.2 is out. I'd appreciate review. > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273125#comment-14273125 ] Jay Kreps commented on KAFKA-1760: -- Updated reviewboard https://reviews.apache.org/r/27799/diff/ against branch trunk > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Attachments: KAFKA-1760.patch, KAFKA-1760_2015-01-11_16:57:15.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204354#comment-14204354 ] Jay Kreps commented on KAFKA-1760: -- I uploaded a very rough draft for this. It implements basically - simple consumer for a specified topic/partition - join group to use the new co-ordinator (implemented a server-side stub for this) - commit offsets (async or sync) - seek (including seekToBeginning and seekToEnd) - auto offset reset Limitations - Not executing the pre and post rebalance callbacks - No real error handling - Need to failure detect a controller that doesn't answer heartbeats - Probably still pretty buggy - Integration tests only cover basic cases - Code is still very ugly Error cases that need to be implemented and tested: - Controller failure - Commit retries - Basically other cases where a specially timed failure can interrupt the non-fetch requests (e.g. fetch offsets, commit offsets, list offsets, etc) > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps >Assignee: Jay Kreps > Attachments: KAFKA-1760.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1760) Implement new consumer client
[ https://issues.apache.org/jira/browse/KAFKA-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204341#comment-14204341 ] Jay Kreps commented on KAFKA-1760: -- Created reviewboard https://reviews.apache.org/r/27799/diff/ against branch trunk > Implement new consumer client > - > > Key: KAFKA-1760 > URL: https://issues.apache.org/jira/browse/KAFKA-1760 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jay Kreps > Attachments: KAFKA-1760.patch > > > Implement a consumer client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
