[jira] [Commented] (KAFKA-5904) Create Connect metrics for worker rebalances

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190826#comment-16190826
 ] 

Randall Hauch commented on KAFKA-5904:
--

The changes for this issue were included in the PR for KAFKA-5903.

> Create Connect metrics for worker rebalances
> 
>
> Key: KAFKA-5904
> URL: https://issues.apache.org/jira/browse/KAFKA-5904
> Project: Kafka
>  Issue Type: Sub-task
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Blocker
> Fix For: 1.0.0
>
>
> See KAFKA-2376 for parent task and 
> [KIP-196|https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework]
>  for the details on the metrics. This subtask is to create the "Worker 
> Rebalance Metrics".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5903) Create Connect metrics for workers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190825#comment-16190825
 ] 

ASF GitHub Bot commented on KAFKA-5903:
---

GitHub user rhauch opened a pull request:

https://github.com/apache/kafka/pull/4011

KAFKA-5903: Added Connect metrics to the worker and distributed herder

Added metrics to the Connect worker and rebalancing metrics to the 
distributed herder.

This is built on top of #3987, and I can rebase this PR once that is merged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rhauch/kafka kafka-5903

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4011


commit d4afe3d2d0ca0c313e70266dca104caa0564d3f3
Author: Randall Hauch 
Date:   2017-09-28T19:12:21Z

KAFKA-5990 Enable generation of metrics docs for Connect

A new mechanism was added recently to the Metrics framework to make it 
easier to generate the documentation. It uses a registry with a 
MetricsNameTemplate for each metric, and then those templates are used when 
creating the actual metrics. The metrics framework provides utilities that can 
generate the HTML documentation from the registry of templates.

This change moves the recently-added Connect metrics over to use these 
templates and to then generate the metric documentation for Connect.

commit d7bab224800c4d56d754b7121abb209df9715ca1
Author: Randall Hauch 
Date:   2017-10-02T20:37:05Z

KAFKA-5903 Use string-valued metrics for connector name, type, version, and 
status

Changed from the indicator metrics for connector and task status that 
reported the various boolean states to a single string-valued metric for 
status. Also added string-valued metrics for connector name, version, and type.

commit 75ffb38f32f552dbf667553f998a9cde7af6465f
Author: Randall Hauch 
Date:   2017-10-03T01:05:44Z

KAFKA-5903 Added Connect worker metrics

commit 24efd5f29fe4b23da0c7f9949361b14fad8fbc4e
Author: Randall Hauch 
Date:   2017-10-04T05:15:19Z

KAFKA-5904 Added Connect rebalancing metrics to distributed herder

Added the rebalancing metrics to the distributed herder.




> Create Connect metrics for workers
> --
>
> Key: KAFKA-5903
> URL: https://issues.apache.org/jira/browse/KAFKA-5903
> Project: Kafka
>  Issue Type: Sub-task
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Blocker
> Fix For: 1.0.0
>
>
> See KAFKA-2376 for parent task and 
> [KIP-196|https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework]
>  for the details on the metrics. This subtask is to create the "Worker 
> Metrics".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5993) Kafka Admin Client throws a warning for sasl.jaas.config

[Stephane Maarek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephane Maarek updated KAFKA-5993:
---
Summary: Kafka Admin Client throws a warning for sasl.jaas.config  (was: 
Kafka AdminClient does not support standard security settings)

> Kafka Admin Client throws a warning for sasl.jaas.config
> 
>
> Key: KAFKA-5993
> URL: https://issues.apache.org/jira/browse/KAFKA-5993
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.1
>Reporter: Stephane Maarek
>
> Kafka Admin Client does not support basic security configurations, such as 
> "sasl.jaas.config".
> Therefore it makes it impossible to use against a secure cluster
> ```
> 14:12:12.948 [main] WARN  org.apache.kafka.clients.admin.AdminClientConfig - 
> The configuration 'sasl.jaas.config' was supplied but isn't a known config.
> ```



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-5993) Kafka AdminClient does not support standard security settings

[Stephane Maarek (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190808#comment-16190808
 ] 

Stephane Maarek edited comment on KAFKA-5993 at 10/4/17 4:52 AM:
-

[~ijuma] Indeed it does support security settings. The log still shows that 
WARN, which is what was misleading on my end.

See full log here (it's using the PLAINTEXT protocol on purpose for now):

{code:java}
15:42:18.548 [main] INFO  org.apache.kafka.clients.admin.AdminClientConfig - 
AdminClientConfig values: 
bootstrap.servers = [localhost:9092]
client.id = 
connections.max.idle.ms = 30
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 12
retries = 5
retry.backoff.ms = 100
sasl.jaas.config = [hidden]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 6
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS

15:42:18.548 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name connections-closed:
15:42:18.548 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name connections-created:
15:42:18.548 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name bytes-sent-received:
15:42:18.548 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name bytes-sent:
15:42:18.549 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name bytes-received:
15:42:18.549 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name select-time:
15:42:18.549 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name io-time:
15:42:18.549 [main] DEBUG org.apache.kafka.clients.Metadata - Updated cluster 
metadata version 1 to Cluster(id = null, nodes = [localhost:9092 (id: -1 rack: 
null)], partitions = [])
15:42:18.550 [main] WARN  org.apache.kafka.clients.admin.AdminClientConfig - 
The configuration 'sasl.jaas.config' was supplied but isn't a known config.
15:42:18.550 [main] INFO  org.apache.kafka.common.utils.AppInfoParser - Kafka 
version : 0.11.0.1
15:42:18.550 [main] INFO  org.apache.kafka.common.utils.AppInfoParser - Kafka 
commitId : c2a0d5f9b1f45bf5
{code}

My code to generate my admin client is:
  
{code:java}
  public static AdminClient getAdminClient() {

Properties adminProps = new Properties();
adminProps.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,

Optional.ofNullable(System.getenv("KAFKA_BOOTSTRAP_SERVERS")).orElse("localhost:9092"));
adminProps.put(AdminClientConfig.SECURITY_PROTOCOL_CONFIG,

Optional.ofNullable(System.getenv("SECURITY_PROTOCOL")).orElse("PLAINTEXT"));
adminProps.put(SaslConfigs.SASL_JAAS_CONFIG,

Optional.ofNullable(System.getenv("SASL_JAAS_CONFIG")).orElse(""));

return AdminClient.create(adminProps);
}

{code}

If you feel that WARN is okay, we can close the JIRA, otherwise maybe we should 
rename the JIRA and address the WARN?


was (Author: stephane.maa...@gmail.com):
[~ijuma] Indeed it does support security settings. The log still shows that 
WARN, which is what was misleading on my end.

See full log here:

{code:java}
15:42:18.548 [main] INFO  org.apache.kafka.clients.admin.AdminClientConfig - 
AdminClientConfig values: 
bootstrap.servers = [localhost:9092]
client.id = 
connections.max.idle.ms = 30
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 12
retries = 5
retry.backoff.ms = 100
sasl.jaas.config = 

[jira] [Commented] (KAFKA-5993) Kafka AdminClient does not support standard security settings

[Stephane Maarek (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190808#comment-16190808
 ] 

Stephane Maarek commented on KAFKA-5993:


[~ijuma] Indeed it does support security settings. The log still shows that 
WARN, which is what was misleading on my end.

See full log here:

{code:java}
15:42:18.548 [main] INFO  org.apache.kafka.clients.admin.AdminClientConfig - 
AdminClientConfig values: 
bootstrap.servers = [localhost:9092]
client.id = 
connections.max.idle.ms = 30
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 12
retries = 5
retry.backoff.ms = 100
sasl.jaas.config = [hidden]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 6
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS

15:42:18.548 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name connections-closed:
15:42:18.548 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name connections-created:
15:42:18.548 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name bytes-sent-received:
15:42:18.548 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name bytes-sent:
15:42:18.549 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name bytes-received:
15:42:18.549 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name select-time:
15:42:18.549 [main] DEBUG org.apache.kafka.common.metrics.Metrics - Added 
sensor with name io-time:
15:42:18.549 [main] DEBUG org.apache.kafka.clients.Metadata - Updated cluster 
metadata version 1 to Cluster(id = null, nodes = [localhost:9092 (id: -1 rack: 
null)], partitions = [])
15:42:18.550 [main] WARN  org.apache.kafka.clients.admin.AdminClientConfig - 
The configuration 'sasl.jaas.config' was supplied but isn't a known config.
15:42:18.550 [main] INFO  org.apache.kafka.common.utils.AppInfoParser - Kafka 
version : 0.11.0.1
15:42:18.550 [main] INFO  org.apache.kafka.common.utils.AppInfoParser - Kafka 
commitId : c2a0d5f9b1f45bf5
{code}

My code to generate my admin client is:
  
{code:java}
  public static AdminClient getAdminClient() {

Properties adminProps = new Properties();
adminProps.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,

Optional.ofNullable(System.getenv("KAFKA_BOOTSTRAP_SERVERS")).orElse("localhost:9092"));
adminProps.put(AdminClientConfig.SECURITY_PROTOCOL_CONFIG,

Optional.ofNullable(System.getenv("SECURITY_PROTOCOL")).orElse("PLAINTEXT"));
adminProps.put(SaslConfigs.SASL_JAAS_CONFIG,

Optional.ofNullable(System.getenv("SASL_JAAS_CONFIG")).orElse(""));

return AdminClient.create(adminProps);
}

{code}

If you feel that WARN is okay, we can close the JIRA, otherwise maybe we should 
rename the JIRA and address the WARN?

> Kafka AdminClient does not support standard security settings
> -
>
> Key: KAFKA-5993
> URL: https://issues.apache.org/jira/browse/KAFKA-5993
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.1
>Reporter: Stephane Maarek
>
> Kafka Admin Client does not support basic security configurations, such as 
> "sasl.jaas.config".
> Therefore it makes it impossible to use against a secure cluster
> ```
> 14:12:12.948 [main] WARN  org.apache.kafka.clients.admin.AdminClientConfig - 
> The configuration 'sasl.jaas.config' was supplied but isn't a known config.
> ```



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Component/s: metrics

> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to {{KafkaConsumer}}, but these metrics 
> put the {{TOPIC-PARTITION_ID}} inside of the metric name itself. These 
> metrics should instead utilize the tags and put {{key="topic-partition"}} and 
> {{value="TOPIC-PARTITION_ID"}}. 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> {{key="node/topic"}} and {{value="NODE_ID/TOPIC_NAME"{}}}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5758) Reassigning a topic's partitions can adversely impact other topics

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190609#comment-16190609
 ] 

Jun Rao commented on KAFKA-5758:


[~dwvangeest], thanks for the info.

> Reassigning a topic's partitions can adversely impact other topics
> --
>
> Key: KAFKA-5758
> URL: https://issues.apache.org/jira/browse/KAFKA-5758
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.1
>Reporter: David van Geest
>Assignee: Ismael Juma
>  Labels: reliability
> Fix For: 1.0.0
>
>
> We've noticed that reassigning a topic's partitions seems to adversely impact 
> other topics. Specifically, followers for other topics fall out of the ISR.
> While I'm not 100% sure about why this happens, the scenario seems to be as 
> follows:
> 1. Reassignment is manually triggered on topic-partition X-Y, and broker A 
> (which used to be a follower for X-Y) is no longer a follower.
> 2. Broker A makes `FetchRequest` including topic-partition X-Y to broker B, 
> just after the reassignment.
> 3. Broker B can fulfill the `FetchRequest`, but while trying to do so it 
> tries to record the position of "follower" A. This fails, because broker A is 
> no longer a follower for X-Y (see exception below).
> 4. The entire `FetchRequest` request fails, and broker A's other followed 
> topics start falling behind.
> 5. Depending on the length of the reassignment, this sequence repeats.
> In step 3, we see exceptions like:
> {noformat}
> Error when handling request Name: FetchRequest; Version: 3; CorrelationId: 
> 46781859; ClientId: ReplicaFetcherThread-0-1001; ReplicaId: 1006; MaxWait: 
> 500 ms; MinBytes: 1 bytes; MaxBytes:10485760 bytes; RequestInfo: 
> 
> kafka.common.NotAssignedReplicaException: Leader 1001 failed to record 
> follower 1006's position -1 since the replica is not recognized to be one of 
> the assigned replicas 1001,1004,1005 for partition [topic_being_reassigned,5].
> at kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:249)
>   at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:923)
>   at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:920)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at 
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:920)
>   at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:481)
>   at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:534)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Does my assessment make sense? If so, this behaviour seems problematic. A few 
> changes that might improve matters (assuming I'm on the right track):
> 1. `FetchRequest` should be able to return partial results
> 2. The broker fulfilling the `FetchRequest` could ignore the 
> `NotAssignedReplicaException`, and return results without recording the 
> not-any-longer-follower position.
> This behaviour was experienced with 0.10.1.1, although looking at the 
> changelogs and the code in question, I don't see any reason why it would have 
> changed in later versions.
> Am very interested to have some discussion on this. Thanks!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6007) Connect can't validate against transforms in plugins.path

[Stephane Maarek (JIRA)]

Stephane Maarek created KAFKA-6007:
--

 Summary: Connect can't validate against transforms in plugins.path
 Key: KAFKA-6007
 URL: https://issues.apache.org/jira/browse/KAFKA-6007
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.1
Reporter: Stephane Maarek


Kafka Connect can't validate a custom transformation if placed in plugins path.
Here's the output I get on the validate call:


{code:java}
Invalid value com.mycorp.kafka.transforms.impl.FlattenSinkRecord for 
configuration transforms.Flat.type: Class 
com.mycorp.kafka.transforms.impl.FlattenSinkRecord could not be found.
Invalid value null for configuration transforms.Flat.type: Not a Transformation
"recommended_values": [   
"com.mycorp.kafka.transforms.Flatten$Key",
"com.mycorp.kafka.transforms.Flatten$Value",
"com.mycorp.kafka.transforms.impl.FlattenSinkRecord",
"org.apache.kafka.connect.transforms.Cast$Key",
"org.apache.kafka.connect.transforms.Cast$Value",
"org.apache.kafka.connect.transforms.ExtractField$Key",
"org.apache.kafka.connect.transforms.ExtractField$Value",
"org.apache.kafka.connect.transforms.Flatten$Key",
"org.apache.kafka.connect.transforms.Flatten$Value",
"org.apache.kafka.connect.transforms.HoistField$Key",
"org.apache.kafka.connect.transforms.HoistField$Value",
"org.apache.kafka.connect.transforms.InsertField$Key",
"org.apache.kafka.connect.transforms.InsertField$Value",
"org.apache.kafka.connect.transforms.MaskField$Key",
"org.apache.kafka.connect.transforms.MaskField$Value",
"org.apache.kafka.connect.transforms.RegexRouter",
"org.apache.kafka.connect.transforms.ReplaceField$Key",
"org.apache.kafka.connect.transforms.ReplaceField$Value",
"org.apache.kafka.connect.transforms.SetSchemaMetadata$Key",
"org.apache.kafka.connect.transforms.SetSchemaMetadata$Value",
"org.apache.kafka.connect.transforms.TimestampConverter$Key",
"org.apache.kafka.connect.transforms.TimestampConverter$Value",
"org.apache.kafka.connect.transforms.TimestampRouter",
"org.apache.kafka.connect.transforms.ValueToKey"],

{code}

As you can see the class appear in the recommended values (!) but can't be 
picked up on the validate call. 

I believe it's because the recommender implements class discovery using plugins:
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L194

But the class inference itself doesn't:
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L199

(I'm not an expert in class loading though, just a guess... Unsure how to fix)

A quick fix is to add the transformations in the ClassPath itself, but that 
defeats the point a bit. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5902) Create Connect metrics for sink tasks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190148#comment-16190148
 ] 

ASF GitHub Bot commented on KAFKA-5902:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3975


> Create Connect metrics for sink tasks
> -
>
> Key: KAFKA-5902
> URL: https://issues.apache.org/jira/browse/KAFKA-5902
> Project: Kafka
>  Issue Type: Sub-task
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Blocker
> Fix For: 1.0.0
>
>
> See KAFKA-2376 for parent task and 
> [KIP-196|https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework]
>  for the details on the metrics. This subtask is to create the "Sink Task 
> Metrics".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5902) Create Connect metrics for sink tasks

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-5902.
--
Resolution: Fixed

Issue resolved by pull request 3975
[https://github.com/apache/kafka/pull/3975]

> Create Connect metrics for sink tasks
> -
>
> Key: KAFKA-5902
> URL: https://issues.apache.org/jira/browse/KAFKA-5902
> Project: Kafka
>  Issue Type: Sub-task
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Blocker
> Fix For: 1.0.0
>
>
> See KAFKA-2376 for parent task and 
> [KIP-196|https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework]
>  for the details on the metrics. This subtask is to create the "Sink Task 
> Metrics".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4108) Improve DumpLogSegments offsets-decoder output format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190041#comment-16190041
 ] 

ASF GitHub Bot commented on KAFKA-4108:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1937


> Improve DumpLogSegments offsets-decoder output format
> -
>
> Key: KAFKA-4108
> URL: https://issues.apache.org/jira/browse/KAFKA-4108
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 1.0.0
>
>
> When using the DumpLogSegments with the "--offsets-decoder" option (for 
> consuming __consumer_offsets), the encoding of group metadata makes it a 
> little difficult to identify individual fields. In particular, we use the 
> following formatted string for group metadata: 
> {code}
> ${protocolType}:${groupMetadata.protocol}:${groupMetadata.generationId}:${assignment}
> {code}
> Keys have a similar formatting. Most users are probably not going to know 
> which field is which based only on the output, so it would be helpful to 
> include field names. Maybe we could just output a JSON object?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5758) Reassigning a topic's partitions can adversely impact other topics

[David van Geest (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190025#comment-16190025
 ] 

David van Geest commented on KAFKA-5758:


[~junrao] yeah, I'm not 100% sure either that it's the root cause - it just 
seemed the most obvious place to start. 

It's a little bit hard to tell how long the (supposedly unaffected) topics were 
under-replicated. Our replication graphs don't differentiate between topics, 
and the topics undergoing re-assignment did take a while to fullly 
re-replicate. If I had to guess, I would say somewhere between 3 - 5 minutes of 
under-replication for the non-reassigned topics. This 3 - 5 minute period 
started at the same time as the first `NotAssignedReplicaException`s. Does that 
help?

> Reassigning a topic's partitions can adversely impact other topics
> --
>
> Key: KAFKA-5758
> URL: https://issues.apache.org/jira/browse/KAFKA-5758
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.1
>Reporter: David van Geest
>Assignee: Ismael Juma
>  Labels: reliability
> Fix For: 1.0.0
>
>
> We've noticed that reassigning a topic's partitions seems to adversely impact 
> other topics. Specifically, followers for other topics fall out of the ISR.
> While I'm not 100% sure about why this happens, the scenario seems to be as 
> follows:
> 1. Reassignment is manually triggered on topic-partition X-Y, and broker A 
> (which used to be a follower for X-Y) is no longer a follower.
> 2. Broker A makes `FetchRequest` including topic-partition X-Y to broker B, 
> just after the reassignment.
> 3. Broker B can fulfill the `FetchRequest`, but while trying to do so it 
> tries to record the position of "follower" A. This fails, because broker A is 
> no longer a follower for X-Y (see exception below).
> 4. The entire `FetchRequest` request fails, and broker A's other followed 
> topics start falling behind.
> 5. Depending on the length of the reassignment, this sequence repeats.
> In step 3, we see exceptions like:
> {noformat}
> Error when handling request Name: FetchRequest; Version: 3; CorrelationId: 
> 46781859; ClientId: ReplicaFetcherThread-0-1001; ReplicaId: 1006; MaxWait: 
> 500 ms; MinBytes: 1 bytes; MaxBytes:10485760 bytes; RequestInfo: 
> 
> kafka.common.NotAssignedReplicaException: Leader 1001 failed to record 
> follower 1006's position -1 since the replica is not recognized to be one of 
> the assigned replicas 1001,1004,1005 for partition [topic_being_reassigned,5].
> at kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:249)
>   at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:923)
>   at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:920)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at 
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:920)
>   at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:481)
>   at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:534)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Does my assessment make sense? If so, this behaviour seems problematic. A few 
> changes that might improve matters (assuming I'm on the right track):
> 1. `FetchRequest` should be able to return partial results
> 2. The broker fulfilling the `FetchRequest` could ignore the 
> `NotAssignedReplicaException`, and return results without recording the 
> not-any-longer-follower position.
> This behaviour was experienced with 0.10.1.1, although looking at the 
> changelogs and the code in question, I don't see any reason why it would have 
> changed in later versions.
> Am very interested to have some discussion on this. Thanks!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5909) Remove source jars from classpath while executing CLI tools

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189965#comment-16189965
 ] 

ASF GitHub Bot commented on KAFKA-5909:
---

GitHub user Kamal15 opened a pull request:

https://github.com/apache/kafka/pull/4010

KAFKA-5909; Removed the source jars from classpath

Executing CLI tools don't require source jars so removed it from the 
classpath.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Kamal15/kafka patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4010


commit e52eceb29420124dc6f3f31a6d026cb05ff1bd82
Author: Kamal C 
Date:   2017-10-03T16:56:37Z

KAFKA-5909; Removed the source jars from classpath

Executing CLI tools don't require source jars so removed it from the 
classpath.




> Remove source jars from classpath while executing CLI tools
> ---
>
> Key: KAFKA-5909
> URL: https://issues.apache.org/jira/browse/KAFKA-5909
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.11.0.0
>Reporter: Kamal Chandraprakash
>Assignee: Kamal Chandraprakash
>Priority: Minor
>  Labels: newbie
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5909) Remove source jars from classpath while executing CLI tools

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189958#comment-16189958
 ] 

ASF GitHub Bot commented on KAFKA-5909:
---

Github user Kamal15 closed the pull request at:

https://github.com/apache/kafka/pull/3871


> Remove source jars from classpath while executing CLI tools
> ---
>
> Key: KAFKA-5909
> URL: https://issues.apache.org/jira/browse/KAFKA-5909
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.11.0.0
>Reporter: Kamal Chandraprakash
>Assignee: Kamal Chandraprakash
>Priority: Minor
>  Labels: newbie
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-6006) Default replication factor for all topics/partitions should be set in server.properties

[Martin M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin M updated KAFKA-6006:

Description: 
The replication factor should be set globally in server.properties such that it 
can be applied by default to all topics/partitions even for topics that are not 
auto-created. Having the configuration globally will allow consumers/producers 
have the same setting. 
One such situation is when consumers are distributed on different servers.

The AdminClient interface for creating a new topic would require a change to 
request the default replication factor, either by changing the argument to 
Short, thereby allowing null, or designating a special value, e.g. -1.

  was:
The replication factor should be set globally in server.properties such that it 
can be applied by default to all topics/partitions even for topics that are not 
auto-created. Having the configuration globally will allow consumers/producers 
have the same setting. 
One such situation is when consumers are distributed on different servers.


> Default replication factor for all topics/partitions should be set in 
> server.properties 
> 
>
> Key: KAFKA-6006
> URL: https://issues.apache.org/jira/browse/KAFKA-6006
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Martin M
>
> The replication factor should be set globally in server.properties such that 
> it can be applied by default to all topics/partitions even for topics that 
> are not auto-created. Having the configuration globally will allow 
> consumers/producers have the same setting. 
> One such situation is when consumers are distributed on different servers.
> The AdminClient interface for creating a new topic would require a change to 
> request the default replication factor, either by changing the argument to 
> Short, thereby allowing null, or designating a special value, e.g. -1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-6000) streams 0.10.2.1 - kafka 0.11.0.1 state restore not working

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189947#comment-16189947
 ] 

Ted Yu commented on KAFKA-6000:
---

I wonder if the following had something to do with the problem:
{code}
13:59:09.538 [StreamThread-2] DEBUG  o.a.k.c.c.i.Fetcher - Ignoring fetched 
records for scratch.lateststate.dsh-2 at offset 1969072 since the current 
position is 2134335
{code}

> streams 0.10.2.1 - kafka 0.11.0.1 state restore not working
> ---
>
> Key: KAFKA-6000
> URL: https://issues.apache.org/jira/browse/KAFKA-6000
> Project: Kafka
>  Issue Type: Bug
>  Components: core, streams
>Affects Versions: 0.10.2.1, 0.11.0.0
>Reporter: Bart Vercammen
>Priority: Blocker
> Attachments: correct-restore.log, failed-restore.log
>
>
> Potential interop issue between Kafka Streams (0.10.2.1) and Kafka (0.11.0.1)
> {noformat}
> 11:24:16.416 [StreamThread-3] DEBUG rocessorStateManager - task [0_2] 
> Registering state store lateststate to its state manager 
> 11:24:16.472 [StreamThread-3] TRACE rocessorStateManager - task [0_2] 
> Restoring state store lateststate from changelog topic 
> scratch.lateststate.dsh 
> 11:24:16.472 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Resetting offset 
> for partition scratch.lateststate.dsh-2 to latest offset. 
> 11:24:16.472 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Partition 
> scratch.lateststate.dsh-2 is unknown for fetching offset, wait for metadata 
> refresh 
> 11:24:16.474 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Sending 
> ListOffsetRequest (type=ListOffsetRequest, replicaId=-1, 
> partitionTimestamps={scratch.lateststate.dsh-2=-1}, minVersion=0) to broker 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.476 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Received 
> ListOffsetResponse 
> {responses=[{topic=scratch.lateststate.dsh,partition_responses=[{partition=2,error_code=0,timestamp=-1,offset=1773763}]}]}
>  from broker broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.476 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Handling 
> ListOffsetResponse response for scratch.lateststate.dsh-2. Fetched offset 
> 1773763, timestamp -1 
> 11:24:16.477 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Resetting offset 
> for partition scratch.lateststate.dsh-2 to earliest offset. 
> 11:24:16.478 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Sending 
> ListOffsetRequest (type=ListOffsetRequest, replicaId=-1, 
> partitionTimestamps={scratch.lateststate.dsh-2=-2}, minVersion=0) to broker 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.480 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Received 
> ListOffsetResponse 
> {responses=[{topic=scratch.lateststate.dsh,partition_responses=[{partition=2,error_code=0,timestamp=-1,offset=0}]}]}
>  from broker broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.481 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Handling 
> ListOffsetResponse response for scratch.lateststate.dsh-2. Fetched offset 0, 
> timestamp -1 
> 11:24:16.483 [StreamThread-3] DEBUG rocessorStateManager - restoring 
> partition scratch.lateststate.dsh-2 from offset 0 to endOffset 1773763 
> 11:24:16.484 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Added fetch 
> request for partition scratch.lateststate.dsh-2 at offset 0 to node 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.485 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Sending fetch for 
> partitions [scratch.lateststate.dsh-2] to broker 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.486 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Skipping fetch for 
> partition scratch.lateststate.dsh-2 because there is an in-flight request to 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.490 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Adding fetched 
> record for partition scratch.lateststate.dsh-2 with offset 0 to buffered 
> record list 
> 11:24:16.492 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Received 3 records 
> in fetch response for partition scratch.lateststate.dsh-2 with offset 0 
> 11:24:16.493 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Returning fetched 
> records at offset 0 for assigned partition scratch.lateststate.dsh-2 and 
> update position to 1586527 
> 11:24:16.494 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Ignoring fetched 
> records for scratch.lateststate.dsh-2 at offset 0 since the current position 
> is 1586527 
> 11:24:16.496 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Added fetch 
> request for partition scratch.lateststate.dsh-2 at offset 1586527 to node 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.496 [StreamThread-3] DEBUG  

[jira] [Commented] (KAFKA-6003) Replication Fetcher thread for a partition with no data fails to start

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189937#comment-16189937
 ] 

Apurva Mehta commented on KAFKA-6003:
-

Unfortunately your summary is correct :(

We have tests which actually do cluster expansion / replica movement, but 
unfortunately the workload in those tests doesn't include idempotent messages, 
so we didn't catch this. 

We will update the tests to use the idempotent producer so that we can catch 
any further bugs in these scenarios. 

For now, we can start the conversation about the 0.11.0.2 release on the 
mailing list, and see what the community thinks of a quick release just to fix 
this issue.

> Replication Fetcher thread for a partition with no data fails to start
> --
>
> Key: KAFKA-6003
> URL: https://issues.apache.org/jira/browse/KAFKA-6003
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.11.0.1
>Reporter: Stanislav Chizhov
>Assignee: Apurva Mehta
>Priority: Blocker
> Fix For: 1.0.0, 0.11.0.2
>
>
> If a partition of a topic with idempotent producer has no data on 1 of the 
> brokers, but it does exist on others and some of the segments for this 
> partition have been already deleted replication thread responsible for this 
> partition on the broker which has no data for it fails to start with out of 
> order sequence exception:
> {code}
> [2017-10-02 09:44:23,825] ERROR [ReplicaFetcherThread-2-4]: Error due to 
> (kafka.server.ReplicaFetcherThread)
> kafka.common.KafkaException: error processing data for partition 
> [stage.data.adevents.v2,20] offset 1660336429
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:203)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:174)
> at scala.Option.foreach(Option.scala:257)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:174)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:171)
> at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:169)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> Caused by: org.apache.kafka.common.errors.OutOfOrderSequenceException: 
> Invalid sequence number for new epoch: 0 (request epoch), 154277489 (seq. 
> number)
> {code}
> We run kafka 0.11.0.1 and we ran into the situation when 1 of replication 
> threads was stopped for few days, while everything else on that broker was 
> functional. This is our staging cluster and retention is less than a day, so 
> everything for partitions for which replication thread was down was cleaned 
> up. At the moment we have a broker which cannot start replication for few 
> partitions. I was also able to reproduce in my local test environment.
> Another possible use case when this might cause real pain is disk failure or 
> any situation when previously deleting all the data for the partition on a 
> broker helped - since it would just fetch all the data from other replicas. 
> Now it does not work for topics with idempotent producers. It might also 
> affect other not-idempotent topics if those are unlucky to share same 
> replication fetcher thread. 
> This seems to be caused by this logic: 
> https://github.com/apache/kafka/blob/0.11.0.1/core/src/main/scala/kafka/log/ProducerStateManager.scala#L119
> and might be fixed in the scope of 
> https://issues.apache.org/jira/browse/KAFKA-5793.
> However any hints on how to get those partition to fully replicated state are 
> highly appreciated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4201) Add an --assignment-strategy option to new-consumer-based Mirror Maker

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189903#comment-16189903
 ] 

Vahid Hashemian commented on KAFKA-4201:


[~ijuma] It wasn't brought up during an earlier discussion. Should I create 
one? Thanks.

> Add an --assignment-strategy option to new-consumer-based Mirror Maker
> --
>
> Key: KAFKA-4201
> URL: https://issues.apache.org/jira/browse/KAFKA-4201
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
> Fix For: 1.0.0
>
>
> The default assignment strategy in mirror maker will be changed from range to 
> round robin in an upcoming release ([see 
> KAFKA-3818|https://issues.apache.org/jira/browse/KAFKA-3818]). In order to 
> make it easier for users to change the assignment strategy, add an 
> {{--assignment-strategy}} option to Mirror Maker command line tool.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-6005) Reject JoinGroup request from first member with empty protocol type/protocol list

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189879#comment-16189879
 ] 

ASF GitHub Bot commented on KAFKA-6005:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3957


> Reject JoinGroup request from first member with empty protocol type/protocol 
> list
> -
>
> Key: KAFKA-6005
> URL: https://issues.apache.org/jira/browse/KAFKA-6005
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Minor
> Fix For: 1.0.0
>
>
> Currently, if the first group member joins with empty 
> partition.assignment.strategy, then the group won't allow any other members 
> with valid protocols.  This JIRA is to add validation to reject JoinGroup 
> requests from the first member with empty protocol type or empty protocol list
> Also, add consumer-side validations to check at least one partition assigner 
> class name is configured while using subscribe APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-6005) Reject JoinGroup request from first member with empty protocol type/protocol list

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-6005.

Resolution: Fixed

Issue resolved by pull request 3957
[https://github.com/apache/kafka/pull/3957]

> Reject JoinGroup request from first member with empty protocol type/protocol 
> list
> -
>
> Key: KAFKA-6005
> URL: https://issues.apache.org/jira/browse/KAFKA-6005
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Minor
> Fix For: 1.0.0
>
>
> Currently, if the first group member joins with empty 
> partition.assignment.strategy, then the group won't allow any other members 
> with valid protocols.  This JIRA is to add validation to reject JoinGroup 
> requests from the first member with empty protocol type or empty protocol list
> Also, add consumer-side validations to check at least one partition assigner 
> class name is configured while using subscribe APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-6006) Default replication factor for all topics should be set in server.properties

[Martin M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin M updated KAFKA-6006:

Summary: Default replication factor for all topics should be set in 
server.properties   (was: Default replication factor should be set in 
server.properties for all topics)

> Default replication factor for all topics should be set in server.properties 
> -
>
> Key: KAFKA-6006
> URL: https://issues.apache.org/jira/browse/KAFKA-6006
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Martin M
>
> The replication factor should be set globally in server.properties such that 
> it can be applied by default to all topics/partitions even when topic is not 
> auto-created. Having the configuration globally will allow 
> consumers/producers have the same setting. 
> One such situation is when consumers are distributed on different servers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-6006) Default replication factor for all topics/partitions should be set in server.properties

[Martin M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin M updated KAFKA-6006:

Summary: Default replication factor for all topics/partitions should be set 
in server.properties   (was: Default replication factor for all topics should 
be set in server.properties )

> Default replication factor for all topics/partitions should be set in 
> server.properties 
> 
>
> Key: KAFKA-6006
> URL: https://issues.apache.org/jira/browse/KAFKA-6006
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Martin M
>
> The replication factor should be set globally in server.properties such that 
> it can be applied by default to all topics/partitions even when topic is not 
> auto-created. Having the configuration globally will allow 
> consumers/producers have the same setting. 
> One such situation is when consumers are distributed on different servers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-6006) Default replication factor for all topics/partitions should be set in server.properties

[Martin M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin M updated KAFKA-6006:

Description: 
The replication factor should be set globally in server.properties such that it 
can be applied by default to all topics/partitions even for topics that are not 
auto-created. Having the configuration globally will allow consumers/producers 
have the same setting. 
One such situation is when consumers are distributed on different servers.

  was:
The replication factor should be set globally in server.properties such that it 
can be applied by default to all topics/partitions even when topic is not 
auto-created. Having the configuration globally will allow consumers/producers 
have the same setting. 
One such situation is when consumers are distributed on different servers.


> Default replication factor for all topics/partitions should be set in 
> server.properties 
> 
>
> Key: KAFKA-6006
> URL: https://issues.apache.org/jira/browse/KAFKA-6006
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Martin M
>
> The replication factor should be set globally in server.properties such that 
> it can be applied by default to all topics/partitions even for topics that 
> are not auto-created. Having the configuration globally will allow 
> consumers/producers have the same setting. 
> One such situation is when consumers are distributed on different servers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-6006) Default replication factor should be set in server.properties for all topics

[Martin M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin M updated KAFKA-6006:

Summary: Default replication factor should be set in server.properties for 
all topics  (was: Default replication factor should be set in server.properties)

> Default replication factor should be set in server.properties for all topics
> 
>
> Key: KAFKA-6006
> URL: https://issues.apache.org/jira/browse/KAFKA-6006
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Martin M
>
> The replication factor should be set globally in server.properties such that 
> it can be applied by default to all topics/partitions even when topic is not 
> auto-created. Having the configuration globally will allow 
> consumers/producers have the same setting. 
> One such situation is when consumers are distributed on different servers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-6000) streams 0.10.2.1 - kafka 0.11.0.1 state restore not working

[Bart Vercammen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bart Vercammen updated KAFKA-6000:
--
Attachment: failed-restore.log
correct-restore.log

I added 2 log-files : 
- {{correct-restore.log}}
   This is the log, grepped on the StreamThread that has a successful restore 
of the changelog _scratch.lateststate.dsh_

- {{failed-restore.log}}
   This is the log, grepped on the StreamThread that has a failing restore of 
the changelog _scratch.lateststate.dsh_

Both restores were running at the same time, but as shown, failing for 1 
partition and successful for the other.

The server logs are a bit more tricky as these logs were taken from a running 
platform on which a lot of consumers and producers are active, so it is hard to 
get detailed logs from the kafka brokers specifically for that one kafka 
streams application.

I'm still trying to reproduce this off-line (in a unit test), but without 
success up till now.
Somehow in my unit tests, all restores seem to work perfectly.

In {{failed-restore.log}} :
 on line 12, we get the end-offset -> 2386570
 on line 82, we see a first successful fetch
 on line 128, it starts to go wrong, and the restore is "stuck"

> streams 0.10.2.1 - kafka 0.11.0.1 state restore not working
> ---
>
> Key: KAFKA-6000
> URL: https://issues.apache.org/jira/browse/KAFKA-6000
> Project: Kafka
>  Issue Type: Bug
>  Components: core, streams
>Affects Versions: 0.10.2.1, 0.11.0.0
>Reporter: Bart Vercammen
>Priority: Blocker
> Attachments: correct-restore.log, failed-restore.log
>
>
> Potential interop issue between Kafka Streams (0.10.2.1) and Kafka (0.11.0.1)
> {noformat}
> 11:24:16.416 [StreamThread-3] DEBUG rocessorStateManager - task [0_2] 
> Registering state store lateststate to its state manager 
> 11:24:16.472 [StreamThread-3] TRACE rocessorStateManager - task [0_2] 
> Restoring state store lateststate from changelog topic 
> scratch.lateststate.dsh 
> 11:24:16.472 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Resetting offset 
> for partition scratch.lateststate.dsh-2 to latest offset. 
> 11:24:16.472 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Partition 
> scratch.lateststate.dsh-2 is unknown for fetching offset, wait for metadata 
> refresh 
> 11:24:16.474 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Sending 
> ListOffsetRequest (type=ListOffsetRequest, replicaId=-1, 
> partitionTimestamps={scratch.lateststate.dsh-2=-1}, minVersion=0) to broker 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.476 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Received 
> ListOffsetResponse 
> {responses=[{topic=scratch.lateststate.dsh,partition_responses=[{partition=2,error_code=0,timestamp=-1,offset=1773763}]}]}
>  from broker broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.476 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Handling 
> ListOffsetResponse response for scratch.lateststate.dsh-2. Fetched offset 
> 1773763, timestamp -1 
> 11:24:16.477 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Resetting offset 
> for partition scratch.lateststate.dsh-2 to earliest offset. 
> 11:24:16.478 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Sending 
> ListOffsetRequest (type=ListOffsetRequest, replicaId=-1, 
> partitionTimestamps={scratch.lateststate.dsh-2=-2}, minVersion=0) to broker 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.480 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Received 
> ListOffsetResponse 
> {responses=[{topic=scratch.lateststate.dsh,partition_responses=[{partition=2,error_code=0,timestamp=-1,offset=0}]}]}
>  from broker broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.481 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Handling 
> ListOffsetResponse response for scratch.lateststate.dsh-2. Fetched offset 0, 
> timestamp -1 
> 11:24:16.483 [StreamThread-3] DEBUG rocessorStateManager - restoring 
> partition scratch.lateststate.dsh-2 from offset 0 to endOffset 1773763 
> 11:24:16.484 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Added fetch 
> request for partition scratch.lateststate.dsh-2 at offset 0 to node 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.485 [StreamThread-3] DEBUG  o.a.k.c.c.i.Fetcher - Sending fetch for 
> partitions [scratch.lateststate.dsh-2] to broker 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.486 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Skipping fetch for 
> partition scratch.lateststate.dsh-2 because there is an in-flight request to 
> broker-1.tt.kafka.marathon.mesos:9091 (id: 1002 rack: null) 
> 11:24:16.490 [StreamThread-3] TRACE  o.a.k.c.c.i.Fetcher - Adding fetched 
> record for partition scratch.lateststate.dsh-2 

[jira] [Updated] (KAFKA-6006) Default replication factor should be set in server.properties

[Martin M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin M updated KAFKA-6006:

Description: 
The replication factor should be set globally in server.properties such that it 
can be applied by default to all topics/partitions even when topic is not 
auto-created. Having the configuration globally will allow consumers/producers 
have the same setting. 
One such situation is when consumers are distributed on different servers.

  was:
The replication factor should be set globally in server.properties such that it 
can be applied by default to all topics/partitions when topic is not 
auto-created. Having the configuration globally will allow consumers/producers 
have the same setting. 
One such situation is when consumers are distributed on different servers.


> Default replication factor should be set in server.properties
> -
>
> Key: KAFKA-6006
> URL: https://issues.apache.org/jira/browse/KAFKA-6006
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Martin M
>
> The replication factor should be set globally in server.properties such that 
> it can be applied by default to all topics/partitions even when topic is not 
> auto-created. Having the configuration globally will allow 
> consumers/producers have the same setting. 
> One such situation is when consumers are distributed on different servers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6006) Default replication factor should be set in server.properties

[Martin M (JIRA)]

Martin M created KAFKA-6006:
---

 Summary: Default replication factor should be set in 
server.properties
 Key: KAFKA-6006
 URL: https://issues.apache.org/jira/browse/KAFKA-6006
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.11.0.0
Reporter: Martin M


The replication factor should be set globally in server.properties such that it 
can be applied by default to all topics/partitions when topic is not 
auto-created. Having the configuration globally will allow consumers/producers 
have the same setting. 
One such situation is when consumers are distributed on different servers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-6003) Replication Fetcher thread for a partition with no data fails to start

[Stanislav Chizhov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189731#comment-16189731
 ] 

Stanislav Chizhov edited comment on KAFKA-6003 at 10/3/17 1:53 PM:
---

So to summarise the impact:
In case there is a topic with finite retention and an idempotent producer there 
is very high risk of not being able to:
- replace a dead broker
- scale up cluster
- reassign partitions (BTW I did try it and it failed in the same way)

If the above is true it renders idempotent producer feature pretty much 
unusable.
Please correct me if I am wrong.



was (Author: schizhov):
So to summarise the impact:
In case there is a topic with finite retention and idempotent producer there is 
very high risk of not being able to:
- replace a dead broker
- scale up cluster
- reassign partitions (BTW I did try it and it failed in the same way)

If the above is true it renders idempotent producer feature pretty much 
unusable.
Please correct me if I am wrong.


> Replication Fetcher thread for a partition with no data fails to start
> --
>
> Key: KAFKA-6003
> URL: https://issues.apache.org/jira/browse/KAFKA-6003
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.11.0.1
>Reporter: Stanislav Chizhov
>Assignee: Apurva Mehta
>Priority: Blocker
> Fix For: 1.0.0, 0.11.0.2
>
>
> If a partition of a topic with idempotent producer has no data on 1 of the 
> brokers, but it does exist on others and some of the segments for this 
> partition have been already deleted replication thread responsible for this 
> partition on the broker which has no data for it fails to start with out of 
> order sequence exception:
> {code}
> [2017-10-02 09:44:23,825] ERROR [ReplicaFetcherThread-2-4]: Error due to 
> (kafka.server.ReplicaFetcherThread)
> kafka.common.KafkaException: error processing data for partition 
> [stage.data.adevents.v2,20] offset 1660336429
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:203)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:174)
> at scala.Option.foreach(Option.scala:257)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:174)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:171)
> at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:169)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> Caused by: org.apache.kafka.common.errors.OutOfOrderSequenceException: 
> Invalid sequence number for new epoch: 0 (request epoch), 154277489 (seq. 
> number)
> {code}
> We run kafka 0.11.0.1 and we ran into the situation when 1 of replication 
> threads was stopped for few days, while everything else on that broker was 
> functional. This is our staging cluster and retention is less than a day, so 
> everything for partitions for which replication thread was down was cleaned 
> up. At the moment we have a broker which cannot start replication for few 
> partitions. I was also able to reproduce in my local test environment.
> Another possible use case when this might cause real pain is disk failure or 
> any situation when previously deleting all the data for the partition on a 
> broker helped - since it would just fetch all the data from other replicas. 
> Now it does not work for topics with idempotent producers. It might also 
> affect other not-idempotent topics if those are unlucky to share same 
> replication fetcher thread. 
> This seems to be caused by this logic: 
> https://github.com/apache/kafka/blob/0.11.0.1/core/src/main/scala/kafka/log/ProducerStateManager.scala#L119
> and might be fixed in the scope of 
> 

[jira] [Comment Edited] (KAFKA-6003) Replication Fetcher thread for a partition with no data fails to start

[Stanislav Chizhov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189731#comment-16189731
 ] 

Stanislav Chizhov edited comment on KAFKA-6003 at 10/3/17 1:53 PM:
---

So to summarise the impact:
In case there is a topic with finite retention and idempotent producer there is 
very high risk of not being able to:
- replace a dead broker
- scale up cluster
- reassign partitions (BTW I did try it and it failed in the same way)

If the above is true it renders idempotent producer feature pretty much 
unusable.
Please correct me if I am wrong.



was (Author: schizhov):
So to summarise the impact:
In case there is topic with finite retention and idempotent producer there is 
very high risk of not being able to:
- replace a dead broker
- scale up cluster
- reassign partitions (BTW I did try it and it failed in the same way)

If the above is true it renders idempotent producer feature pretty much 
unusable.
Please correct me if I am wrong.


> Replication Fetcher thread for a partition with no data fails to start
> --
>
> Key: KAFKA-6003
> URL: https://issues.apache.org/jira/browse/KAFKA-6003
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.11.0.1
>Reporter: Stanislav Chizhov
>Assignee: Apurva Mehta
>Priority: Blocker
> Fix For: 1.0.0, 0.11.0.2
>
>
> If a partition of a topic with idempotent producer has no data on 1 of the 
> brokers, but it does exist on others and some of the segments for this 
> partition have been already deleted replication thread responsible for this 
> partition on the broker which has no data for it fails to start with out of 
> order sequence exception:
> {code}
> [2017-10-02 09:44:23,825] ERROR [ReplicaFetcherThread-2-4]: Error due to 
> (kafka.server.ReplicaFetcherThread)
> kafka.common.KafkaException: error processing data for partition 
> [stage.data.adevents.v2,20] offset 1660336429
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:203)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:174)
> at scala.Option.foreach(Option.scala:257)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:174)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:171)
> at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:169)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> Caused by: org.apache.kafka.common.errors.OutOfOrderSequenceException: 
> Invalid sequence number for new epoch: 0 (request epoch), 154277489 (seq. 
> number)
> {code}
> We run kafka 0.11.0.1 and we ran into the situation when 1 of replication 
> threads was stopped for few days, while everything else on that broker was 
> functional. This is our staging cluster and retention is less than a day, so 
> everything for partitions for which replication thread was down was cleaned 
> up. At the moment we have a broker which cannot start replication for few 
> partitions. I was also able to reproduce in my local test environment.
> Another possible use case when this might cause real pain is disk failure or 
> any situation when previously deleting all the data for the partition on a 
> broker helped - since it would just fetch all the data from other replicas. 
> Now it does not work for topics with idempotent producers. It might also 
> affect other not-idempotent topics if those are unlucky to share same 
> replication fetcher thread. 
> This seems to be caused by this logic: 
> https://github.com/apache/kafka/blob/0.11.0.1/core/src/main/scala/kafka/log/ProducerStateManager.scala#L119
> and might be fixed in the scope of 
> 

[jira] [Updated] (KAFKA-4879) KafkaConsumer.position may hang forever when deleting a topic

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4879:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> KafkaConsumer.position may hang forever when deleting a topic
> -
>
> Key: KAFKA-4879
> URL: https://issues.apache.org/jira/browse/KAFKA-4879
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.2.0
>Reporter: Shixiong Zhu
>Assignee: Balint Molnar
> Fix For: 1.1.0
>
>
> KafkaConsumer.position may hang forever when deleting a topic. The problem is 
> this line 
> https://github.com/apache/kafka/blob/022bf129518e33e165f9ceefc4ab9e622952d3bd/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L374
> The timeout is "Long.MAX_VALUE", and it will just retry forever for 
> UnknownTopicOrPartitionException.
> Here is a reproducer
> {code}
> import org.apache.kafka.clients.consumer.KafkaConsumer;
> import org.apache.kafka.common.TopicPartition;
> import org.apache.kafka.common.serialization.StringDeserializer;
> import java.util.Collections;
> import java.util.Properties;
> import java.util.Set;
> public class KafkaReproducer {
>   public static void main(String[] args) {
> // Make sure "delete.topic.enable" is set to true.
> // Please create the topic test with "3" partitions manually.
> // The issue is gone when there is only one partition.
> String topic = "test";
> Properties props = new Properties();
> props.put("bootstrap.servers", "localhost:9092");
> props.put("group.id", "testgroup");
> props.put("value.deserializer", StringDeserializer.class.getName());
> props.put("key.deserializer", StringDeserializer.class.getName());
> props.put("enable.auto.commit", "false");
> KafkaConsumer kc = new KafkaConsumer(props);
> kc.subscribe(Collections.singletonList(topic));
> kc.poll(0);
> Set partitions = kc.assignment();
> System.out.println("partitions: " + partitions);
> kc.pause(partitions);
> kc.seekToEnd(partitions);
> System.out.println("please delete the topic in 30 seconds");
> try {
>   // Sleep 30 seconds to give us enough time to delete the topic.
>   Thread.sleep(3);
> } catch (InterruptedException e) {
>   e.printStackTrace();
> }
> System.out.println("sleep end");
> for (TopicPartition p : partitions) {
>   System.out.println(p + " offset: " + kc.position(p));
> }
> System.out.println("cannot reach here");
> kc.close();
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4879) KafkaConsumer.position may hang forever when deleting a topic

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189600#comment-16189600
 ] 

Ismael Juma commented on KAFKA-4879:


This needs a KIP, so pushing to the next release.

> KafkaConsumer.position may hang forever when deleting a topic
> -
>
> Key: KAFKA-4879
> URL: https://issues.apache.org/jira/browse/KAFKA-4879
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.2.0
>Reporter: Shixiong Zhu
>Assignee: Balint Molnar
> Fix For: 1.1.0
>
>
> KafkaConsumer.position may hang forever when deleting a topic. The problem is 
> this line 
> https://github.com/apache/kafka/blob/022bf129518e33e165f9ceefc4ab9e622952d3bd/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L374
> The timeout is "Long.MAX_VALUE", and it will just retry forever for 
> UnknownTopicOrPartitionException.
> Here is a reproducer
> {code}
> import org.apache.kafka.clients.consumer.KafkaConsumer;
> import org.apache.kafka.common.TopicPartition;
> import org.apache.kafka.common.serialization.StringDeserializer;
> import java.util.Collections;
> import java.util.Properties;
> import java.util.Set;
> public class KafkaReproducer {
>   public static void main(String[] args) {
> // Make sure "delete.topic.enable" is set to true.
> // Please create the topic test with "3" partitions manually.
> // The issue is gone when there is only one partition.
> String topic = "test";
> Properties props = new Properties();
> props.put("bootstrap.servers", "localhost:9092");
> props.put("group.id", "testgroup");
> props.put("value.deserializer", StringDeserializer.class.getName());
> props.put("key.deserializer", StringDeserializer.class.getName());
> props.put("enable.auto.commit", "false");
> KafkaConsumer kc = new KafkaConsumer(props);
> kc.subscribe(Collections.singletonList(topic));
> kc.poll(0);
> Set partitions = kc.assignment();
> System.out.println("partitions: " + partitions);
> kc.pause(partitions);
> kc.seekToEnd(partitions);
> System.out.println("please delete the topic in 30 seconds");
> try {
>   // Sleep 30 seconds to give us enough time to delete the topic.
>   Thread.sleep(3);
> } catch (InterruptedException e) {
>   e.printStackTrace();
> }
> System.out.println("sleep end");
> for (TopicPartition p : partitions) {
>   System.out.println(p + " offset: " + kc.position(p));
> }
> System.out.println("cannot reach here");
> kc.close();
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-4665) Inconsistent handling of non-existing topics in offset fetch handling

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4665:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Inconsistent handling of non-existing topics in offset fetch handling
> -
>
> Key: KAFKA-4665
> URL: https://issues.apache.org/jira/browse/KAFKA-4665
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, core
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 1.1.0
>
>
> For version 0 of the offset fetch API, the broker returns 
> UNKNOWN_TOPIC_OR_PARTITION for any topics/partitions which do not exist at 
> the time of fetching. In later versions, we skip this check. We do, however, 
> continue to return UNKNOWN_TOPIC_OR_PARTITION for authorization errors (i.e. 
> if the principal does not have Describe access to the corresponding topic). 
> We should probably make this behavior consistent across versions.
> Note also that currently the consumer raises {{KafkaException}} when it 
> encounters an UNKNOWN_TOPIC_OR_PARTITION error in the offset fetch response, 
> which is inconsistent with how we usually handle this error. This probably 
> doesn't cause any problems currently only because of the inconsistency 
> mentioned in the first paragraph above.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4201) Add an --assignment-strategy option to new-consumer-based Mirror Maker

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189599#comment-16189599
 ] 

Ismael Juma commented on KAFKA-4201:


Was a KIP done for this?

> Add an --assignment-strategy option to new-consumer-based Mirror Maker
> --
>
> Key: KAFKA-4201
> URL: https://issues.apache.org/jira/browse/KAFKA-4201
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
> Fix For: 1.0.0
>
>
> The default assignment strategy in mirror maker will be changed from range to 
> round robin in an upcoming release ([see 
> KAFKA-3818|https://issues.apache.org/jira/browse/KAFKA-3818]). In order to 
> make it easier for users to change the assignment strategy, add an 
> {{--assignment-strategy}} option to Mirror Maker command line tool.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-6005) Reject JoinGroup request from first member with empty protocol type/protocol list

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189591#comment-16189591
 ] 

Ismael Juma commented on KAFKA-6005:


[~hachikuji], is this something that we need to do for 1.0.0 or can it wait 
until 1.1.0?

> Reject JoinGroup request from first member with empty protocol type/protocol 
> list
> -
>
> Key: KAFKA-6005
> URL: https://issues.apache.org/jira/browse/KAFKA-6005
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Minor
> Fix For: 1.0.0
>
>
> Currently, if the first group member joins with empty 
> partition.assignment.strategy, then the group won't allow any other members 
> with valid protocols.  This JIRA is to add validation to reject JoinGroup 
> requests from the first member with empty protocol type or empty protocol list
> Also, add consumer-side validations to check at least one partition assigner 
> class name is configured while using subscribe APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4972) Kafka 0.10.0 Found a corrupted index file during Kafka broker startup

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189588#comment-16189588
 ] 

Ismael Juma commented on KAFKA-4972:


I am not sure if it's the root cause, but it's a good observation that locking 
seems to be missing. We should fix this for 1.0.0.

> Kafka 0.10.0  Found a corrupted index file during Kafka broker startup
> --
>
> Key: KAFKA-4972
> URL: https://issues.apache.org/jira/browse/KAFKA-4972
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
> Environment: JDK: HotSpot  x64  1.7.0_80
> Tag: 0.10.0
>Reporter: fangjinuo
>Priority: Critical
> Fix For: 1.0.0, 0.11.0.2
>
> Attachments: Snap3.png
>
>
> -deleted text-After force shutdown all kafka brokers one by one, restart them 
> one by one, but a broker startup failure.
> The following WARN leval log was found in the log file:
> found a corrutped index file,  .index , delet it  ...
> you can view details by following attachment.
> ~I look up some codes in core module, found out :
> the nonthreadsafe method LogSegment.append(offset, messages)  has tow caller:
> 1) Log.append(messages)  // here has a synchronized 
> lock 
> 2) LogCleaner.cleanInto(topicAndPartition, source, dest, map, retainDeletes, 
> messageFormatVersion)   // here has not 
> So I guess this may be the reason for the repeated offset in 0xx.log file 
> (logsegment's .log) ~
> Although this is just my inference, but I hope that this problem can be 
> quickly repaired



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-4740) Using new consumer API with a Deserializer that throws SerializationException can lead to infinite loop

[Chang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189567#comment-16189567
 ] 

Chang Liu edited comment on KAFKA-4740 at 10/3/17 11:27 AM:


I am wondering what is the current solution?

Is following the way for future changes? Is there any issue on github I could 
follow in order to get update of this, Or this is it? 

{code:java}
try {
  records = consumer.poll(500);
  // process
} catch (SerializationException e) {
  consumer.seek(e.partition(), e.offset() + 1);
}
{code}

Thanks:)



was (Author: fluency03):
I am wondering what is the current solution?

Is following the way for future changes? Is there any issue I could follow in 
order to get update of this? 

{code:java}
try {
  records = consumer.poll(500);
  // process
} catch (SerializationException e) {
  consumer.seek(e.partition(), e.offset() + 1);
}
{code}

Thanks:)


> Using new consumer API with a Deserializer that throws SerializationException 
> can lead to infinite loop
> ---
>
> Key: KAFKA-4740
> URL: https://issues.apache.org/jira/browse/KAFKA-4740
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
> Environment: Kafka broker 0.10.1.1 (but this bug is not dependent on 
> the broker version)
> Kafka clients 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Sébastien Launay
>Assignee: Sébastien Launay
>Priority: Critical
>
> The old consumer supports deserializing records into typed objects and throws 
> a {{SerializationException}} through {{MessageAndMetadata#key()}} and 
> {{MessageAndMetadata#message()}} that can be catched by the client \[1\].
> When using the new consumer API with kafka-clients version < 0.10.0.1, such 
> the exception is swallowed by the {{NetworkClient}} class and result in an 
> infinite loop which the client has no control over like:
> {noformat}
> DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - Resetting offset 
> for partition test2-0 to earliest offset.
> DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - Fetched offset 0 
> for partition test2-0
> ERROR org.apache.kafka.clients.NetworkClient - Uncaught error in request 
> completion:
> org.apache.kafka.common.errors.SerializationException: Size of data received 
> by IntegerDeserializer is not 4
> ERROR org.apache.kafka.clients.NetworkClient - Uncaught error in request 
> completion:
> org.apache.kafka.common.errors.SerializationException: Size of data received 
> by IntegerDeserializer is not 4
> ...
> {noformat}
> Thanks to KAFKA-3977, this has been partially fixed in 0.10.1.0 but another 
> issue still remains.
> Indeed, the client can now catch the {{SerializationException}} but the next 
> call to {{Consumer#poll(long)}} will throw the same exception indefinitely.
> The following snippet (full example available on Github \[2\] for most 
> released kafka-clients versions):
> {code:java}
> try (KafkaConsumer kafkaConsumer = new 
> KafkaConsumer<>(consumerConfig, new StringDeserializer(), new 
> IntegerDeserializer())) {
> kafkaConsumer.subscribe(Arrays.asList("topic"));
> // Will run till the shutdown hook is called
> while (!doStop) {
> try {
> ConsumerRecords records = 
> kafkaConsumer.poll(1000);
> if (!records.isEmpty()) {
> logger.info("Got {} messages", records.count());
> for (ConsumerRecord record : records) {
> logger.info("Message with partition: {}, offset: {}, key: 
> {}, value: {}",
> record.partition(), record.offset(), record.key(), 
> record.value());
> }
> } else {
> logger.info("No messages to consume");
> }
> } catch (SerializationException e) {
> logger.warn("Failed polling some records", e);
> }
>  }
> }
> {code}
> when run with the following records (third record has an invalid Integer 
> value):
> {noformat}
> printf "\x00\x00\x00\x00\n" | bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic topic
> printf "\x00\x00\x00\x01\n" | bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic topic
> printf "\x00\x00\x00\n" | bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic topic
> printf "\x00\x00\x00\x02\n" | bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic topic
> {noformat}
> will produce the following logs:
> {noformat}
> INFO  consumer.Consumer - Got 2 messages
> INFO  consumer.Consumer - Message with partition: 0, 

[jira] [Commented] (KAFKA-4740) Using new consumer API with a Deserializer that throws SerializationException can lead to infinite loop

[Chang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189567#comment-16189567
 ] 

Chang Liu commented on KAFKA-4740:
--

I am wondering what is the current solution?

Is following the way for future changes? Is there any issue I could follow in 
order to get update of this? 

{code:java}
try {
  records = consumer.poll(500);
  // process
} catch (SerializationException e) {
  consumer.seek(e.partition(), e.offset() + 1);
}
{code}

Thanks:)


> Using new consumer API with a Deserializer that throws SerializationException 
> can lead to infinite loop
> ---
>
> Key: KAFKA-4740
> URL: https://issues.apache.org/jira/browse/KAFKA-4740
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
> Environment: Kafka broker 0.10.1.1 (but this bug is not dependent on 
> the broker version)
> Kafka clients 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Sébastien Launay
>Assignee: Sébastien Launay
>Priority: Critical
>
> The old consumer supports deserializing records into typed objects and throws 
> a {{SerializationException}} through {{MessageAndMetadata#key()}} and 
> {{MessageAndMetadata#message()}} that can be catched by the client \[1\].
> When using the new consumer API with kafka-clients version < 0.10.0.1, such 
> the exception is swallowed by the {{NetworkClient}} class and result in an 
> infinite loop which the client has no control over like:
> {noformat}
> DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - Resetting offset 
> for partition test2-0 to earliest offset.
> DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - Fetched offset 0 
> for partition test2-0
> ERROR org.apache.kafka.clients.NetworkClient - Uncaught error in request 
> completion:
> org.apache.kafka.common.errors.SerializationException: Size of data received 
> by IntegerDeserializer is not 4
> ERROR org.apache.kafka.clients.NetworkClient - Uncaught error in request 
> completion:
> org.apache.kafka.common.errors.SerializationException: Size of data received 
> by IntegerDeserializer is not 4
> ...
> {noformat}
> Thanks to KAFKA-3977, this has been partially fixed in 0.10.1.0 but another 
> issue still remains.
> Indeed, the client can now catch the {{SerializationException}} but the next 
> call to {{Consumer#poll(long)}} will throw the same exception indefinitely.
> The following snippet (full example available on Github \[2\] for most 
> released kafka-clients versions):
> {code:java}
> try (KafkaConsumer kafkaConsumer = new 
> KafkaConsumer<>(consumerConfig, new StringDeserializer(), new 
> IntegerDeserializer())) {
> kafkaConsumer.subscribe(Arrays.asList("topic"));
> // Will run till the shutdown hook is called
> while (!doStop) {
> try {
> ConsumerRecords records = 
> kafkaConsumer.poll(1000);
> if (!records.isEmpty()) {
> logger.info("Got {} messages", records.count());
> for (ConsumerRecord record : records) {
> logger.info("Message with partition: {}, offset: {}, key: 
> {}, value: {}",
> record.partition(), record.offset(), record.key(), 
> record.value());
> }
> } else {
> logger.info("No messages to consume");
> }
> } catch (SerializationException e) {
> logger.warn("Failed polling some records", e);
> }
>  }
> }
> {code}
> when run with the following records (third record has an invalid Integer 
> value):
> {noformat}
> printf "\x00\x00\x00\x00\n" | bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic topic
> printf "\x00\x00\x00\x01\n" | bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic topic
> printf "\x00\x00\x00\n" | bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic topic
> printf "\x00\x00\x00\x02\n" | bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic topic
> {noformat}
> will produce the following logs:
> {noformat}
> INFO  consumer.Consumer - Got 2 messages
> INFO  consumer.Consumer - Message with partition: 0, offset: 0, key: null, 
> value: 0
> INFO  consumer.Consumer - Message with partition: 0, offset: 1, key: null, 
> value: 1
> WARN  consumer.Consumer - Failed polling some records
> org.apache.kafka.common.errors.SerializationException: Error deserializing 
> key/value for partition topic-0 at offset 2
> Caused by: org.apache.kafka.common.errors.SerializationException: Size of 
> data received by IntegerDeserializer is not 4
> 

[jira] [Commented] (KAFKA-5843) Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to true

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189524#comment-16189524
 ] 

ASF GitHub Bot commented on KAFKA-5843:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3797


> Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to 
> true
> ---
>
> Key: KAFKA-5843
> URL: https://issues.apache.org/jira/browse/KAFKA-5843
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-4946) Evaluation of kafka_mx4jenable is inverted

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4946.

Resolution: Duplicate

Marking as duplicate of KAFKA-5843 as that had a PR. Credited Ralph as well in 
the merged commit:

https://github.com/apache/kafka/commit/716330a5b2f629e6bb0316b5f036837660916a0e

> Evaluation of kafka_mx4jenable is inverted
> --
>
> Key: KAFKA-4946
> URL: https://issues.apache.org/jira/browse/KAFKA-4946
> Project: Kafka
>  Issue Type: Bug
>  Components: core, metrics
>Affects Versions: 0.10.2.0
>Reporter: Ralph Weires
>Priority: Minor
> Attachments: mx4j-enable.patch
>
>
> Mx4j stopped working for me after upgrade from 0.8.2 -> 0.10.2.0
> I had the "-Dkafka_mx4jenable=true" flag set in a custom start script, which 
> however now actually prevents MX4J from being loaded in 0.10.2.0
> The flag is only taken into account since the patch from KAFKA-2002 it seems 
> to me - we should however skip trying to load MX4J if the flag is set to 
> false, rather than if it's set to true (as it's currently done).
> Trivial patch attached, check should be negated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-6005) Reject JoinGroup request from first member with empty protocol type/protocol list

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189475#comment-16189475
 ] 

Manikumar commented on KAFKA-6005:
--

PR: https://github.com/apache/kafka/pull/3957

> Reject JoinGroup request from first member with empty protocol type/protocol 
> list
> -
>
> Key: KAFKA-6005
> URL: https://issues.apache.org/jira/browse/KAFKA-6005
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Minor
> Fix For: 1.0.0
>
>
> Currently, if the first group member joins with empty 
> partition.assignment.strategy, then the group won't allow any other members 
> with valid protocols.  This JIRA is to add validation to reject JoinGroup 
> requests from the first member with empty protocol type or empty protocol list
> Also, add consumer-side validations to check at least one partition assigner 
> class name is configured while using subscribe APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6005) Reject JoinGroup request from first member with empty protocol type/protocol list

[Manikumar (JIRA)]

Manikumar created KAFKA-6005:


 Summary: Reject JoinGroup request from first member with empty 
protocol type/protocol list
 Key: KAFKA-6005
 URL: https://issues.apache.org/jira/browse/KAFKA-6005
 Project: Kafka
  Issue Type: Bug
Reporter: Manikumar
Assignee: Manikumar
Priority: Minor
 Fix For: 1.0.0


Currently, if the first group member joins with empty 
partition.assignment.strategy, then the group won't allow any other members 
with valid protocols.  This JIRA is to add validation to reject JoinGroup 
requests from the first member with empty protocol type or empty protocol list

Also, add consumer-side validations to check at least one partition assigner 
class name is configured while using subscribe APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)