[jira] [Updated] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Bentley updated KAFKA-5692: --- Flags: Patch Labels: kip patch-available (was: kip) > Refactor PreferredReplicaLeaderElectionCommand to use AdminClient > - > > Key: KAFKA-5692 > URL: https://issues.apache.org/jira/browse/KAFKA-5692 > Project: Kafka > Issue Type: Improvement >Reporter: Tom Bentley >Assignee: Tom Bentley >Priority: Minor > Labels: kip, patch-available > Fix For: 1.0.0 > > > The PreferredReplicaLeaderElectionCommand currently uses a direct connection > to zookeeper. The zookeeper dependency should be deprecated and an > AdminClient API created to be used instead. > This change will require a KIP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5657) Connect REST API should include the connector type when describing a connector
[
https://issues.apache.org/jira/browse/KAFKA-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165653#comment-16165653
]
Ted Yu commented on KAFKA-5657:
---
>From PR:
bq. Who'd have thought that adding the connector type would get this
complicated?
Suggest dropping the newbie tag.
> Connect REST API should include the connector type when describing a connector
> --
>
> Key: KAFKA-5657
> URL: https://issues.apache.org/jira/browse/KAFKA-5657
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
> Labels: needs-kip, newbie
> Fix For: 1.0.0
>
> Attachments: 5657.v1.txt
>
>
> Kafka Connect's REST API's {{connectors/}} and {{connectors/\{name\}}}
> endpoints should include whether the connector is a source or a sink.
> See KAFKA-4343 and KIP-151 for the related modification of the
> {{connector-plugins}} endpoint.
> Also see KAFKA-4279 for converter-related endpoints.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5889) MetricsTest is flaky
Ted Yu created KAFKA-5889:
-
Summary: MetricsTest is flaky
Key: KAFKA-5889
URL: https://issues.apache.org/jira/browse/KAFKA-5889
Project: Kafka
Issue Type: Test
Reporter: Ted Yu
The following appeared in several recent builds (e.g.
https://builds.apache.org/job/kafka-trunk-jdk7/2758) :
{code}
kafka.metrics.MetricsTest > testMetricsLeak FAILED
java.lang.AssertionError: expected:<1216> but was:<1219>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
kafka.metrics.MetricsTest$$anonfun$testMetricsLeak$1.apply$mcVI$sp(MetricsTest.scala:68)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
at kafka.metrics.MetricsTest.testMetricsLeak(MetricsTest.scala:66)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5882) NullPointerException in StreamTask
[
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-5882:
---
Summary: NullPointerException in StreamTask (was: NullPointerException in
ConsumerCoordinator)
> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073]
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
> [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO StreamThread:1040 - stream-thread
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO KafkaProducer:972 - Closing the Kafka producer with
> timeoutMillis = 9223372036854775807 ms.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5301) Improve exception handling on consumer path
[ https://issues.apache.org/jira/browse/KAFKA-5301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165405#comment-16165405 ] ASF GitHub Bot commented on KAFKA-5301: --- GitHub user ConcurrencyPractitioner reopened a pull request: https://github.com/apache/kafka/pull/3842 KAFKA-5301 Improve exception handling on consumer path This is an improvised approach towards fixing @guozhangwang 's second issue. I have changed the method return type as well as override such that it returns exception. If the exception returned is not null (the default value), than we skip the callback. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ConcurrencyPractitioner/kafka trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3842.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3842 commit 6290df2070f215d0b355f3e59717d911e50b8973 Author: Richard Yu Date: 2017-09-13T03:19:24Z [Kafka-5301] Improve exception handling on consumer path commit 3a01de2d4e293d15da5c390bc5179243bbdb833e Author: Richard Yu Date: 2017-09-13T22:34:11Z Exception handling add-on > Improve exception handling on consumer path > --- > > Key: KAFKA-5301 > URL: https://issues.apache.org/jira/browse/KAFKA-5301 > Project: Kafka > Issue Type: Sub-task > Components: streams >Affects Versions: 0.11.0.0 >Reporter: Eno Thereska > Fix For: 1.0.0 > > Attachments: 5301.v1.patch > > > Used in StreamsThread.java, mostly to .poll() but also to restore data. > Used in StreamsTask.java, mostly to .pause(), .resume() > All exceptions here are currently caught all the way up to the main running > loop in a broad catch(Exception e) statement in StreamThread.run(). > One main concern on the consumer path is handling deserialization errors that > happen before streams has even had a chance to look at the data: > https://issues.apache.org/jira/browse/KAFKA-5157 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5880) Transactional producer and read committed consumer causes consumer to stuck
[
https://issues.apache.org/jira/browse/KAFKA-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165326#comment-16165326
]
Apurva Mehta commented on KAFKA-5880:
-
Thanks for the information, Lae. I will have a look soon.
> Transactional producer and read committed consumer causes consumer to stuck
> ---
>
> Key: KAFKA-5880
> URL: https://issues.apache.org/jira/browse/KAFKA-5880
> Project: Kafka
> Issue Type: Bug
>Reporter: Lae
> Attachments: index-updates-3.zip
>
>
> We use transactional producers, and have configured isolation level on the
> consumer to only read committed data. The consumer has somehow got into a
> stuck state where it can no longer move forward because the Kafka server
> always return empty list of records despite there are thousands more
> successful transactions after the offset.
> This is an example producer code:
> {code:java}
> Properties config = new Properties();
> config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
> config.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG,
> UUID.randomUUID().toString());
> config.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
> config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> try (Producer producer = new KafkaProducer<>(config)) {
> producer.initTransactions();
> try {
> producer.beginTransaction();
> // Multiple producer.send(...) here
> producer.commitTransaction();
> } catch (Throwable e) {
> producer.abortTransaction();
> }
> }
> {code}
> This is the test consumer code:
> {code:java}
> Properties config = new Properties();
> config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
> config.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
> config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
> config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
> config.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG,
> IsolationLevel.READ_COMMITTED.toString().toLowerCase(ENGLISH));
> config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
> StringDeserializer.class.getName());
> config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
> StringDeserializer.class.getName());
> try (KafkaConsumer consumer = new KafkaConsumer<>(config)) {
> consumer.subscribe(Collections.singleton("index-updates"));
> while (true) {
> ConsumerRecords records = consumer.poll(5000);
> for (ConsumerRecord record : records) {
> System.err.println(record.value());
> }
> consumer.commitSync();
> }
> }
> {code}
> I have also attached the problematic partition data index-updates-3.zip, to
> reproduce the issue using the data, you can run a local Kafka instance, then
> create a topic called "index-updates" with 10 partitions, and replace the
> content of the index-updates-3 log directory with the attached content, then
> running the above consumer code.
> Then the consumer will be stuck at some point (not always at the same offset)
> not making anymore progress even if you send new data into the partition
> (other partitions seem fine). The following example is when the consumer was
> stuck at offset 46644, and the Kafka server always return empty list of
> records when the consumer fetches from 46644:
> {noformat}
> root@0b1e67f0c34b:/# /opt/kafka/bin/kafka-consumer-groups.sh --describe
> --group my-group --bootstrap-server localhost:9092
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
>CONSUMER-ID HOST
> CLIENT-ID
> index-updates 0 15281 15281 0
>consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer-1
> index-updates 1 0 0 0
>consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer-1
> index-updates 2 0 0 0
>consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer-1
> index-updates 3 46644 65735
> 19091 consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer-1
> index-updates 4 0 0 0
>consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer
[jira] [Updated] (KAFKA-5888) Transactions system test should check for message order
[ https://issues.apache.org/jira/browse/KAFKA-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta updated KAFKA-5888: Issue Type: Sub-task (was: Bug) Parent: KAFKA-5795 > Transactions system test should check for message order > --- > > Key: KAFKA-5888 > URL: https://issues.apache.org/jira/browse/KAFKA-5888 > Project: Kafka > Issue Type: Sub-task >Reporter: Apurva Mehta >Assignee: Apurva Mehta > Labels: exactly-once > Fix For: 1.0.0 > > > Currently, the transactions system test doesn't check for correct ordering of > the messages in a transaction. With KAFKA-5494, we can have multiple inflight > requests for a single transaction, which could yield to out of order messages > in the log if there are bugs. So we should assert that order is maintained in > our system tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5888) Transactions system test should check for message order
[ https://issues.apache.org/jira/browse/KAFKA-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta updated KAFKA-5888: Issue Type: Bug (was: Sub-task) Parent: (was: KAFKA-5870) > Transactions system test should check for message order > --- > > Key: KAFKA-5888 > URL: https://issues.apache.org/jira/browse/KAFKA-5888 > Project: Kafka > Issue Type: Bug >Reporter: Apurva Mehta >Assignee: Apurva Mehta > Labels: exactly-once > Fix For: 1.0.0 > > > Currently, the transactions system test doesn't check for correct ordering of > the messages in a transaction. With KAFKA-5494, we can have multiple inflight > requests for a single transaction, which could yield to out of order messages > in the log if there are bugs. So we should assert that order is maintained in > our system tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5888) Transactions system test should check for message order
[ https://issues.apache.org/jira/browse/KAFKA-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta updated KAFKA-5888: Issue Type: Sub-task (was: Bug) Parent: KAFKA-5870 > Transactions system test should check for message order > --- > > Key: KAFKA-5888 > URL: https://issues.apache.org/jira/browse/KAFKA-5888 > Project: Kafka > Issue Type: Sub-task >Reporter: Apurva Mehta >Assignee: Apurva Mehta > Labels: exactly-once > Fix For: 1.0.0 > > > Currently, the transactions system test doesn't check for correct ordering of > the messages in a transaction. With KAFKA-5494, we can have multiple inflight > requests for a single transaction, which could yield to out of order messages > in the log if there are bugs. So we should assert that order is maintained in > our system tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5888) Transactions system test should check for message order
[ https://issues.apache.org/jira/browse/KAFKA-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta updated KAFKA-5888: Labels: exactly-once (was: ) > Transactions system test should check for message order > --- > > Key: KAFKA-5888 > URL: https://issues.apache.org/jira/browse/KAFKA-5888 > Project: Kafka > Issue Type: Bug >Reporter: Apurva Mehta >Assignee: Apurva Mehta > Labels: exactly-once > Fix For: 1.0.0 > > > Currently, the transactions system test doesn't check for correct ordering of > the messages in a transaction. With KAFKA-5494, we can have multiple inflight > requests for a single transaction, which could yield to out of order messages > in the log if there are bugs. So we should assert that order is maintained in > our system tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5888) Transactions system test should check for message order
Apurva Mehta created KAFKA-5888: --- Summary: Transactions system test should check for message order Key: KAFKA-5888 URL: https://issues.apache.org/jira/browse/KAFKA-5888 Project: Kafka Issue Type: Bug Reporter: Apurva Mehta Assignee: Apurva Mehta Fix For: 1.0.0 Currently, the transactions system test doesn't check for correct ordering of the messages in a transaction. With KAFKA-5494, we can have multiple inflight requests for a single transaction, which could yield to out of order messages in the log if there are bugs. So we should assert that order is maintained in our system tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5880) Transactional producer and read committed consumer causes consumer to stuck
[
https://issues.apache.org/jira/browse/KAFKA-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165252#comment-16165252
]
Lae edited comment on KAFKA-5880 at 9/13/17 8:40 PM:
-
This is the log from the consumer, it's repeatedly logging the following:
{noformat}
08:31:02.412 TRACE org.apache.kafka.clients.NetworkClient Completed receive
from node 0, for key 1, received
{throttle_time_ms=0,responses=[{topic=index-updates,partition_responses=[{partition_header={partition=0,error_code=0,high_watermark=6,last_stable_offset=6,log_start_offset=6,aborted_transactions=[]},record_set=[]},{partition_header={partition=9,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=7,error_code=0,high_watermark=2,last_stable_offset=2,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=8,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=5,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=6,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=3,error_code=0,high_watermark=65737,last_stable_offset=46644,log_start_offset=26888,aborted_transactions=[]},record_set=[]},{partition_header={partition=4,error_code=0,high_watermark=1,last_stable_offset=1,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=1,error_code=0,high_watermark=255,last_stable_offset=255,log_start_offset=254,aborted_transactions=[]},record_set=[]},{partition_header={partition=2,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]}]}]}
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 6 for partition index-updates-0 returned fetch data
(error=NONE, highWaterMark=6, lastStableOffset = 6, logStartOffset = 6,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-9 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 2 for partition index-updates-7 returned fetch data
(error=NONE, highWaterMark=2, lastStableOffset = 2, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-8 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-5 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-6 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 46644 for partition index-updates-3 returned fetch
data (error=NONE, highWaterMark=65737, lastStableOffset = 46644, logStartOffset
= 26888, abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 1 for partition index-updates-4 returned fetch data
(error=NONE, highWaterMark=1, lastStableOffset = 1, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 255 for partition index-updates-1 returned fetch data
(error=NONE, highWaterMark=255, lastStableOffset = 255, logStartOffset = 254,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-2 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 TRACE org.apache.kafka.clients.consumer.internals.Fetcher
Preparing to read 0 bytes of data for partition index-updates-0 with offset 6
08:31:02.412 TRACE org.apache.kafka.clients.consumer.internals.Fetcher
[jira] [Commented] (KAFKA-5880) Transactional producer and read committed consumer causes consumer to stuck
[
https://issues.apache.org/jira/browse/KAFKA-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165254#comment-16165254
]
Lae commented on KAFKA-5880:
Another strange observation is that if I restart the Kafka server, the consumer
will be able to move forward a bit and process some messages once it's
reconnected, then it will get stuck again.
> Transactional producer and read committed consumer causes consumer to stuck
> ---
>
> Key: KAFKA-5880
> URL: https://issues.apache.org/jira/browse/KAFKA-5880
> Project: Kafka
> Issue Type: Bug
>Reporter: Lae
> Attachments: index-updates-3.zip
>
>
> We use transactional producers, and have configured isolation level on the
> consumer to only read committed data. The consumer has somehow got into a
> stuck state where it can no longer move forward because the Kafka server
> always return empty list of records despite there are thousands more
> successful transactions after the offset.
> This is an example producer code:
> {code:java}
> Properties config = new Properties();
> config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
> config.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG,
> UUID.randomUUID().toString());
> config.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
> config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> try (Producer producer = new KafkaProducer<>(config)) {
> producer.initTransactions();
> try {
> producer.beginTransaction();
> // Multiple producer.send(...) here
> producer.commitTransaction();
> } catch (Throwable e) {
> producer.abortTransaction();
> }
> }
> {code}
> This is the test consumer code:
> {code:java}
> Properties config = new Properties();
> config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
> config.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
> config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
> config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
> config.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG,
> IsolationLevel.READ_COMMITTED.toString().toLowerCase(ENGLISH));
> config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
> StringDeserializer.class.getName());
> config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
> StringDeserializer.class.getName());
> try (KafkaConsumer consumer = new KafkaConsumer<>(config)) {
> consumer.subscribe(Collections.singleton("index-updates"));
> while (true) {
> ConsumerRecords records = consumer.poll(5000);
> for (ConsumerRecord record : records) {
> System.err.println(record.value());
> }
> consumer.commitSync();
> }
> }
> {code}
> I have also attached the problematic partition data index-updates-3.zip, to
> reproduce the issue using the data, you can run a local Kafka instance, then
> create a topic called "index-updates" with 10 partitions, and replace the
> content of the index-updates-3 log directory with the attached content, then
> running the above consumer code.
> Then the consumer will be stuck at some point (not always at the same offset)
> not making anymore progress even if you send new data into the partition
> (other partitions seem fine). The following example is when the consumer was
> stuck at offset 46644, and the Kafka server always return empty list of
> records when the consumer fetches from 46644:
> {noformat}
> root@0b1e67f0c34b:/# /opt/kafka/bin/kafka-consumer-groups.sh --describe
> --group my-group --bootstrap-server localhost:9092
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
>CONSUMER-ID HOST
> CLIENT-ID
> index-updates 0 15281 15281 0
>consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer-1
> index-updates 1 0 0 0
>consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer-1
> index-updates 2 0 0 0
>consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer-1
> index-updates 3 46644 65735
> 19091 consumer-1-c8f3fede-ef6f-4d12-a426-44aa9c47e71f /10.100.1.97
> consumer-1
> index-updates 4 0 0
[jira] [Commented] (KAFKA-5880) Transactional producer and read committed consumer causes consumer to stuck
[
https://issues.apache.org/jira/browse/KAFKA-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165252#comment-16165252
]
Lae commented on KAFKA-5880:
This is the log from the consumer:
{noformat}
08:31:02.412 TRACE org.apache.kafka.clients.NetworkClient Completed receive
from node 0, for key 1, received
{throttle_time_ms=0,responses=[{topic=index-updates,partition_responses=[{partition_header={partition=0,error_code=0,high_watermark=6,last_stable_offset=6,log_start_offset=6,aborted_transactions=[]},record_set=[]},{partition_header={partition=9,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=7,error_code=0,high_watermark=2,last_stable_offset=2,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=8,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=5,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=6,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=3,error_code=0,high_watermark=65737,last_stable_offset=46644,log_start_offset=26888,aborted_transactions=[]},record_set=[]},{partition_header={partition=4,error_code=0,high_watermark=1,last_stable_offset=1,log_start_offset=0,aborted_transactions=[]},record_set=[]},{partition_header={partition=1,error_code=0,high_watermark=255,last_stable_offset=255,log_start_offset=254,aborted_transactions=[]},record_set=[]},{partition_header={partition=2,error_code=0,high_watermark=0,last_stable_offset=0,log_start_offset=0,aborted_transactions=[]},record_set=[]}]}]}
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 6 for partition index-updates-0 returned fetch data
(error=NONE, highWaterMark=6, lastStableOffset = 6, logStartOffset = 6,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-9 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 2 for partition index-updates-7 returned fetch data
(error=NONE, highWaterMark=2, lastStableOffset = 2, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-8 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-5 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-6 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 46644 for partition index-updates-3 returned fetch
data (error=NONE, highWaterMark=65737, lastStableOffset = 46644, logStartOffset
= 26888, abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 1 for partition index-updates-4 returned fetch data
(error=NONE, highWaterMark=1, lastStableOffset = 1, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 255 for partition index-updates-1 returned fetch data
(error=NONE, highWaterMark=255, lastStableOffset = 255, logStartOffset = 254,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 DEBUG org.apache.kafka.clients.consumer.internals.Fetcher Fetch
READ_COMMITTED at offset 0 for partition index-updates-2 returned fetch data
(error=NONE, highWaterMark=0, lastStableOffset = 0, logStartOffset = 0,
abortedTransactions = [], recordsSizeInBytes=0)
08:31:02.412 TRACE org.apache.kafka.clients.consumer.internals.Fetcher
Preparing to read 0 bytes of data for partition index-updates-0 with offset 6
08:31:02.412 TRACE org.apache.kafka.clients.consumer.internals.Fetcher Updating
high watermark for partition index-updates-0 to 6
08:31:02.412 TRACE org.apac
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165226#comment-16165226 ] Guozhang Wang commented on KAFKA-4392: -- Could you try out 0.11.0.1, which is just released today? > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_9 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_1 > drwxr-xr-x 3 root root 58 Nov 2 15:43
[jira] [Commented] (KAFKA-5885) NPE in ZKClient
[
https://issues.apache.org/jira/browse/KAFKA-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165166#comment-16165166
]
Mickael Maison commented on KAFKA-5885:
---
We hit exactly the same issue a few weeks ago with 0.10.2.1 as well. In our
case, a few (~20) topics were affected.
> NPE in ZKClient
> ---
>
> Key: KAFKA-5885
> URL: https://issues.apache.org/jira/browse/KAFKA-5885
> Project: Kafka
> Issue Type: Bug
> Components: zkclient
>Affects Versions: 0.10.2.1
>Reporter: Dustin Cote
>
> A null znode for a topic (reason how this happen isn't totally clear, but not
> the focus of this issue) can currently cause controller leader election to
> fail. When looking at the broker logging, you can see there is a
> NullPointerException emanating from the ZKClient:
> {code}
> [2017-09-11 00:00:21,441] ERROR Error while electing or becoming leader on
> broker 1010674 (kafka.server.ZookeeperLeaderElector)
> kafka.common.KafkaException: Can't parse json string: null
> at kafka.utils.Json$.liftedTree1$1(Json.scala:40)
> at kafka.utils.Json$.parseFull(Json.scala:36)
> at
> kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:704)
> at
> kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:700)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.utils.ZkUtils.getReplicaAssignmentForTopics(ZkUtils.scala:700)
> at
> kafka.controller.KafkaController.initializeControllerContext(KafkaController.scala:742)
> at
> kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:333)
> at
> kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:160)
> at
> kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:85)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:154)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:154)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:154)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:153)
> at org.I0Itec.zkclient.ZkClient$9.run(ZkClient.java:825)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:72)
> Caused by: java.lang.NullPointerException
> {code}
> Regardless of how a null topic znode ended up in ZooKeeper, we can probably
> handle this better, at least by printing the path up to the problematic znode
> in the log. The way this particular problem was resolved was by using the
> ``kafka-topics`` command and seeing it persistently fail trying to read a
> particular topic with this same message. Then deleting the null znode allowed
> the leader election to complete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5882) NullPointerException in ConsumerCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165085#comment-16165085
]
Matthias J. Sax commented on KAFKA-5882:
I did have a look into the code, and don't understand how a NPE could occur
here. It this issue reproducible? Do you have any logs?
Btw: it seems to be unrelated to KAFKA-5073 from my current understanding.
> NullPointerException in ConsumerCoordinator
> ---
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073]
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
> ~[myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
> [myapp-streamer.jar:?]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
> [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO StreamThread:1040 - stream-thread
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO KafkaProducer:972 - Closing the Kafka producer with
> timeoutMillis = 9223372036854775807 ms.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5887) Enable findBugs (or equivalent) when building with Java 9
[ https://issues.apache.org/jira/browse/KAFKA-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5887: --- Description: findBugs doesn't support Java 9 and it seems to be abandonware at this point: https://github.com/findbugsproject/findbugs/issues/105 https://github.com/gradle/gradle/issues/720 It has been forked, but the fork requires Java 8: https://github.com/spotbugs/spotbugs https://github.com/spotbugs/spotbugs/blob/master/docs/migration.rst#findbugs-gradle-plugin We should migrate once we move to Java 8 if spotbugs is still being actively developed and findBugs continues to be dead. Additional tasks: 1. Remove the code that disables the Gradle plugin for findBugs (or spotbugs) when building with Java 9. 2. Enable the findBugs plugin in Jenkins for the relevant builds: https://builds.apache.org/job/kafka-trunk-jdk9/ https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/ was: findBugs doesn't support Java 9 and it seems to be abandonware at this point: https://github.com/findbugsproject/findbugs/issues/105 https://github.com/gradle/gradle/issues/720 It has been forked, but the fork requires Java 8: https://github.com/spotbugs/spotbugs https://github.com/spotbugs/spotbugs/blob/master/docs/migration.rst#findbugs-gradle-plugin We should migrate once we move to Java 8 if spotbugs is still being actively developed and findBugs continues to be dead. Additional tasks: 1. Remove the code that disables the Gradle plugin for findBugs (or spotbugs) when building with Java 9. 2. Enable the findBugs plugin in Jenkins for the relevant builds. > Enable findBugs (or equivalent) when building with Java 9 > - > > Key: KAFKA-5887 > URL: https://issues.apache.org/jira/browse/KAFKA-5887 > Project: Kafka > Issue Type: Task >Reporter: Ismael Juma > Fix For: 2.0.0 > > > findBugs doesn't support Java 9 and it seems to be abandonware at this point: > https://github.com/findbugsproject/findbugs/issues/105 > https://github.com/gradle/gradle/issues/720 > It has been forked, but the fork requires Java 8: > https://github.com/spotbugs/spotbugs > https://github.com/spotbugs/spotbugs/blob/master/docs/migration.rst#findbugs-gradle-plugin > We should migrate once we move to Java 8 if spotbugs is still being actively > developed and findBugs continues to be dead. > Additional tasks: > 1. Remove the code that disables the Gradle plugin for findBugs (or spotbugs) > when building with Java 9. > 2. Enable the findBugs plugin in Jenkins for the relevant builds: > https://builds.apache.org/job/kafka-trunk-jdk9/ > https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5887) Enable findBugs (or equivalent) when building with Java 9
Ismael Juma created KAFKA-5887: -- Summary: Enable findBugs (or equivalent) when building with Java 9 Key: KAFKA-5887 URL: https://issues.apache.org/jira/browse/KAFKA-5887 Project: Kafka Issue Type: Task Reporter: Ismael Juma Fix For: 2.0.0 findBugs doesn't support Java 9 and it seems to be abandonware at this point: https://github.com/findbugsproject/findbugs/issues/105 https://github.com/gradle/gradle/issues/720 It has been forked, but the fork requires Java 8: https://github.com/spotbugs/spotbugs https://github.com/spotbugs/spotbugs/blob/master/docs/migration.rst#findbugs-gradle-plugin We should migrate once we move to Java 8 if spotbugs is still being actively developed and findBugs continues to be dead. Additional tasks: 1. Remove the code that disables the Gradle plugin for findBugs (or spotbugs) when building with Java 9. 2. Enable the findBugs plugin in Jenkins for the relevant builds. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5886) Implement KIP-91
Sumant Tambe created KAFKA-5886: --- Summary: Implement KIP-91 Key: KAFKA-5886 URL: https://issues.apache.org/jira/browse/KAFKA-5886 Project: Kafka Issue Type: Improvement Components: producer Reporter: Sumant Tambe Assignee: Sumant Tambe Fix For: 1.0.0 Implement [KIP-91|https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-4501) Support Java 9
[ https://issues.apache.org/jira/browse/KAFKA-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164988#comment-16164988 ] ASF GitHub Bot commented on KAFKA-4501: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3845 > Support Java 9 > -- > > Key: KAFKA-4501 > URL: https://issues.apache.org/jira/browse/KAFKA-4501 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 1.0.0 > > > Java 9 is scheduled to be released in September 2017. We should support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164874#comment-16164874 ] ASF GitHub Bot commented on KAFKA-5692: --- GitHub user tombentley opened a pull request: https://github.com/apache/kafka/pull/3848 KAFKA-5692: Change PreferredReplicaLeaderElectionCommand to use Admin… …Client See also KIP-183. The contribution is my original work and I license the work to the project under the project's open source license. This implements the following algorithm: 1. AdminClient sends ElectPreferredLeadersRequest. 2. KafakApis receives ElectPreferredLeadersRequest and delegates to ReplicaManager.electPreferredLeaders() 3. ReplicaManager delegates to KafkaController.electPreferredLeaders() 4. KafkaController adds a PreferredReplicaLeaderElection to the EventManager, 5. ReplicaManager.electPreferredLeaders()'s callback uses the delayedElectPreferredReplicasPurgatory to wait for the results of the election to appear in the metadata cache. If there are no results because of errors, or because the preferred leaders are already leading the partitions then a response is returned immediately. In the EventManager work thread the preferred leader is elected as follows: 1. The EventManager runs PreferredReplicaLeaderElection.process() 2. process() calls KafkaController.onPreferredReplicaElectionWithResults() 3. KafkaController.onPreferredReplicaElectionWithResults() calls the PartitionStateMachine.handleStateChangesWithResults() to perform the election (asynchronously the PSM will send LeaderAndIsrRequest to the new and old leaders and UpdateMetadataRequest to all brokers) then invokes the callback. Note: the change in parameter type for CollectionUtils.groupDataByTopic(). This makes sense because the AdminClient APIs use Collection consistently, rather than List or Set. If binary compatiblity is a consideration the old version should be kept, delegating to the new version. I had to add PartitionStateMachine.handleStateChangesWithResults() in order to be able to process a set of state changes in the PartitionStateMachine *and get back individual results*. At the same time I noticed that all callers of existing handleStateChange() were destructuring a TopicAndPartition that they already had in order to call handleStateChange(), and that handleStateChange() immediately instantiated a new TopicAndPartition. Since TopicAndPartition is immutable this is pointless, so I refactored it. handleStateChange() also now returns any exception it caught, which is necessary for handleStateChangesWithResults() You can merge this pull request into a Git repository by running: $ git pull https://github.com/tombentley/kafka KAFKA-5692-elect-preferred Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3848.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3848 commit 6b9bf178049e1eedfb5f07771cc3c595c02484d9 Author: Tom Bentley Date: 2017-09-06T14:39:24Z KAFKA-5692: Change PreferredReplicaLeaderElectionCommand to use AdminClient See also KIP-183. This implements the following algorithm: 1. AdminClient sends ElectPreferredLeadersRequest. 2. KafakApis receives ElectPreferredLeadersRequest and delegates to ReplicaManager.electPreferredLeaders() 3. ReplicaManager delegates to KafkaController.electPreferredLeaders() 4. KafkaController adds a PreferredReplicaLeaderElection to the EventManager, 5. ReplicaManager.electPreferredLeaders()'s callback uses the delayedElectPreferredReplicasPurgatory to wait for the results of the election to appear in the metadata cache. If there are no results because of errors, or because the preferred leaders are already leading the partitions then a response is returned immediately. In the EventManager work thread the preferred leader is elected as follows: 1. The EventManager runs PreferredReplicaLeaderElection.process() 2. process() calls KafkaController.onPreferredReplicaElectionWithResults() 3. KafkaController.onPreferredReplicaElectionWithResults() calls the PartitionStateMachine.handleStateChangesWithResults() to perform the election (asynchronously the PSM will send LeaderAndIsrRequest to the new and old leaders and UpdateMetadataRequest to all brokers) then invokes the callback. Note: the change in parameter type for CollectionUtils.groupDataByTopic(). This makes sense because the AdminClient APIs use Collection consistently, rather
[jira] [Updated] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Bentley updated KAFKA-5692: --- Fix Version/s: 1.0.0 > Refactor PreferredReplicaLeaderElectionCommand to use AdminClient > - > > Key: KAFKA-5692 > URL: https://issues.apache.org/jira/browse/KAFKA-5692 > Project: Kafka > Issue Type: Improvement >Reporter: Tom Bentley >Assignee: Tom Bentley >Priority: Minor > Labels: kip > Fix For: 1.0.0 > > > The PreferredReplicaLeaderElectionCommand currently uses a direct connection > to zookeeper. The zookeeper dependency should be deprecated and an > AdminClient API created to be used instead. > This change will require a KIP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5301) Improve exception handling on consumer path
[ https://issues.apache.org/jira/browse/KAFKA-5301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164823#comment-16164823 ] ASF GitHub Bot commented on KAFKA-5301: --- Github user ConcurrencyPractitioner closed the pull request at: https://github.com/apache/kafka/pull/3842 > Improve exception handling on consumer path > --- > > Key: KAFKA-5301 > URL: https://issues.apache.org/jira/browse/KAFKA-5301 > Project: Kafka > Issue Type: Sub-task > Components: streams >Affects Versions: 0.11.0.0 >Reporter: Eno Thereska > Fix For: 1.0.0 > > Attachments: 5301.v1.patch > > > Used in StreamsThread.java, mostly to .poll() but also to restore data. > Used in StreamsTask.java, mostly to .pause(), .resume() > All exceptions here are currently caught all the way up to the main running > loop in a broad catch(Exception e) statement in StreamThread.run(). > One main concern on the consumer path is handling deserialization errors that > happen before streams has even had a chance to look at the data: > https://issues.apache.org/jira/browse/KAFKA-5157 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5885) NPE in ZKClient
Dustin Cote created KAFKA-5885:
--
Summary: NPE in ZKClient
Key: KAFKA-5885
URL: https://issues.apache.org/jira/browse/KAFKA-5885
Project: Kafka
Issue Type: Bug
Components: zkclient
Affects Versions: 0.10.2.1
Reporter: Dustin Cote
A null znode for a topic (reason how this happen isn't totally clear, but not
the focus of this issue) can currently cause controller leader election to
fail. When looking at the broker logging, you can see there is a
NullPointerException emanating from the ZKClient:
{code}
[2017-09-11 00:00:21,441] ERROR Error while electing or becoming leader on
broker 1010674 (kafka.server.ZookeeperLeaderElector)
kafka.common.KafkaException: Can't parse json string: null
at kafka.utils.Json$.liftedTree1$1(Json.scala:40)
at kafka.utils.Json$.parseFull(Json.scala:36)
at
kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:704)
at
kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:700)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at kafka.utils.ZkUtils.getReplicaAssignmentForTopics(ZkUtils.scala:700)
at
kafka.controller.KafkaController.initializeControllerContext(KafkaController.scala:742)
at
kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:333)
at
kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:160)
at
kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:85)
at
kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:154)
at
kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:154)
at
kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:154)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
at
kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:153)
at org.I0Itec.zkclient.ZkClient$9.run(ZkClient.java:825)
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:72)
Caused by: java.lang.NullPointerException
{code}
Regardless of how a null topic znode ended up in ZooKeeper, we can probably
handle this better, at least by printing the path up to the problematic znode
in the log. The way this particular problem was resolved was by using the
``kafka-topics`` command and seeing it persistently fail trying to read a
particular topic with this same message. Then deleting the null znode allowed
the leader election to complete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4501) Support Java 9
[ https://issues.apache.org/jira/browse/KAFKA-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4501: --- Description: Java 9 is scheduled to be released in September 2017. We should support it. (was: Java 9 is scheduled to be released in July 2017. We should support it. The new module system enforces access control and things like `setAccessible` cannot, by default, be used to circumvent access control in other modules. There are command-line flags available to disable the behaviour on a module by module basis. Right now, Gradle fails with the latest Java 9 snapshot and Scala 2.12.1 is required if building with Java 9. So we are blocked until the Gradle issues are fixed. I set the "Fix version" to 0.10.2.0, but it's likely to happen for the release after that.) > Support Java 9 > -- > > Key: KAFKA-4501 > URL: https://issues.apache.org/jira/browse/KAFKA-4501 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 1.0.0 > > > Java 9 is scheduled to be released in September 2017. We should support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5884) Enable PowerMock tests when running on Java 9
[
https://issues.apache.org/jira/browse/KAFKA-5884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-5884:
---
Fix Version/s: 1.0.1
> Enable PowerMock tests when running on Java 9
> -
>
> Key: KAFKA-5884
> URL: https://issues.apache.org/jira/browse/KAFKA-5884
> Project: Kafka
> Issue Type: Task
>Reporter: Ismael Juma
> Fix For: 1.1.0, 1.0.1
>
>
> PowerMock 2.0.0 will support Java 9. Once that is released, we should upgrade
> to it and remove the following code from build.gradle:
> {code}
> String[] testsToExclude = []
> if (JavaVersion.current().isJava9Compatible()) {
> testsToExclude = [
> "**/KafkaProducerTest.*", "**/BufferPoolTest.*",
> "**/SourceTaskOffsetCommitterTest.*", "**/WorkerSinkTaskTest.*",
> "**/WorkerSinkTaskThreadedTest.*",
> "**/WorkerSourceTaskTest.*", "**/WorkerTest.*",
> "**/DistributedHerderTest.*", "**/WorkerCoordinatorTest.*",
> "**/RestServerTest.*", "**/ConnectorPluginsResourceTest.*",
> "**/ConnectorsResourceTest.*",
> "**/StandaloneHerderTest.*", "**/FileOffsetBakingStoreTest.*",
> "**/KafkaConfigBackingStoreTest.*",
> "**/KafkaOffsetBackingStoreTest.*", "**/OffsetStorageWriterTest.*",
> "**/KafkaBasedLogTest.*"
> ]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5884) Enable PowerMock tests when running on Java 9
Ismael Juma created KAFKA-5884:
--
Summary: Enable PowerMock tests when running on Java 9
Key: KAFKA-5884
URL: https://issues.apache.org/jira/browse/KAFKA-5884
Project: Kafka
Issue Type: Task
Reporter: Ismael Juma
Fix For: 1.1.0
PowerMock 2.0.0 will support Java 9. Once that is released, we should upgrade
to it and remove the following code from build.gradle:
{code}
String[] testsToExclude = []
if (JavaVersion.current().isJava9Compatible()) {
testsToExclude = [
"**/KafkaProducerTest.*", "**/BufferPoolTest.*",
"**/SourceTaskOffsetCommitterTest.*", "**/WorkerSinkTaskTest.*",
"**/WorkerSinkTaskThreadedTest.*",
"**/WorkerSourceTaskTest.*", "**/WorkerTest.*",
"**/DistributedHerderTest.*", "**/WorkerCoordinatorTest.*",
"**/RestServerTest.*", "**/ConnectorPluginsResourceTest.*",
"**/ConnectorsResourceTest.*",
"**/StandaloneHerderTest.*", "**/FileOffsetBakingStoreTest.*",
"**/KafkaConfigBackingStoreTest.*",
"**/KafkaOffsetBackingStoreTest.*", "**/OffsetStorageWriterTest.*",
"**/KafkaBasedLogTest.*"
]
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5883) Run tests on Java 9 with –illegal-access=deny
[ https://issues.apache.org/jira/browse/KAFKA-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164635#comment-16164635 ] Ismael Juma commented on KAFKA-5883: A few changes that are likely to be required: 1. Accessing `sun.nio.ch.DirectBuffer` in `AbstractIndex`. A commit that shows how Lucene changed their code to run without warnings: https://github.com/apache/lucene-solr/commit/7e03427fa14a024ce257babcb8362d2451941e21 2. Accessing `sun.security.krb5.Config` for the `getDefaultRealm` method. 3. Upgrade to newer versions of EasyMock and PowerMock. > Run tests on Java 9 with –illegal-access=deny > - > > Key: KAFKA-5883 > URL: https://issues.apache.org/jira/browse/KAFKA-5883 > Project: Kafka > Issue Type: Task >Reporter: Ismael Juma > Fix For: 1.1.0 > > > The default was changed from –illegal-access=deny to –illegal-access=warn > late in the Java 9 cycle. By using the former, we will ensure that our code > is not relying on functionality that will be removed in a future Java version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5076) Remove usage of java.xml.bind.* classes hidden by default in JDK9
[ https://issues.apache.org/jira/browse/KAFKA-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5076. Resolution: Fixed Done as part of https://github.com/apache/kafka/pull/3647. > Remove usage of java.xml.bind.* classes hidden by default in JDK9 > - > > Key: KAFKA-5076 > URL: https://issues.apache.org/jira/browse/KAFKA-5076 > Project: Kafka > Issue Type: Improvement >Reporter: Xavier Léauté >Assignee: Ismael Juma >Priority: Minor > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KAFKA-5076) Remove usage of java.xml.bind.* classes hidden by default in JDK9
[ https://issues.apache.org/jira/browse/KAFKA-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-5076: -- Assignee: Ismael Juma (was: Xavier Léauté) > Remove usage of java.xml.bind.* classes hidden by default in JDK9 > - > > Key: KAFKA-5076 > URL: https://issues.apache.org/jira/browse/KAFKA-5076 > Project: Kafka > Issue Type: Improvement >Reporter: Xavier Léauté >Assignee: Ismael Juma >Priority: Minor > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5076) Remove usage of java.xml.bind.* classes hidden by default in JDK9
[ https://issues.apache.org/jira/browse/KAFKA-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5076: --- Fix Version/s: 1.0.0 > Remove usage of java.xml.bind.* classes hidden by default in JDK9 > - > > Key: KAFKA-5076 > URL: https://issues.apache.org/jira/browse/KAFKA-5076 > Project: Kafka > Issue Type: Improvement >Reporter: Xavier Léauté >Assignee: Xavier Léauté >Priority: Minor > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5883) Run tests on Java 9 with –illegal-access=deny
Ismael Juma created KAFKA-5883: -- Summary: Run tests on Java 9 with –illegal-access=deny Key: KAFKA-5883 URL: https://issues.apache.org/jira/browse/KAFKA-5883 Project: Kafka Issue Type: Task Reporter: Ismael Juma Fix For: 1.1.0 The default was changed from –illegal-access=deny to –illegal-access=warn late in the Java 9 cycle. By using the former, we will ensure that our code is not relying on functionality that will be removed in a future Java version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-3124) Update protocol wiki page to reflect latest request/response formats
[ https://issues.apache.org/jira/browse/KAFKA-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3124. -- Resolution: Fixed KAFKA-3361 added support for generating protocol documentation page to Kafka docs. > Update protocol wiki page to reflect latest request/response formats > > > Key: KAFKA-3124 > URL: https://issues.apache.org/jira/browse/KAFKA-3124 > Project: Kafka > Issue Type: Bug > Components: clients >Reporter: Joel Koshy > Labels: newbie > > The protocol wiki > (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol) > is slightly out of date. It does not have some of the newer request/response > formats. > We should actually figure out a way to _source_ the protocol definitions from > the last official release version into that wiki. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-1354) Failed to load class "org.slf4j.impl.StaticLoggerBinder"
[ https://issues.apache.org/jira/browse/KAFKA-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-1354. -- Resolution: Fixed This was fixed in 0.8.2 by adding slf4j-log4j12 binding to Kafka libs. > Failed to load class "org.slf4j.impl.StaticLoggerBinder" > > > Key: KAFKA-1354 > URL: https://issues.apache.org/jira/browse/KAFKA-1354 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: RHEL >Reporter: RakeshAcharya > Labels: newbie, patch, usability > Original Estimate: 672h > Remaining Estimate: 672h > > Getting below errors during Kafka startup > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > [2014-03-31 18:55:36,488] INFO Will not load MX4J, mx4j-tools.jar is not in > the classpath (kafka.utils.Mx4jLoader$) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-4501) Support Java 9
[ https://issues.apache.org/jira/browse/KAFKA-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164561#comment-16164561 ] ASF GitHub Bot commented on KAFKA-4501: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/3845 KAFKA-4501: Fix EasyMock and disable PowerMock tests under Java 9 - EasyMock 3.5 supports Java 9. - Removed unnecessary PowerMock dependency from 3 tests. - Disable remaining PowerMock tests when running with Java 9 until https://github.com/powermock/powermock/issues/783 is done. - Once we merge this PR, we can run tests on PRs with Java 9. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-4501-easymock-powermock-java-9 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3845.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3845 commit ab60da36358a792e1079aed25859828602c1a619 Author: Ismael Juma Date: 2017-08-21T13:51:00Z Use EasyMock instead of PowerMock in 3 connect tests commit 6e0f8c7d9c147d0340c711b6edd8cbadac344a31 Author: Ismael Juma Date: 2017-08-21T13:51:37Z Exclude PowerMock tests if running with Java 9 commit f076943217f522d6a5a516a559939dc1e7711b40 Author: Ismael Juma Date: 2017-09-13T12:02:35Z Upgrade easymock to 3.5 > Support Java 9 > -- > > Key: KAFKA-4501 > URL: https://issues.apache.org/jira/browse/KAFKA-4501 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 1.0.0 > > > Java 9 is scheduled to be released in July 2017. We should support it. > The new module system enforces access control and things like `setAccessible` > cannot, by default, be used to circumvent access control in other modules. > There are command-line flags available to disable the behaviour on a module > by module basis. > Right now, Gradle fails with the latest Java 9 snapshot and Scala 2.12.1 is > required if building with Java 9. So we are blocked until the Gradle issues > are fixed. > I set the "Fix version" to 0.10.2.0, but it's likely to happen for the > release after that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5177) Automatic creation of topic prompts warnings into console
[
https://issues.apache.org/jira/browse/KAFKA-5177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-5177.
--
Resolution: Won't Fix
As mentioned above, these are transient warning messages. Please reopen if you
think otherwise
> Automatic creation of topic prompts warnings into console
> -
>
> Key: KAFKA-5177
> URL: https://issues.apache.org/jira/browse/KAFKA-5177
> Project: Kafka
> Issue Type: Wish
>Affects Versions: 0.10.1.0
> Environment: Mac OSX 16.5.0 Darwin Kernel Version 16.5.0
> root:xnu-3789.51.2~3/RELEASE_X86_64 x86_64 & JDK 1.8.0_121
>Reporter: Pranas Baliuka
>Priority: Minor
>
> The quick guide https://kafka.apache.org/0101/documentation.html#quickstart
> Leaves the bad first impression at the step when test messages are appended:
> {code}
> kafka_2.11-0.10.1.0 pranas$ bin/kafka-topics.sh --list --zookeeper
> localhost:2181
> session-1
> kafka_2.11-0.10.1.0 pranas$ bin/kafka-console-producer.sh --broker-list
> localhost:9092 --topic test
> Message 1
> [2017-05-05 09:05:10,923] WARN Error while fetching metadata with correlation
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> Message 2
> Message 3
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KAFKA-5296) Unable to write to some partitions of newly created topic in 10.2
[ https://issues.apache.org/jira/browse/KAFKA-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-5296. -- Resolution: Fixed Closing as per above discussion. Multiple controllers can create these kind of issues. Controlled related issues are being addressed in Kafka Controller redesign jiras/KAFKA-5027 > Unable to write to some partitions of newly created topic in 10.2 > - > > Key: KAFKA-5296 > URL: https://issues.apache.org/jira/browse/KAFKA-5296 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.2.0 >Reporter: Abhisek Saikia > > We are using kafka 10.2 and the cluster was running fine for a month with 50 > topics and now we are having issue in producing message by creating new > topics. The create topic command is successful but producers are throwing > error while writing to some partitions. > Error in producer- > java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for > [topic1]-8: 30039 ms has passed since batch creation plus linger time > at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70) > ~[kafka-clients-0.10.2.0.jar:na] > at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57) > ~[kafka-clients-0.10.2.0.jar:na] > at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) > ~[kafka-clients-0.10.2.0.jar:na] > On the broker side, I don't see any topic-parition folder getting created for > the broker who is the leader for the partition. > While using 0.8 client, the write used to hang while it starts writing to the > partition having this issue. With 10.2 it resolved the the producer hang issue > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Attachment: (was: Tests.scala) > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Attachment: (was: log-output.txt) > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[
https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Worsley updated KAFKA-5825:
Attachment: log-output.txt
Tests.scala
Thanks for your detailed help [~guozhang].
I've made the change you suggested, but it hasn't worked, please see the new
log output that I've attached (I've also attached the latest version of
{{Tests.scala}}.)
I experimented with turning the replication factor down to 1 with;
{{transaction.state.log.replication.factor = 1}}
{{transaction.state.log.min.isr = 1}}
I also increased the test timeout to 5 minutes.
Unfortunately the effect is the same - however the log output is now
significantly smaller which may be helpful to you.
What can we try next? Is this a bug?
> Streams not processing when exactly once is set
> ---
>
> Key: KAFKA-5825
> URL: https://issues.apache.org/jira/browse/KAFKA-5825
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
> Environment: EmbeddedKafka running on Windows. Relevant files
> attached.
>Reporter: Ryan Worsley
> Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala
>
>
> +Set-up+
> I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/]
> for ScalaTest.
> This spins up a single broker internally on a random port.
> I've written two tests - the first without transactions, the second with.
> They're nearly identical apart from the config and the transactional
> semantics. I've written the transactional version based on Neha's
> [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/]
> which is the closest thing I could find to instructions.
> The tests wait until a single message is processed by the streams topology,
> they use this message to complete a promise that the test is waiting on.
> Once the promise completes the test verifies the value of the promise as
> being the expected value of the message.
> +Observed behaviour+
> The first test passes fine, the second test times out, the stream processor
> never seems to read the transactional message.
> +Notes+
> I've attached my build.sbt, log4j.properties and my Tests.scala file in order
> to make it as easy as possible for someone to re-create. I'm running on
> Windows and using Scala as this reflects my workplace. I completely expect
> there to be some configuration issue that's causing this, but am unable to
> proceed at this time.
> Related information:
> https://github.com/manub/scalatest-embedded-kafka/issues/82
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5060) Offset not found while broker is rebuilding its index after an index corruption
[
https://issues.apache.org/jira/browse/KAFKA-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164275#comment-16164275
]
Romaric Parmentier commented on KAFKA-5060:
---
Hi Spiros,
Very thank you for your answer, I really though it was the root cause of this
index corruption but after talking with our OPS team, it appears that our
systemd conf is ok:
{noformat}
:~$ systemctl show kafka.service | grep -i timeout
TimeoutStartUSec=infinity
TimeoutStopUSec=infinity
JobTimeoutUSec=infinity
JobTimeoutAction=none
{noformat}
Any idea ?
> Offset not found while broker is rebuilding its index after an index
> corruption
> ---
>
> Key: KAFKA-5060
> URL: https://issues.apache.org/jira/browse/KAFKA-5060
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 0.10.1.0
>Reporter: Romaric Parmentier
>Priority: Critical
>
> After rebooting our kafka servers to change a configuration, one of my
> consumers running old consumer has fail to find a new leader for a period of
> 15 minutes. The topic has a replication factor of 2.
> When the spare server has finally been found and elected leader, the previous
> consumed offset was not able to be found because the broker was rebuilding
> index.
> So my consumer has decided to follow the configuration auto.offset.reset
> which is pretty bad because the offset will exist 2 minutes later:
> 2017-04-12 14:59:08,568] WARN Found a corrupted index file due to requirement
> failed: Corrupt index found, index file
> (/var/lib/kafka/my_topic-6/130248110337.index) has non-zero size but
> the last offset is 130248110337 which is no larger than the base offset
> 130248110337.}. deleting
> /var/lib/kafka/my_topic-6/130248110337.timeindex,
> /var/lib/kafka/my_topic-6/130248110337.index and rebuilding index...
> (kafka.log.Log)
> [2017-04-12 15:01:41,490] INFO Completed load of log my_topic-6 with 6146 log
> segments and log end offset 130251895436 in 169696 ms (kafka.log.Log)
> Maybe it is handled by the new consumer or there is a some configuration to
> handle this case but I didn't find anything
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
