Re: Use self contained tokens instead of ACL

[Manikumar]

Hi,

In the first phase, we are trying implement the components/design discussed
in the KIP.
Yes, we can definitely improve some of the components to be more
extensible.
We are planning to implement in future KIPs/PRs.

Thanks


On Fri, Oct 27, 2017 at 8:22 PM, Sönke Liebau <
soenke.lie...@opencore.com.invalid> wrote:

> Hi Manikumar,
>
> I've looked over the KIP and had a quick look at the code in the PR as
> well. In principle I think this would help Peter along depending on how
> plugable some of the components are. Since Peter wants to generate Tokens
> not in Kafka but in an external System the entire part in Kafka of
> generating DelegationTokens would simply not be used, which I think would
> be fine. To validate externally generated tokens an option to substitute
> for example the TokenCache for a custom implementation or/and substitute
> the method of authenticating a delegation token for a custom class.
>
> Apologies for asking questions I could look up in the code myself, but at a
> first glance I haven't seen any indications of this token system being
> extendable, do you plan to allow extending the system to different external
> token providers? OAuth would come to mind as a fairly wide spread candidate
> that could probably be implemented fairly easily.
>
> Kind regards,
> Sönke
>
> On Fri, Oct 27, 2017 at 11:17 AM, Manikumar 
> wrote:
>
> > Hi,
> >
> > We have a accepted KIP for adding delegation token support for Kafka.
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 48+Delegation+token+support+for+Kafka
> >
> > currently the PR is under review process. Maybe this can used be as
> > starting point for your requirement.
> >
> > https://github.com/apache/kafka/pull/3616
> >
> >
> >
> > On Fri, Oct 27, 2017 at 2:34 PM, Sönke Liebau <
> > soenke.lie...@opencore.com.invalid> wrote:
> >
> > > Hi Peter,
> > >
> > > thanks for the explanation, it all makes sense now :)
> > >
> > > I can't say that I immediately see an easy way forward though to be
> > honest.
> > > The big issue, I think, is getting the token to Kafka (and hopefully
> > there
> > > is an easy way that I simply don't know of and someone will correct
> me) -
> > > implementing a custom principalbuilder and authorizer should be almost
> > > trivial.
> > >
> > > If transmitting the token as part of the ssl certificate or a Kerberos
> > > ticket is out though the air gets a bit thin if you don't want to
> > maintain
> > > your own fork of Kafka. The only potential solution that I can come up
> > with
> > > is to piggyback on SASL and provide your own LoginModule in Kafka's
> jaas
> > > file. If you use the SASL_SSL endpoint certificate checking should
> still
> > > have occured before the SASL handshake is initialized, so you
> > authenticated
> > > the user at that point. You could then use that handshake to transmit
> > your
> > > token, have your custom principalbuilder extract the topics from that
> and
> > > your custom authorizer authorize based on the extracted topicnames.
> > > A word of caution though: this is based on a few minutes looking at
> code
> > > and my dangerous half knowledge of SASL, so there are any number of
> > things
> > > that could make this impossible, either with SASL or in the Kafka
> > codebase
> > > itself. Might be a direction to explore though.
> > >
> > > Hopefully that makes sense and is targeted at least in the vicinity of
> > whet
> > > you are looking for?
> > >
> > > Kind regards,
> > > Sönke
> > >
> > > On Fri, Oct 27, 2017 at 9:33 AM, Postmann, P. (Peter) <
> > > peter.postm...@ing.com.invalid> wrote:
> > >
> > > > Hi Sönke,
> > > >
> > > > Thanks for your feedback, sorry that I didn’t gave you the whole
> > picture
> > > > in first place:
> > > >
> > > > We are using an Architecture, which tries to avoid to fetch or pull
> > > > anything from a 3rd party during runtime. Therefore we are using
> > > > self-contained tokens and client side load balancing with a micro
> > service
> > > > alike architecture.
> > > >
> > > > In this architecture we have two tokens:
> > > > - the manifest which enabled services to provide APIs
> > > > - the peer token which enables services to call APIs
> > > >
> > > > API providers publish their APIs in a Portal. API consumers subscribe
> > to
> > > > those APIs. The portal generates a manifest for the provider and a
> peer
> > > > token for the consumer. Both tokens contain a list of endpoints and
> are
> > > > signed by the portal. The tokens are valid for a certain amount of
> > time.
> > > >
> > > > Furthermore we use a Service Registry to discover those services. The
> > > flow
> > > > works as follows:
> > > >
> > > > - A service instance registers itself (endpoint --> IP:Port) at the
> > > > Service Discovery (SD) using the manifest.
> > > > - A client queries SD for instances of a specific Endpoint and
> > receives a
> > > > list of IP:Port combinations
> > > > - The client connects to the service and provides its peer token
> > > >
> > > > T

Experimenting with Kafka and OpenSSL

[Jaikiran Pai]

We have been using Kafka in some of our projects for the past couple of 
years. Our experience with Kafka and SSL had shown some performance 
issues when we had seriously tested it (which admittedly was around a 
year back). Our basic tests did show that things had improved over time 
with newer versions, but we didn't get a chance to fully test and move 
to SSL for Kafka.


Incidentally, I happened to be looking into some other things related to 
SSL and decided to experiment with using openssl as the SSL provider for 
Kafka. I had heard OpenSSL performs better than the engine shipped 
default in JRE, but hadn't ever got a chance to do any experiments. This 
past few weeks, I decided to spend some time trying it. I have noted the 
experimentation and the performance numbers in my blog[1]. The initial 
basic performance testing (using the scripts shipped in Kafka) does show 
promising improvements. Like I note in my blog, this was a very basic 
performance test just to see if OpenSSL can be pursued as an option 
(both in terms of being functional and performant) if we do decide to.


I know some of the members in these lists do extensive performance 
testing with Kafka (and SSL), so I thought I will bring this to their 
notice.


[1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html

-Jaikiran

[GitHub] kafka pull request #4156: Merge pull request #2 from apache/trunk

[hejiefang]

GitHub user hejiefang opened a pull request:

https://github.com/apache/kafka/pull/4156

Merge pull request #2 from apache/trunk

pull

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hejiefang/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4156


commit a9014ffac2c0c2f477589e650cb8c6eac5770674
Author: JieFang.He 
Date:   2017-08-03T05:41:55Z

Merge pull request #2 from apache/trunk

pull




---

[jira] [Resolved] (KAFKA-788) Periodic refresh of topic metadata on the producer doesn't include all topics

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-788.
-
Resolution: Auto Closed

Closing  inactive issue. The old producer is no longer supported.


> Periodic refresh of topic metadata on the producer doesn't include all topics
> -
>
> Key: KAFKA-788
> URL: https://issues.apache.org/jira/browse/KAFKA-788
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>  Labels: kafka-0.8, p2
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> We added a patch to the producer to refresh the metadata for all topics 
> periodically. However, the producer only does this for the topics in the last 
> batch. But some topics sent by the producer could be low throughput and might 
> not be present in every batch. If we bounce the cluster or if brokers fail 
> and leaders change, the metadata for those low throughput topic is not 
> refreshed by this periodic topic metadata request. The next produce request 
> for those topics have to fail and then a separate metadata request needs to 
> be reissued to handle the produce request. This is especially a problem for 
> the migration tool. So even if the producer had a chance to refresh the 
> metadata when the leader changed, it throws LeaderNotAvailableExceptions much 
> later when it sends a request for that topic. 
> I propose we just fetch data for all topics sent by the producer in the 
> periodic refresh of topic metadata



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Experimenting with Kafka and OpenSSL

[Mickael Maison]

Thanks for sharing, very interesting read.

Did you get a chance to try JDK 9 ?

We also considered using OpenSSL instead of JSSE especially since
Netty made an easy to re-use package (netty-tcnative).

There was KAFKA-2561
(https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
a few numbers and what would be need to get it working.

On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai  wrote:
> We have been using Kafka in some of our projects for the past couple of
> years. Our experience with Kafka and SSL had shown some performance issues
> when we had seriously tested it (which admittedly was around a year back).
> Our basic tests did show that things had improved over time with newer
> versions, but we didn't get a chance to fully test and move to SSL for
> Kafka.
>
> Incidentally, I happened to be looking into some other things related to SSL
> and decided to experiment with using openssl as the SSL provider for Kafka.
> I had heard OpenSSL performs better than the engine shipped default in JRE,
> but hadn't ever got a chance to do any experiments. This past few weeks, I
> decided to spend some time trying it. I have noted the experimentation and
> the performance numbers in my blog[1]. The initial basic performance testing
> (using the scripts shipped in Kafka) does show promising improvements. Like
> I note in my blog, this was a very basic performance test just to see if
> OpenSSL can be pursued as an option (both in terms of being functional and
> performant) if we do decide to.
>
> I know some of the members in these lists do extensive performance testing
> with Kafka (and SSL), so I thought I will bring this to their notice.
>
> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
>
> -Jaikiran
>

[GitHub] kafka pull request #4157: KAFKA-6130: VerifiableConsumer with --max-messages

[tombentley]

GitHub user tombentley opened a pull request:

https://github.com/apache/kafka/pull/4157

KAFKA-6130: VerifiableConsumer with --max-messages

@hachikuji can you review this? Do you know the best place to write a test, 
since there doesn't seem to be a unit test for the `VerifiableConsumer`?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tombentley/kafka 
KAFKA-6130-verifiable-consumer-max-messages

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4157.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4157


commit 7ee4341babbf9616594169cf6c5cf2d093870d4e
Author: Tom Bentley 
Date:   2017-10-26T10:41:15Z

KAFKA-6130: VerifiableConsumer with --max-messages




---

[jira] [Resolved] (KAFKA-3071) Kafka Server 0.8.2 ERROR OOME with siz

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3071.

Resolution: Auto Closed

The SimpleConsumer is no longer supported and will be removed in a future 
release. Please upgrade to the Java Consumer whenever possible.

> Kafka Server 0.8.2 ERROR OOME with siz
> --
>
> Key: KAFKA-3071
> URL: https://issues.apache.org/jira/browse/KAFKA-3071
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.2.0
> Environment: Linux * 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed 
> Jul 16 06:12:23 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: vijay bhaskar
>  Labels: build
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> [2016-01-06 12:34:18.819-0700] INFO Truncating log hughes-order-status-73 to 
> offset 18. (kafka.log.Log)
> [2016-01-06 12:34:18.819-0700] INFO Truncating log troubleshoot-completed-125 
> to offset 3. (kafka.log.Log)
> [2016-01-06 12:34:19.064-0700] DEBUG Scheduling task highwatermark-checkpoint 
> with initial delay 0 ms and period 5000 ms. (kafka.utils.KafkaScheduler)
> [2016-01-06 12:34:19.106-0700] DEBUG Scheduling task [__consumer_offsets,0] 
> with initial delay 0 ms and period -1 ms. (kafka.utils.KafkaScheduler)
> [2016-01-06 12:34:19.106-0700] INFO Loading offsets from 
> [__consumer_offsets,0] (kafka.server.OffsetManager)
> [2016-01-06 12:34:19.108-0700] INFO Finished loading offsets from 
> [__consumer_offsets,0] in 2 milliseconds. (kafka.server.OffsetManager)
> [2016-01-06 12:34:27.023-0700] ERROR OOME with size 743364196 
> (kafka.network.BoundedByteBufferReceive)
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
> at 
> kafka.network.BoundedByteBufferReceive.byteBufferAllocate(BoundedByteBufferReceive.scala:80)
> at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:63)
> at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
> at 
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:72)
> at 
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
> at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:113)
> at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:113)
> at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:113)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:112)
> at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:112)
> at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:112)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:111)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:97)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2016-01-06 12:34:32.003-0700] ERROR OOME with size 743364196 
> (kafka.network.BoundedByteBufferReceive)
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
> at 
> kafka.network.BoundedByteBufferReceive.byteBufferAllocate(BoundedByteBufferReceive.scala:80)
> at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:63)
> at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108)
> at 
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:80)
> at 
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
> at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:113)
> at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:113)
> at 
> k

[GitHub] kafka pull request #4158: KAFKA-2903: FileRecords.read doesn't handle size >...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/4158

KAFKA-2903: FileRecords.read doesn't handle size > sizeInBytes when start 
is not zero



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-2903-file-records-read-slice-size-greater

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4158.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4158


commit b251038e2c569fda88376b85918a5003927b8152
Author: Ismael Juma 
Date:   2017-10-30T11:45:00Z

KAFKA-2903: FileRecords.read doesn't handle size > sizeInBytes when start 
is not zero




---

Re: Experimenting with Kafka and OpenSSL

[Jaikiran Pai]

I haven't yet had a chance to try out Java 9, but that's definitely on 
my TODO list, maybe sometime this weekend.


Thanks for pointing me to KAFKA-2561. I had missed that.

-Jaikiran


On 30/10/17 4:17 PM, Mickael Maison wrote:

Thanks for sharing, very interesting read.

Did you get a chance to try JDK 9 ?

We also considered using OpenSSL instead of JSSE especially since
Netty made an easy to re-use package (netty-tcnative).

There was KAFKA-2561
(https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
a few numbers and what would be need to get it working.

On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai  wrote:

We have been using Kafka in some of our projects for the past couple of
years. Our experience with Kafka and SSL had shown some performance issues
when we had seriously tested it (which admittedly was around a year back).
Our basic tests did show that things had improved over time with newer
versions, but we didn't get a chance to fully test and move to SSL for
Kafka.

Incidentally, I happened to be looking into some other things related to SSL
and decided to experiment with using openssl as the SSL provider for Kafka.
I had heard OpenSSL performs better than the engine shipped default in JRE,
but hadn't ever got a chance to do any experiments. This past few weeks, I
decided to spend some time trying it. I have noted the experimentation and
the performance numbers in my blog[1]. The initial basic performance testing
(using the scripts shipped in Kafka) does show promising improvements. Like
I note in my blog, this was a very basic performance test just to see if
OpenSSL can be pursued as an option (both in terms of being functional and
performant) if we do decide to.

I know some of the members in these lists do extensive performance testing
with Kafka (and SSL), so I thought I will bring this to their notice.

[1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html

-Jaikiran

[jira] [Resolved] (KAFKA-2725) high level consumer rebalances with auto-commit disabled should throw an exception

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2725.

Resolution: Auto Closed

The Scala consumers are no longer supported. If this still applies to the Java 
consumer, please file a new issue.

> high level consumer rebalances with auto-commit disabled should throw an 
> exception
> --
>
> Key: KAFKA-2725
> URL: https://issues.apache.org/jira/browse/KAFKA-2725
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1
> Environment: Experienced on Java running in linux
>Reporter: Cliff Rhyne
>
> Auto-commit is a very resilient mode.  Drops in zookeeper sessions due to JVM 
> garbage collection, network, rebalance or other interference are handled 
> gracefully within the kafka client.
> Systems still can drop due to unexpected gc or network behavior.  My proposal 
> is to handle this drop better when auto-commit is turned off:
> - If a rebalance or similar occur (which cause the offset to get reverted in 
> the client), check and see if the client was assigned back to the same 
> partition or a different one.  If it's the same partition, find the place 
> last consumed (it doesn't do this today for us).  This is to make a graceful 
> recovery.
> - If the partition assignment changes (which can mean duplicate data is 
> getting processed), throw an exception back to the application code.  This 
> lets the application code handle this exception-case with respect to the work 
> it's doing (with might be transactional).  Failing "silently" (yes it's still 
> getting logged) is very dangerous in our situation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2662) Make ConsumerIterator thread-safe for multiple threads in different Kafka groups

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2662.

Resolution: Auto Closed

The Scala consumers are no longer supported. Please upgrade to the Java 
consumer whenever possible.

> Make ConsumerIterator thread-safe for multiple threads in different Kafka 
> groups
> 
>
> Key: KAFKA-2662
> URL: https://issues.apache.org/jira/browse/KAFKA-2662
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.8.2.1
>Reporter: Andrew Pennebaker
>Assignee: Neha Narkhede
>
> The API for obtaining a ConsumerIterator requires a group parameter, implying 
> that ConsumerIterators are thread-safe, as long as each thread is in a 
> different Kafka group. However, in practice, attempting to call hasNext() on 
> ConsumerIterators for a thread in one group, and for a thread in another 
> group, results in an InvalidStateException.
> In the future, can we please make ConsumerIterator thread-safe, for a common 
> use case of one consumer thread per group?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2636) Producer connectivity obscured connection failure logging

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2636.

Resolution: Not A Problem

The producer should throw a TimeoutException to the callback or if `Future.get` 
is called.

> Producer connectivity obscured connection failure logging
> -
>
> Key: KAFKA-2636
> URL: https://issues.apache.org/jira/browse/KAFKA-2636
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1, 0.8.2.2
> Environment: Windows 8 running java implementation of Kafka Producer
>Reporter: Jason Kania
>
> The Kafka Producer does not generate a visible exception when a connection 
> cannot be made. Instead DEBUG settings are required to observe the problem as 
> shown below:
> [2015-10-12 21:23:20,335] DEBUG Error connecting to node 0 at 
> 482f4769eed1:9092: (org.apache.kafka.clients.NetworkClient)
> java.io.IOException: Can't resolve address: 482f4769eed1:9092
>   at org.apache.kafka.common.network.Selector.connect(Selector.java:138)
>   at 
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:417)
>   at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:116)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:165)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.nio.channels.UnresolvedAddressException
>   at sun.nio.ch.Net.checkAddress(Unknown Source)
>   at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
>   at org.apache.kafka.common.network.Selector.connect(Selector.java:135)
>   ... 5 more
> [2015-10-12 21:23:20,358] DEBUG Initiating connection to node 0 at 
> 482f4769eed1:9092. (org.apache.kafka.clients.NetworkClient)
> Secondly, the errors do not identify the node by IP address making error 
> investigation more difficult especially when learning to use Kafka.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2616) Improve Kakfa client exceptions

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2616.

Resolution: Auto Closed

BlockingChannel is only used by the Scala clients. which are no longer 
supported. Please upgrade to the Java clients whenever possible.


> Improve Kakfa client exceptions
> ---
>
> Key: KAFKA-2616
> URL: https://issues.apache.org/jira/browse/KAFKA-2616
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.2.1
>Reporter: Hurshal Patel
>Priority: Minor
>
> Any sort of network failure results in a {{java.nio.ClosedChannelException}} 
> which is bubbled up from {{kafka.network.BlockingChannel}}. 
> Displaying such an exception to a user with little knowledge about Kafka can 
> be more confusing than informative. A better user experience for the Kafka 
> consumer would be to throw a more appropriately named exception when a 
> {{ClosedChannelException}} is encountered.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (KAFKA-3917) Some __consumer_offsets replicas grow way too big

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-3917:


If you're running 0.10.2.0, then potentially KAFKA-5413, but I'll reopen until 
we can verify.

> Some __consumer_offsets replicas grow way too big
> -
>
> Key: KAFKA-3917
> URL: https://issues.apache.org/jira/browse/KAFKA-3917
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0
> Environment: Runs with Docker 1.10.1 in a container on 
> Linux 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64
>Reporter: Maxim Vladimirskiy
>  Labels: reliability
>
> We noticed that some replicas of partitions of the __consumer_offsets topic 
> grow way too big. Looking inside respective folders it became apparent that 
> old segments had not been cleaned up. Please see below example of disk usage 
> data for both affected and not affected partitions:
> Not affected partitions:
> Partition: 0  Leader: 2   Replicas: 2,3,4 Isr: 2,4,3
> 2: 49M
> 3: 49M
> 4: 49M
> Affected partitions:
> Partition: 10 Leader: 2   Replicas: 2,0,1 Isr: 1,2,0
> 0: 86M
> 1: 22G <<< too big!
> 2: 86M
> Partition: 38 Leader: 0   Replicas: 0,4,1 Isr: 1,0,4
> 0: 43M
> 1: 26G <<<  too big!
> 4: 26G <<<  too big!
> As you can see sometimes only one replica is affected, sometimes both 
> replicas are affected.
> When I try to restart a broker that has affected replicas it fails to start 
> with an exception that looks like this:
> [2016-06-28 23:15:20,441] ERROR There was an error in one of the threads 
> during logs loading: java.lang.IllegalArgumentException: requirement failed: 
> Corrupt index found, index file 
> (/var/kafka/__consumer_offsets-38/.index) has non-zero 
> size but the last offset is -676703869 and the base offset is 0 
> (kafka.log.LogManager)
> [2016-06-28 23:15:20,442] FATAL [Kafka Server 1], Fatal error during 
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.IllegalArgumentException: requirement failed: Corrupt index found, 
> index file (/var/kafka/__consumer_offsets-38/.index) has 
> non-zero size but the last offset is -676703869 and the base offset is 0
> at scala.Predef$.require(Predef.scala:233)
> at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:183)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.Log.loadSegments(Log.scala:183)
> at kafka.log.Log.(Log.scala:67)
> at 
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
> at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> After the content of the affected partition is deleted broker starts 
> successfully. 
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-407) Uncaught InputStream.close() exception in CompressionUtils.compress()

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-407.
-
Resolution: Won't Fix

This part of the code is removed. So closing the issue. 

> Uncaught InputStream.close() exception in CompressionUtils.compress()
> -
>
> Key: KAFKA-407
> URL: https://issues.apache.org/jira/browse/KAFKA-407
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.7.1
>Reporter: Lorenzo Alberton
>Priority: Minor
>
> In CompressionUtils.compress(), in this try/catch block:
> ==
> try {
>   cf.write(messageByteBuffer.array)
> } catch {
>   case e: IOException => error("Error while writing to the GZIP output 
> stream", e)
>   cf.close()
>   throw e
> } finally {
>   cf.close()
> }
> ==
> cf.close() might throw an IOException, which is not handled.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-354) Refactor getter and setter API to conform to the new convention

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-354.
-
Resolution: Won't Fix

Closing inactive issue.

> Refactor getter and setter API to conform to the new convention
> ---
>
> Key: KAFKA-354
> URL: https://issues.apache.org/jira/browse/KAFKA-354
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>Assignee: Joe Stein
>  Labels: optimization
>
> We just agreed on a new convention for getter/setter APIs. It will be good to 
> refactor code to conform to that.
> > We can actually go with public vals or vars - there is not much point in
> > defining a custom getter/setter as that is redundant.
> >
> > For example:
> > - start with "val x"
> > - over time, we determine that it needs to be mutable - change it to "var
> > x"
> > - if you need something more custom (e.g., enforce constraints on the
> > values that you can assign) then we can add the custom setter
> >  private[this] var underyling: T = ...
> >  def  x = underlying
> >  def x_=(update: T)  { if (constraint satisfied) {underlying = update}
> > else {throw new Exception} }
> >
> > All of the above changes will be binary compatible since under the covers,
> > reads/assignments are all through getter/setter methods.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-435) Keep track of the transient test failure for Kafka-343 on Apache Jenkins

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-435.
-
Resolution: Cannot Reproduce

Closing inactive issue.

> Keep track of the transient test failure for Kafka-343 on Apache Jenkins
> 
>
> Key: KAFKA-435
> URL: https://issues.apache.org/jira/browse/KAFKA-435
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.8.0
>Reporter: Yang Ye
>Assignee: Yang Ye
>Priority: Minor
>  Labels: transient-unit-test-failure
>
> See: 
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-commits/201208.mbox/browser
> Error message:
> --
> [...truncated 3415 lines...]
> [2012-08-01 17:27:08,432] ERROR KafkaApi on Broker 0, error when processing 
> request (test_topic,0,-1,1048576)
> (kafka.server.KafkaApis:99)
> kafka.common.OffsetOutOfRangeException: offset -1 is out of range
>   at kafka.log.Log$.findRange(Log.scala:46)
>   at kafka.log.Log.read(Log.scala:265)
>   at 
> kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:377)
>   at 
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1$$anonfun$apply$21.apply(KafkaApis.scala:333)
>   at 
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1$$anonfun$apply$21.apply(KafkaApis.scala:332)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
>   at 
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:332)
>   at 
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:328)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
>   at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
>   at 
> kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSets(KafkaApis.scala:328)
>   at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:272)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:38)
>   at java.lang.Thread.run(Thread.java:662)
> [2012-08-01 17:27:08,446] ERROR Closing socket for /67.195.138.9 because of 
> error (kafka.network.Processor:99)
> java.io.IOException: Connection reset by peer
>   at sun.nio.ch.FileDispatcher.read0(Native Method)
>   at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>   at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
>   at sun.nio.ch.IOUtil.read(IOUtil.java:171)
>   at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
>   at kafka.utils.Utils$.read(Utils.scala:630)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Processor.read(SocketServer.scala:296)
>   at kafka.network.Processor.run(SocketServer.scala:212)
>   at java.lang.Thread.run(Thread.java:662)
> [info] Test Passed: 
> testResetToEarliestWhenOffsetTooLow(kafka.integration.AutoOffsetResetTest)
> [info] Test Starting: 
> testResetToLatestWhenOffsetTooHigh(kafka.integration.AutoOffsetResetTest)
> [2012-08-01 17:27:09,203] ERROR KafkaApi on Broker 0, error when processing 
> request (test_topic,0,1,1048576)
> (kafka.server.KafkaApis:99)
> kafka.common.OffsetOutOfRangeException: offset 1 is out of range
>   at kafka.log.Log$.findRange(Log.scala:46)
>   at kafka.log.Log.read(Log.scala:265)
>   at 
> kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:377)
>   at 
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1$$anonfun$apply$21.apply(KafkaApis.scala:333)
>   at 
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1$$anonfun$apply$21.apply(KafkaApis.scala:332)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
>   at 
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:332)
>   at 
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:328)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
>   at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
>   at 
> kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSets(KafkaApis.scala:328)
>   at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala

[jira] [Resolved] (KAFKA-599) SimpleConsumerShell ONLY connects to the first host in the broker-list string to fetch topic metadata

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-599.
-
Resolution: Auto Closed

The Scala consumers are no longer supported. 


> SimpleConsumerShell ONLY connects to the first host in the broker-list string 
> to fetch topic metadata
> -
>
> Key: KAFKA-599
> URL: https://issues.apache.org/jira/browse/KAFKA-599
> Project: Kafka
>  Issue Type: Bug
>Reporter: John Fung
>Priority: Critical
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-610) connect.timeout.ms seems to do the wrong thing in the producer

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-610.
-
Resolution: Auto Closed

Closing  inactive issue. The old producer is no longer supported.


> connect.timeout.ms seems to do the wrong thing in the producer
> --
>
> Key: KAFKA-610
> URL: https://issues.apache.org/jira/browse/KAFKA-610
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jay Kreps
>
> This setting used to control the socket connection timeout. This is also what 
> the documentation says:
> "the maximum time spent by kafka.producer.SyncProducer trying to connect to 
> the kafka broker. Once it elapses, the producer throws an ERROR and stops."
> But it doesn't look to me that this parameter is being used at all. The only 
> thing we do with it is check in a catch statement if that much time has 
> ellapsed and then throw an error. Since we haven't set the connection timeout 
> this is silly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-1006) Consumer loses messages of a new topic with auto.offset.reset = largest

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1006.
--
Resolution: Auto Closed

Closing inactive issue. The old consumer is no longer supported, please upgrade 
to the Java consumer whenever possible.

> Consumer loses messages of a new topic with auto.offset.reset = largest
> ---
>
> Key: KAFKA-1006
> URL: https://issues.apache.org/jira/browse/KAFKA-1006
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Swapnil Ghike
>Assignee: Guozhang Wang
>  Labels: usability
>
> Consumer currently uses auto.offset.reset = largest by default. If a new 
> topic is created, consumer's topic watcher is fired. The consumer will first 
> finish partition reassignment as part of rebalance and then start consuming 
> from the tail of each partition. Until the partition reassignment is over, 
> the server may have appended new messages to the new topic, consumer won't 
> consume these messages. Thus, multiple batches of messages may be lost when a 
> topic is newly created. 
> The fix is to start consuming from the earliest offset for newly created 
> topics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-1066) Reduce logging on producer connection failures

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1066.
--
Resolution: Auto Closed

Closing inactive issue. The old producer is no longer supported

> Reduce logging on producer connection failures
> --
>
> Key: KAFKA-1066
> URL: https://issues.apache.org/jira/browse/KAFKA-1066
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.0
>Reporter: Jason Rosenberg
>Assignee: Jun Rao
>
> Below is a stack trace from a unit test, where a producer tries to send a 
> message, but no server is available.
> The exception/message logging seems to be inordinately verbose.  
> I'm thinking a simple change could be to not log full stack traces for simple 
> things like "Connection refused", etc.  Seems it would be fine to just log 
> the exception message in such cases.
> Also, the log levels could be tuned, such that things logged as ERROR 
> indicate that all possible retries have been attempted, rather than having it 
> be an ERROR for each step of the retry/failover process.  Thus, for a 
> redundant, clustered service, it should be considered normal that single 
> nodes will be unavailable (such as when we're doing a rolling restart of the 
> cluster, etc.).  It should only be an ERROR if all brokers/all replicas are 
> unavailable, etc.  This way, we can selectively set our log level to ERROR, 
> and have it be useful.
> This is from one of my unit tests.  I am using the default retry count (3) 
> here, but even if I reduced that, it seems this is a crazy amount of logging. 
>  (I've edited this to remove from each stack trace the portion of 
> testing/calling code, into the Producer.send() call).
> Here's the code snippet that produced the logging below (and note, the server 
> was not available on the requested port).
>   KeyedMessage msg = new KeyedMessage(topic, 
> message);
>   producer.send(msg);
> Jason
> 599 [main] ERROR kafka.producer.SyncProducer  - Producer connection to 
> localhost:1025 unsuccessful
> java.net.ConnectException: Connection refused
>   at sun.nio.ch.Net.connect0(Native Method)
>   at sun.nio.ch.Net.connect(Net.java:465)
>   at sun.nio.ch.Net.connect(Net.java:457)
>   at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639)
>   at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>   at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
>   at 
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
>   at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
>   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
>   at 
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
>   at 
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:69)
>   at kafka.utils.Utils$.swallow(Utils.scala:186)
>   at kafka.utils.Logging$class.swallowError(Logging.scala:105)
>   at kafka.utils.Utils$.swallowError(Utils.scala:45)
>   at 
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:69)
>   at kafka.producer.Producer.send(Producer.scala:74)
>   at kafka.javaapi.producer.Producer.send(Producer.scala:32)
>   ...
> 615 [main] WARN kafka.client.ClientUtils$  - Fetching topic metadata with 
> correlation id 0 for topics [Set(test-topic)] from broker 
> [id:0,host:localhost,port:1025] failed
> java.net.ConnectException: Connection refused
>   at sun.nio.ch.Net.connect0(Native Method)
>   at sun.nio.ch.Net.connect(Net.java:465)
>   at sun.nio.ch.Net.connect(Net.java:457)
>   at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639)
>   at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>   at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
>   at 
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
>   at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
>   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
>   at 
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
>   at 
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:69)
>   at kafka.utils.Utils$.swallow(Utils.scala:186)
>   at kafka.utils.Logging$class.swallowError(Logging.scala:105)
>   at kafka.utils.Utils$.swallowError(Utils.scala:45)
>   at 
> kafka.producer.async.DefaultEventHandler.handle(DefaultEven

[jira] [Resolved] (KAFKA-1415) Async producer.send can block forever if async.ProducerSendThread dies

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1415.
--
Resolution: Auto Closed

Closing inactive issue. The old producer is no longer supported, please upgrade 
to the Java producer whenever possible.

> Async producer.send can block forever if async.ProducerSendThread dies
> --
>
> Key: KAFKA-1415
> URL: https://issues.apache.org/jira/browse/KAFKA-1415
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.8.0
> Environment: kafka_2.9.2-0.8.0.jar
>Reporter: James Blackburn
>Assignee: Jun Rao
>
> We noticed that if something goes fundamentally wrong (in this case the jars 
> were replaced under a running Producer's feet) then async calls to: 
> {{producer.send}} can lockup forever.
> I saw in the log file the following exception logged:
> {code}
> [2014-04-17 16:45:36,484] INFO Disconnecting from cn2:9092 
> (kafka.producer.SyncProducer)
> Exception in thread "ProducerSendThread-" java.lang.NoClassDefFoundError: 
> kafka/producer/async/ProducerSendThread$$anonfun$run$1
> at 
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
> Caused by: java.lang.ClassNotFoundException: 
> kafka.producer.async.ProducerSendThread$$anonfun$run$1
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 1 more
> {code}
> However my application continued running. Jstack showed that the 
> producer.send calls had all locked up:
> {code}
> "SubscriberEventQueue0Executor-1" prio=10 tid=0x2aaab0a88000 nid=0x44f5 
> waiting on condition [0x44ac4000]
>java.lang.Thread.State: WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0x000790c47918> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>   at 
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
>   at kafka.producer.Producer$$anonfun$asyncSend$1.apply(Producer.scala:98)
>   at kafka.producer.Producer$$anonfun$asyncSend$1.apply(Producer.scala:90)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
>   at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
>   at kafka.producer.Producer.asyncSend(Producer.scala:90)
>   at kafka.producer.Producer.send(Producer.scala:77)
>   - locked <0x000791768ee8> (a java.lang.Object)
>   at kafka.javaapi.producer.Producer.send(Producer.scala:33)
>   at com.mi.ahl.kafka.rmds2kafka.Bridge$1.onMarketData(Bridge.java:165)
>Locked ownable synchronizers:
>   - <0x000792205cd0> (a 
> java.util.concurrent.ThreadPoolExecutor$Worker)
> "SubscriberEventQueue1Executor-2" prio=10 tid=0x2aaab0aa nid=0x4511 
> waiting for monitor entry [0x44dc7000]
>java.lang.Thread.State: BLOCKED (on object monitor)
>   at kafka.producer.Producer.send(Producer.scala:71)
>   - waiting to lock <0x000791768ee8> (a java.lang.Object)
>   at kafka.javaapi.producer.Producer.send(Producer.scala:33)
>   at com.mi.ahl.kafka.rmds2kafka.Bridge$1.onMarketData(Bridge.java:165)
> "SubscriberEventQueue2Executor-3" prio=10 tid=0x2aaab0ab6800 nid=0x4512 
> waiting for monitor entry [0x44ec8000]
>java.lang.Thread.State: BLOCKED (on object monitor)
>   at kafka.producer.Producer.send(Producer.scala:71)
>   - waiting to lock <0x000791768ee8> (a java.lang.Object)
>   at kafka.javaapi.producer.Producer.send(Producer.scala:33)
>   at com.mi.ahl.kafka.rmds2kafka.Bridge$1.onMarketData(Bridge.java:165)
> "SubscriberEventQueue3Executor-4" prio=10 tid=0x2aaab0ab8800 nid=0x4513 
> waiting for monitor entry [0x44fc9000]
>java.lang.Thread.State: BLOCKED (on object monitor)
>   at kafka.producer.Producer.send(Producer.scala:71)
>   - waiting to lock <0x000791768ee8> (a java.lang.Object)
>   at kafka.javaapi.producer.Producer.send(Producer.scala:33)
>   at com.mi.ahl.kafka.rmds2kafka.Bridge$1.onMarketData(Bridge.java:165)
> {code}
> *Expectation:*
> {{producer.send}} woul

[jira] [Resolved] (KAFKA-1958) ZookeeperConsumerConnector doesn't remove consumer node on shutdown.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1958.
--
Resolution: Auto Closed

Closing inactive issue. The old consumer is no longer supported, please upgrade 
to the Java consumer whenever possible.

> ZookeeperConsumerConnector doesn't remove consumer node on shutdown.
> 
>
> Key: KAFKA-1958
> URL: https://issues.apache.org/jira/browse/KAFKA-1958
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.1.1
>Reporter: Beletsky Andrey
>Assignee: Neha Narkhede
>  Labels: consumer, shutdown, zookeeper
>
> We use kafka with ZooKeeper via high level consumer.
> There is a scheduled job that creates a consumer with specific group, does 
> necessary logic and shuts down this consumer.
> +An issue:+
> Nobody deletes */consumers/myGroup/ids/myGroup__*. And after 
> several job runs there are a lot of dead consumer IDs under myGroup. I've got 
> [an 
> issue|https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whysomeoftheconsumersinaconsumergroupneverreceiveanymessage?]
>  that new consumer doesn't see a partition.
> We start to implement an approach to remove a consumer nodes from Zookeeper 
> manually after consumer is shutted down.
> I think better way to remove this node during 
> *ZookeeperConsumerConnector.shutdown()*.
> *P.S.:*
> If I missed something in your sources please let me know.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2062) Sync Producer, Variable Message Length, Multiple Threads = Direct memory overuse

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2062.
--
Resolution: Auto Closed

Closing inactive issue. The old producer is no longer supported, please upgrade 
to the Java producer whenever possible.

> Sync Producer, Variable Message Length, Multiple Threads = Direct memory 
> overuse
> 
>
> Key: KAFKA-2062
> URL: https://issues.apache.org/jira/browse/KAFKA-2062
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.1.1
>Reporter: Michael Braun
>Assignee: Jun Rao
>
> Using a synchronous producer with multiple threads each calling .send on the 
> single producer object, each thread ends up maintaining a threadlocal direct 
> memory buffer. In a case of messages where the sizing varies(for instance, 
> 99% of messages are 1MB and 1% are 100MB), eventually the buffers seem to 
> expand to this level for all the threads which can cause an out of memory - 
> direct buffer memory error:
> java.lang.OutOfMemoryError: Direct buffer memory
>   at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_67]
>   at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[na:1.7.0_67]
>   at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) ~[na:1.7.0_67]
>   at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) ~[na:1.7.0_67]
>   at sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[na:1.7.0_67]
>   at sun.nio.ch.SocketChannelImpl.write(SocketChannel.java:493) ~[na:1.7.0_67]
>   at java.nio.channels.SocketChannel.write(SocketChannel.java:493) 
> ~[na:1.7.0_67]
>   at 
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56) 
> ~[kafka_2.10-0.8.1.1.jar:na]
>   at kafka.network.Send$class.writeCompletely(Transmission.scala:75) 
> ~[kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:26)
>  ~[kafka_2.10-0.8.1.1.jar:na]
>   at kafka.network.BlockingChannel.send(BlockingChannel.scala:92) 
> ~[kafka_2.10-0.8.1.1.jar:na]
>   at kafka.producer.SyncProducer.liftedTree$1(SyncProducer.scala:72) 
> ~[kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
>  ~[kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
>  ~[kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
>  ~[kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
>  ~[kafka_2.10-0.8.1.1.jar:na]
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) 
> ~[kafka_2.10-0.8.1.1.jar:na]
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:100) 
> ~[kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DEfaultEventHandler$$send(DefaultEventHandler.scala:255)
>  [kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:106)
>  [kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:100)
>  [kafka_2.10-0.8.1.1.jar:na]
>   at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>  [scala-library-2.10.1.jar:na]
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
> [scala-library-2.10.1.jar:na]
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
> [scala-library-2.10.1.jar:na]
>   at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTasble.scala:226) 
> [scala-library-2.10.1.jar:na]
>   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) 
> [scala-library-2.10.1.jar:na]
>   at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) 
> [scala-library-2.10.1.jar:na]
>   at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>  [scala-library-2.10.1.jar:na]
>   at 
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DEfaultEventHandler.scala:100)
>  [kafka_2.10-0.8.1.1.jar:na]
>   at 
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72) 
> [kafka_2.10-0.8.1.1.jar:na]
>   at kafka.producer.Producer.send(Producer.scala:76) 
> [kafka_2.10-0.8.1.1.jar:na]
>   at kafka.javaapi.producer.Producer.send(Producer.scala:33) 
> [kafka_2.10-0.8.1.1.jar:na]
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2995) in 0.9.0.0 Old Consumer's commitOffsets with specify partition can submit not exists topic and partition to zk

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2995.
--
Resolution: Auto Closed

Closing inactive issue. The old consumer is no longer supported, please upgrade 
to the Java consumer whenever possible.

> in 0.9.0.0 Old Consumer's commitOffsets with specify partition can submit not 
> exists topic and partition to zk
> --
>
> Key: KAFKA-2995
> URL: https://issues.apache.org/jira/browse/KAFKA-2995
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Pengwei
>Assignee: Neha Narkhede
>
> in 0.9.0.0 Version, the Old Consumer's commit interface is below:
> def commitOffsets(offsetsToCommit: immutable.Map[TopicAndPartition, 
> OffsetAndMetadata], isAutoCommit: Boolean) {
> trace("OffsetMap: %s".format(offsetsToCommit))
> var retriesRemaining = 1 + (if (isAutoCommit) 0 else 
> config.offsetsCommitMaxRetries) // no retries for commits from auto-commit
> var done = false
> while (!done) {
>   val committed = offsetsChannelLock synchronized {
> // committed when we receive either no error codes or only 
> MetadataTooLarge errors
> if (offsetsToCommit.size > 0) {
>   if (config.offsetsStorage == "zookeeper") {
> offsetsToCommit.foreach { case (topicAndPartition, 
> offsetAndMetadata) =>
>   commitOffsetToZooKeeper(topicAndPartition, 
> offsetAndMetadata.offset)
> }
>   
> this interface does not check the parameter offsetsToCommit, if 
> offsetsToCommit has some topic or partition which is not exist in the kafka. 
> Then will create an entry in the  /consumers/[group]/offsets/[Not exists 
> topic]   directory.
> We should check the offsetsToCommit's topic and partition is exists or just 
> check it is contain in the topicRegistry or checkpointedZkOffsets ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4159: KAFKA-5520: KIP-171 - Extend Consumer Group Reset ...

[jeqo]

GitHub user jeqo opened a pull request:

https://github.com/apache/kafka/pull/4159

KAFKA-5520: KIP-171 - Extend Consumer Group Reset Offset for Stream 
Application - Merge with KIP-198

KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+Application

Merge changes from KIP-198

Ref: https://github.com/apache/kafka/pull/3831

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeqo/kafka feature/kip-171

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4159.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4159






---

[GitHub] kafka pull request #3831: KAFKA-5520: KIP-171 - Extend Consumer Group Reset ...

[jeqo]

Github user jeqo closed the pull request at:

https://github.com/apache/kafka/pull/3831


---

[jira] [Created] (KAFKA-6142) Connect worker configurations and connector configurations should accept environment variables

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-6142:


 Summary: Connect worker configurations and connector 
configurations should accept environment variables
 Key: KAFKA-6142
 URL: https://issues.apache.org/jira/browse/KAFKA-6142
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 1.0.0
Reporter: Randall Hauch


Currently, when a worker or connector configuration is parsed, the values are 
used as-is without any kind of pre-processing before the value is used. It 
should be possible to define configuration properties such that string literal 
values or default values can use *_configuration variables_* that reference 
environment variables and/or system properties, and that these configuration 
variables are resolved/replaced before the configuration value is used.

I propose doing this enhancement in Kafka client's {{ConfigDef}} by adding a 
{{ConfigDef.Transformer}} interface:

{code:java}
/**
 * Transform the configuration value.
 */
public interface Transformer {
/**
 * Transform the configuration value.
 * @param name The name of the configuration
 * @param value The value of the configuration
 * @return the preprocessed value
 * @throws ConfigException if the value is invalid.
 */
Object apply(String name, Object value);
}
{code}

and then allowing {{Transformer}} implementations to be passed to 
{{ConfigDef.define(...)}} such all existing signatures are maintained for 
backward compatibility. By default, the definition would use an identity 
transform that simply returns the value. The transformers would be called in 
{{ConfigDef.parseValue(...)}} before the {{parseType(...)}} method is called, 
and would also be called on the default value if one is provided.

Then, a {{ConfigDef.ReplaceSystemVariables}} implementation would be provided 
to look in {{String}} values for zero or more variables defined with this EBNF 
grammar:

{noformat}
'$' '{' varName { ',' varName } [ ':' defaultValue] '}'
{noformat}

where:
* {{varName}} is the name of a Java system property or {{env.}} followed by the 
name of an environment variable, and 
* {{defaultValue}} specifies the replacement value used when no environment 
variable or system property is found, and defaults to an empty string. 

The value of the first system property or environment variable resolved is then 
used to replace the variable expression. This implementation would have trace 
or debug level logging to describe what it is doing.

For example, the variable {{$\{env.KAFKA_HOME\}}} would replace the variable 
expression with the value of the {{KAFKA_HOME}} environment variable or an 
blank string if that variable doesn't exist. 

Likewise, the variable {{$\{foo.prop1,foo.prop2,env.MY_ENV_VAR:value\}}} would 
be replaced with the value of the {{foo.prop1}} system property if it exists, 
or with the value of the {{foo.prop2}} system property if it exists, or with 
the value of the {{MY_ENV_VAR}} environment variable if it exists, or {{value}} 
if none of the system properties exist.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6143) VerifiableProducer & VerifiableConsumer need tests

[Tom Bentley (JIRA)]

Tom Bentley created KAFKA-6143:
--

 Summary: VerifiableProducer & VerifiableConsumer need tests
 Key: KAFKA-6143
 URL: https://issues.apache.org/jira/browse/KAFKA-6143
 Project: Kafka
  Issue Type: Bug
Reporter: Tom Bentley
Priority: Minor


The {{VerifiableProducer}} and {{VerifiableConsumer}} used use for system 
tests, but don't have any tests themselves. They should have.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4111: KAFKA-6073: Use ZookeeperClient in KafkaApis

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4111


---

[GitHub] kafka pull request #4157: KAFKA-6130: VerifiableConsumer with --max-messages

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4157


---

[jira] [Resolved] (KAFKA-6130) VerifiableConsumer with --max-messages doesn't exit

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-6130.

   Resolution: Fixed
Fix Version/s: 1.1.0

> VerifiableConsumer with --max-messages doesn't exit
> ---
>
> Key: KAFKA-6130
> URL: https://issues.apache.org/jira/browse/KAFKA-6130
> Project: Kafka
>  Issue Type: Bug
>Reporter: Tom Bentley
>Assignee: Tom Bentley
>Priority: Minor
> Fix For: 1.1.0
>
>
> If I run {{kafka-verifiable-consumer.sh --max-messages=N}} I expect the tool 
> to consume N messages and then exit. It will actually consume as many 
> messages as are in the topic and then block.
> The problem is that although  the max messages will cause the loop in 
> onRecordsReceived() to break, the loop in run() will just call 
> onRecordsReceived() again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Vahid S Hashemian]

Bump!



From:   Vahid S Hashemian/Silicon Valley/IBM
To: dev 
Date:   10/18/2017 04:45 PM
Subject:[DISCUSS] KIP-211: Revise Expiration Semantics of Consumer 
Group Offsets


Hi all,

I created a KIP to address the group offset expiration issue reported in 
KAFKA-4682:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets

Your feedback is welcome!

Thanks.
--Vahid

Re: [VOTE] KIP-175: Additional '--describe' views for ConsumerGroupCommand

[Vahid S Hashemian]

Bumping this again!



From:   "Vahid S Hashemian" 
To: dev@kafka.apache.org
Date:   09/28/2017 12:13 PM
Subject:Re: [VOTE] KIP-175: Additional '--describe' views for 
ConsumerGroupCommand



I'm bumping this up as it's awaiting one more binding +1, but I'd like to 
also mention a recent change to the KIP.

Since the current DescribeGroup response protocol does not include 
member-specific information such as preferred assignment strategies, or 
topic subscriptions, I've removed the corresponding ASSIGNMENT-STRATEGY 
and SUBSCRIPTION columns from --members and --members --verbose options, 
respectively. These columns will be added back once KIP-181 (that aims to 
enhance DescribeGroup response) is in place. I hope this small 
modification is reasonable. If needed, we can continue the discussion on 
the discussion thread.

And I'm not sure if this change requires a re-vote.

Thanks.
--Vahid



From:   "Vahid S Hashemian" 
To: dev 
Date:   07/27/2017 02:04 PM
Subject:[VOTE] KIP-175: Additional '--describe' views for 
ConsumerGroupCommand



Hi all,

Thanks to everyone who participated in the discussion on KIP-175, and 
provided feedback.
The KIP can be found at 
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D175-253A-2BAdditional-2B-2527-2D-2Ddescribe-2527-2Bviews-2Bfor-2BConsumerGroupCommand&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=m8Sjp4mLFIcTZ9zDzAkg1PAYjdCskeME4GnAjhscWcE&s=PoDGOd7DqeL_Tg-TCQqyaIGH3L5QyxUfYZ9UBLMT7Vw&e=
 


.
I believe the concerns have been addressed in the recent version of the 
KIP; so I'd like to start a vote.

Thanks.
--Vahid

[jira] [Created] (KAFKA-6144) Allow state stores to serve stale reads during rebalance

[Antony Stubbs (JIRA)]

Antony Stubbs created KAFKA-6144:


 Summary: Allow state stores to serve stale reads during rebalance
 Key: KAFKA-6144
 URL: https://issues.apache.org/jira/browse/KAFKA-6144
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Reporter: Antony Stubbs


Currently when expanding the KS cluster, the new node's partitions will be 
unavailable during the rebalance, which for large states can take a very long 
time, or for small state stores even more than a few ms can be a deal breaker 
for micro service use cases.

One workaround is to allow stale data to be read from the state stores when use 
case allows.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6145) Warm up new KS instances before migrating tasks - potentially a two phase rebalance

[Antony Stubbs (JIRA)]

Antony Stubbs created KAFKA-6145:


 Summary: Warm up new KS instances before migrating tasks - 
potentially a two phase rebalance
 Key: KAFKA-6145
 URL: https://issues.apache.org/jira/browse/KAFKA-6145
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Reporter: Antony Stubbs


Currently when expanding the KS cluster, the new node's partitions will be 
unavailable during the rebalance, which for large states can take a very long 
time, or for small state stores even more than a few ms can be a deal breaker 
for micro service use cases.
One workaround would be two execute the rebalance in two phases:
1) start running state store building on the new node
2) once the state store is fully populated on the new node, only then rebalance 
the tasks - there will still be a rebalance pause, but would be greatly reduced



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk9 #159

[Apache Jenkins Server]

See 


Changes:

[junrao] KAFKA-6073; Use ZookeeperClient in KafkaApis

[jason] KAFKA-6130; Ensure VerifiableConsumer halts when --max-messages is

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-us1 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 6118ecb590afed5fd41376b521a5dffe522f3c07 
(refs/remotes/origin/trunk)
Commit message: "KAFKA-6130; Ensure VerifiableConsumer halts when 
--max-messages is reached"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6118ecb590afed5fd41376b521a5dffe522f3c07
 > git rev-list f4e9c84c52fbb7ec6a4ca50c9ac35cacbd0df082 # timeout=10
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins942782290154472365.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2/bin/gradle

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine java version from '9.0.1'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found 
but none of them are new. Did tests run? 
For example, 

 is 10 days old

Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2

Build failed in Jenkins: kafka-trunk-jdk7 #2932

[Apache Jenkins Server]

See 


Changes:

[junrao] KAFKA-6073; Use ZookeeperClient in KafkaApis

--
[...truncated 383.29 KB...]
kafka.log.ProducerStateManagerTest > testRecoverFromSnapshot PASSED

kafka.log.ProducerStateManagerTest > 
testSequenceNotValidatedForGroupMetadataTopic STARTED

kafka.log.ProducerStateManagerTest > 
testSequenceNotValidatedForGroupMetadataTopic PASSED

kafka.log.ProducerStateManagerTest > 
testLoadFromSnapshotRemovesNonRetainedProducers STARTED

kafka.log.ProducerStateManagerTest > 
testLoadFromSnapshotRemovesNonRetainedProducers PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffset STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffset PASSED

kafka.log.ProducerStateManagerTest > testTxnFirstOffsetMetadataCached STARTED

kafka.log.ProducerStateManagerTest > testTxnFirstOffsetMetadataCached PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencedAfterReload STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencedAfterReload PASSED

kafka.log.ProducerStateManagerTest > testControlRecordBumpsEpoch STARTED

kafka.log.ProducerStateManagerTest > testControlRecordBumpsEpoch PASSED

kafka.log.ProducerStateManagerTest > 
testAcceptAppendWithoutProducerStateOnReplica STARTED

kafka.log.ProducerStateManagerTest > 
testAcceptAppendWithoutProducerStateOnReplica PASSED

kafka.log.ProducerStateManagerTest > testLoadFromCorruptSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromCorruptSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testProducerSequenceWrapAround STARTED

kafka.log.ProducerStateManagerTest > testProducerSequenceWrapAround PASSED

kafka.log.ProducerStateManagerTest > testPidExpirationTimeout STARTED

kafka.log.ProducerStateManagerTest > testPidExpirationTimeout PASSED

kafka.log.ProducerStateManagerTest > testAcceptAppendWithSequenceGapsOnReplica 
STARTED

kafka.log.ProducerStateManagerTest > testAcceptAppendWithSequenceGapsOnReplica 
PASSED

kafka.log.ProducerStateManagerTest > testOldEpochForControlRecord STARTED

kafka.log.ProducerStateManagerTest > testOldEpochForControlRecord PASSED

kafka.log.ProducerStateManagerTest > 
testTruncateAndReloadRemovesOutOfRangeSnapshots STARTED

kafka.log.ProducerStateManagerTest > 
testTruncateAndReloadRemovesOutOfRangeSnapshots PASSED

kafka.log.ProducerStateManagerTest > testStartOffset STARTED

kafka.log.ProducerStateManagerTest > testStartOffset PASSED

kafka.log.ProducerStateManagerTest > testProducerSequenceInvalidWrapAround 
STARTED

kafka.log.ProducerStateManagerTest > testProducerSequenceInvalidWrapAround 
PASSED

kafka.log.ProducerStateManagerTest > testTruncateHead STARTED

kafka.log.ProducerStateManagerTest > testTruncateHead PASSED

kafka.log.ProducerStateManagerTest > 
testNonTransactionalAppendWithOngoingTransaction STARTED

kafka.log.ProducerStateManagerTest > 
testNonTransactionalAppendWithOngoingTransaction PASSED

kafka.log.ProducerStateManagerTest > testSkipSnapshotIfOffsetUnchanged STARTED

kafka.log.ProducerStateManagerTest > testSkipSnapshotIfOffsetUnchanged PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[1] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[1] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[2] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[2] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[3] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[3] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[10] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[10] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[11] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[11] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[12] STARTED

kafka.log.BrokerCompr

[jira] [Created] (KAFKA-6146) re-register the exist watch on PreferredReplicaElectionZNode after the preferred leader election completes

[Jun Rao (JIRA)]

Jun Rao created KAFKA-6146:
--

 Summary: re-register the exist watch on 
PreferredReplicaElectionZNode after the preferred leader election completes 
 Key: KAFKA-6146
 URL: https://issues.apache.org/jira/browse/KAFKA-6146
 Project: Kafka
  Issue Type: Sub-task
Affects Versions: 1.1.0
Reporter: Jun Rao
 Fix For: 1.1.0


Currently, after the PreferredReplicaElectionZNode is removed, we don't 
register the exist watcher on the path again. This means that future preferred 
replica election event will be missed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4158: KAFKA-2903: FileRecords.read doesn't handle size >...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4158


---

[GitHub] kafka-site pull request #105: Adding Trivago logo to streams page

[manjuapu]

GitHub user manjuapu opened a pull request:

https://github.com/apache/kafka-site/pull/105

Adding Trivago logo to streams page

@guozhangwang Please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manjuapu/kafka-site lighthouse-logo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/105.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #105


commit ae71097e124e9b505fbf3ab06076becbd968f58b
Author: Manjula K 
Date:   2017-10-30T18:12:13Z

Adding Trivago logo




---

[jira] [Resolved] (KAFKA-5212) Consumer ListOffsets request can starve group heartbeats

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-5212.

   Resolution: Fixed
Fix Version/s: (was: 1.0.1)

Issue resolved by pull request 4110
[https://github.com/apache/kafka/pull/4110]

> Consumer ListOffsets request can starve group heartbeats
> 
>
> Key: KAFKA-5212
> URL: https://issues.apache.org/jira/browse/KAFKA-5212
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Jason Gustafson
>Assignee: Richard Yu
> Fix For: 1.1.0
>
> Attachments: 5212.patch
>
>
> The consumer is not able to send heartbeats while it is awaiting a 
> ListOffsets response. Typically this is not a problem because ListOffsets 
> requests are handled quickly, but in the worst case if the request takes 
> longer than the session timeout, the consumer will fall out of the group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4110: [KAFKA-5212] Consumer ListOffsets request can star...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4110


---

Build failed in Jenkins: kafka-trunk-jdk9 #160

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-2903; FileRecords.read doesn't handle size > sizeInBytes when

[jason] KAFKA-5212; Consumer ListOffsets request can starve group heartbeats

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H25 (couchdbtest ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 7fe88e8bd9acedcb74a4b3e61440f61481b60711 
(refs/remotes/origin/trunk)
Commit message: "KAFKA-5212; Consumer ListOffsets request can starve group 
heartbeats"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7fe88e8bd9acedcb74a4b3e61440f61481b60711
 > git rev-list 6118ecb590afed5fd41376b521a5dffe522f3c07 # timeout=10
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins4021895702527168504.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2/bin/gradle

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine java version from '9.0.1'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did tests run? 
For example, 

 is 3 days 3 hr old

Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2

[GitHub] kafka-site pull request #105: Adding Trivago logo to streams page

[guozhangwang]

Github user guozhangwang commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/105#discussion_r147801292
  
--- Diff: 0110/streams/index.html ---
@@ -17,20 +17,20 @@
 
 

[GitHub] kafka-site pull request #105: Adding Trivago logo to streams page

[miguno]

Github user miguno commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/105#discussion_r147804493
  
--- Diff: 0110/streams/index.html ---
@@ -17,20 +17,20 @@
 
 

Jenkins build is back to normal : kafka-trunk-jdk8 #2178

[Apache Jenkins Server]

See 

[GitHub] kafka-site issue #105: Adding Trivago logo to streams page

[manjuapu]

Github user manjuapu commented on the issue:

https://github.com/apache/kafka-site/pull/105
  
@guozhangwang Good catch removed it "s"


---

[GitHub] kafka-site pull request #105: Adding Trivago logo to streams page

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/105


---

[GitHub] kafka-site issue #105: Adding Trivago logo to streams page

[guozhangwang]

Github user guozhangwang commented on the issue:

https://github.com/apache/kafka-site/pull/105
  
LGTM. Merged to asf-site.


---

Jenkins build is back to normal : kafka-trunk-jdk7 #2933

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6147) Error reading field 'api_versions': Error reading field 'max_version': java.nio.BufferUnderflowException

[Sandro Simas (JIRA)]

Sandro Simas created KAFKA-6147:
---

 Summary: Error reading field 'api_versions': Error reading field 
'max_version': java.nio.BufferUnderflowException
 Key: KAFKA-6147
 URL: https://issues.apache.org/jira/browse/KAFKA-6147
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.11.0.0
Reporter: Sandro Simas


I'm getting the following error on my kafka client 0.11.0.1:
{code:java}
[2017-10-30 19:18:30] ERROR o.a.k.c.producer.internals.Sender - Uncaught error 
in kafka producer I/O thread: 
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
'api_versions': Error reading field 'max_version': 
java.nio.BufferUnderflowException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:75)
at 
org.apache.kafka.common.protocol.ApiKeys.parseResponse(ApiKeys.java:163)
at 
org.apache.kafka.common.protocol.ApiKeys$1.parseResponse(ApiKeys.java:54)
at 
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:560)
at 
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
at java.lang.Thread.run(Thread.java:745)
{code}

Another similar error:
{code:java}
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
'api_versions': Error reading array of size 65546, only 10 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:75)
at 
org.apache.kafka.common.protocol.ApiKeys.parseResponse(ApiKeys.java:163)
at 
org.apache.kafka.common.protocol.ApiKeys$1.parseResponse(ApiKeys.java:54)
at 
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:560)
at 
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
at java.lang.Thread.run(Thread.java:745)
{code}

The server is also 0.11.0.1 and I'm running Kafka and ZooKeeper without any 
previous data. This errors appears suddenly even without producing messages.
Although this error occurs, I can produce messages without any problems after 
this error.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4160: Customer logo stream

[manjuapu]

GitHub user manjuapu opened a pull request:

https://github.com/apache/kafka/pull/4160

Customer logo stream

@guozhangwang Please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka customer-logo-stream

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4160.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4160


commit dd4789b35187b4820f67d092011c85e9bebd5a10
Author: Manjula K 
Date:   2017-09-20T00:51:44Z

Adding See how Kafka Streams is being used section to Streams page

commit c5382a47fa25e115c245845cf69173399497ca93
Author: manjuapu 
Date:   2017-09-20T15:23:07Z

Update index.html

commit c7c9e8a9d04cf63673d885d9bd6f357fc4d84cf2
Author: Manjula K 
Date:   2017-09-21T22:30:33Z

Adding LINE corp logo to Streams page

commit 0062cdac3a39509a4e6203a58350a662282f6408
Author: Manjula K 
Date:   2017-09-21T22:57:51Z

Resolving conflict in docs/streams/index.html

commit 6958aee923fe9a8bd8d819a1d45562ad703ba93a
Author: Manjula K 
Date:   2017-09-23T01:42:32Z

MINOR:Adding New York Logo to streams page

commit 808613e8b50b1b585c1dd122c718ed171a305b8e
Author: manjuapu 
Date:   2017-09-23T01:45:43Z

Merge branch 'trunk' into customer-logo-stream

commit 6a0c30498536c1865b343bdc6d08a0bcd0ebddcf
Author: manjuapu 
Date:   2017-09-23T02:46:28Z

Update index.html

commit 7dca97769e0975190bc27ef29bae39692a2fc9b2
Author: Manjula K 
Date:   2017-09-26T16:48:26Z

MINOR:Updated Rabobank description

commit da1293d3d7eb3a356aab975f7a6594be4abb2952
Author: manjuapu 
Date:   2017-09-26T17:57:49Z

Merge branch 'trunk' into customer-logo-stream

commit 26a97ac3486555d6ca968e220ad3b82d04003d62
Author: Manjula K 
Date:   2017-10-30T19:51:14Z

Lighthouse logos and navbar




---

[GitHub] kafka pull request #4128: MINOR: random cleanup and JavaDoc improvements for...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4128


---

Build failed in Jenkins: kafka-trunk-jdk9 #161

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Code cleanup and JavaDoc improvements for clients and Streams

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H25 (couchdbtest ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision c7ab3efcbe5d34c28e19a5a6a59962c2abfd2235 
(refs/remotes/origin/trunk)
Commit message: "MINOR: Code cleanup and JavaDoc improvements for clients and 
Streams"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c7ab3efcbe5d34c28e19a5a6a59962c2abfd2235
 > git rev-list 7fe88e8bd9acedcb74a4b3e61440f61481b60711 # timeout=10
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins7501979839922454648.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2/bin/gradle

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine java version from '9.0.1'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did tests run? 
For example, 

 is 3 days 5 hr old

Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
Not sending mail to unregistered user wangg...@gmail.com

[GitHub] kafka pull request #4160: Customer logo stream

[manjuapu]

Github user manjuapu closed the pull request at:

https://github.com/apache/kafka/pull/4160


---

[VOTE] KIP-208: Add SSL support to Kafka Connect REST interface

[Jakub Scholz]

Hi,

It seems there are no more comments for this KIP, so I would like to start
the voting .

For more details about the KIP-208 go to
*https://cwiki.apache.org/confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
*

Thanks & Regards
Jakub

[GitHub] kafka pull request #4161: Adding lighthouse logos and nav bar

[manjuapu]

GitHub user manjuapu opened a pull request:

https://github.com/apache/kafka/pull/4161

Adding lighthouse logos and nav bar

@guozhangwang Please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka lighthouse-logo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4161


commit e8edd7e53aaacb2131133a2d4ec6aff51339625c
Author: Manjula K 
Date:   2017-10-30T20:28:19Z

Adding lighthouse logos and nav bar




---

Re: [VOTE] KIP-205: Add all() and range() API to ReadOnlyWindowStore

[Gwen Shapira]

+1 for the KIP and Rename

On Sat, Oct 28, 2017 at 5:51 AM Bill Bejeck  wrote:

> +1 for the KIP and +1 on renaming "range" to "fetchAll"
>
> Thanks,
> Bill
>
> On Thu, Oct 26, 2017 at 4:39 PM, Guozhang Wang  wrote:
>
> > BTW, Xavier has an argument to rename "range" as it does not take a key
> or
> > key range at all. I'd propose we rename this function to "fetchAll".
> >
> > Guozhang
> >
> > On Wed, Oct 25, 2017 at 7:27 AM, Damian Guy 
> wrote:
> >
> > > +1
> > >
> > > On Tue, 24 Oct 2017 at 16:46 Guozhang Wang  wrote:
> > >
> > > > +1. Thanks.
> > > >
> > > > On Mon, Oct 23, 2017 at 8:11 PM, Richard Yu <
> > yohan.richard...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I want to propose KIP-205 for the addition of new API. It is about
> > > adding
> > > > > methods similar to those found in ReadOnlyKeyValueStore to the
> > > > > ReadOnlyWindowStore class. As it appears the discussion has
> reached a
> > > > > conclusion, I would like to start the voting process.
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 205%3A+Add+all%28%29+and+range%28%29+API+to+ReadOnlyWindowStore
> > > > >
> > > > > Thanks for your patience!
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Build failed in Jenkins: kafka-trunk-jdk8 #2179

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-6130; Ensure VerifiableConsumer halts when --max-messages is

[jason] KAFKA-2903; FileRecords.read doesn't handle size > sizeInBytes when

[jason] KAFKA-5212; Consumer ListOffsets request can starve group heartbeats

--
[...truncated 3.31 MB...]
kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandlerWithHeaders 
STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandlerWithHeaders 
PASSED

kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs STARTED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.tools.ReplicaVerificationToolTest > testReplicaBufferVerifyChecksum 
STARTED

kafka.tools.ReplicaVerificationToolTest > testReplicaBufferVerifyChecksum PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetSmallest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetSmallest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithNoOffsetReset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithNoOffsetReset PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured STARTED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
STARTED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer STARTED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.tools.ConsumerPerformanceTest > testDetailedHeaderMatchBody STARTED

kafka.tools.ConsumerPerformanceTest > testDetailedHeaderMatchBody PASSED

kafka.tools.ConsumerPerformanceTest > testNonDetailedHeaderMatchBody STARTED

kafka.tools.ConsumerPerformanceTest > testN

[jira] [Resolved] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-4084.

   Resolution: Fixed
Fix Version/s: 1.1.0

The auto leader balancing logic now uses the async ZK api and batches the 
requests from the controller to the brokers. So, the process should be much 
faster with many partitions. Closing this for now.

> automated leader rebalance causes replication downtime for clusters with too 
> many partitions
> 
>
> Key: KAFKA-4084
> URL: https://issues.apache.org/jira/browse/KAFKA-4084
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8.2.2, 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Tom Crayford
>  Labels: reliability
> Fix For: 1.1.0
>
>
> If you enable {{auto.leader.rebalance.enable}} (which is on by default), and 
> you have a cluster with many partitions, there is a severe amount of 
> replication downtime following a restart. This causes 
> `UnderReplicatedPartitions` to fire, and replication is paused.
> This is because the current automated leader rebalance mechanism changes 
> leaders for *all* imbalanced partitions at once, instead of doing it 
> gradually. This effectively stops all replica fetchers in the cluster 
> (assuming there are enough imbalanced partitions), and restarts them. This 
> can take minutes on busy clusters, during which no replication is happening 
> and user data is at risk. Clients with {{acks=-1}} also see issues at this 
> time, because replication is effectively stalled.
> To quote Todd Palino from the mailing list:
> bq. There is an admin CLI command to trigger the preferred replica election 
> manually. There is also a broker configuration “auto.leader.rebalance.enable” 
> which you can set to have the broker automatically perform the PLE when 
> needed. DO NOT USE THIS OPTION. There are serious performance issues when 
> doing so, especially on larger clusters. It needs some development work that 
> has not been fully identified yet.
> This setting is extremely useful for smaller clusters, but with high 
> partition counts causes the huge issues stated above.
> One potential fix could be adding a new configuration for the number of 
> partitions to do automated leader rebalancing for at once, and *stop* once 
> that number of leader rebalances are in flight, until they're done. There may 
> be better mechanisms, and I'd love to hear if anybody has any ideas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-6139) error when loading plugins

[Eugene Burd (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Burd resolved KAFKA-6139.

Resolution: Information Provided

Error was due to having the plugin directory set a level too low (pointing 
inside the plugin). 

> error when loading plugins
> --
>
> Key: KAFKA-6139
> URL: https://issues.apache.org/jira/browse/KAFKA-6139
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Eugene Burd
>
> I am trying to run the Big Query connector for KafkaConnect 
> (https://github.com/wepay/kafka-connect-bigquery).  I have configured it 
> using a docker container, dropped the jar files for the connector in the 
> container and configured the plugin.path in the kafka connect config to point 
> to the directory with the jars.  
> Upon startup, Kafka is scanning the plugin folder and evaluating all the jar 
> files found.  It encounters the connector jar file, tests to see if it 
> extends a connector (in this case kcbq-connector-1.0.0-SNAPSHOT.jar), but 
> then it proceeds to create a new instance of the connector.  Code snippet 
> from DelegatingClassLoader.java below.  The problem is that the connector 
> class relies on other jar files that are in the plugin folder, but not in the 
> path.  This causes a class not found exception (below). 
> I suspect when the plugin connector is actually executed, it is done through 
> an isolated context and that context's classpath is set to the plugin folder, 
> so i think the connector would actually work.  But scanning for the connector 
> fails prior to ever getting there.
> I understand that you need a version of the connector and hence why this code 
> is running, but i think the new instance creation for version check needs to 
> be done in a sandbox to support the classpaths of the plugin connector. 
> [2017-10-28 06:04:08,961] INFO Loading plugin from: 
> /usr/share/java/kafka-connect-bigquery/kcbq-connector-1.0.0-SNAPSHOT.jar 
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:176)
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/google/cloud/bigquery/BigQuery
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.newInstance(Class.java:412)
> at 
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:242)
> at 
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:223)
> at 
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
> at 
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:190)
> at 
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:150)
> at 
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
> at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:63)
> Caused by: java.lang.ClassNotFoundException: 
> com.google.cloud.bigquery.BigQuery
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at 
> org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:62)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 11 more
> -
> private  Collection> getPluginDesc( 
> Reflections reflections,
> Class klass,
> ClassLoader loader
> ) throws InstantiationException, IllegalAccessException {
> Set> plugins = reflections.getSubTypesOf(klass);
> Collection> result = new ArrayList<>();
> for (Class plugin : plugins) {
> if (PluginUtils.isConcrete(plugin)) {
> // Temporary workaround until all the plugins are versioned.
> if (Connector.class.isAssignableFrom(plugin)) {
> result.add(
> new PluginDesc<>(
> plugin,
> ((Connector) 
> plugin.newInstance()).version(),
> loader
> )
> );
> } else {
> result.add(new PluginDesc<>(plugin, "undefined", loader));
> }
> }
> }
> return result;
> }



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-207:The Offsets which ListOffsetsResponse returns should monotonically increase even during a partition leader change

[Colin McCabe]

Hi all,

With binding +1 votes from Jason Gustafson, Jun Rao, and Ismael Juma,
non-binding +1 votes from Manikumar, and no +0 or -1 votes, the vote
passes.

best,
Colin


On Fri, Oct 27, 2017, at 11:09, Colin McCabe wrote:
> Thanks, everyone.  I'm going to close the vote tomorrow if there are no
> more comments or votes.
> 
> regards,
> Colin
> 
> 
> On Thu, Oct 26, 2017, at 08:09, Manikumar wrote:
> > Thanks for the KIP.
> > +1 (non-binding)
> > 
> > 
> > On Thu, Oct 26, 2017 at 5:58 AM, Jason Gustafson 
> > wrote:
> > 
> > > +1. Thanks for the KIP.
> > >
> > > On Mon, Oct 23, 2017 at 11:30 AM, Colin McCabe  wrote:
> > >
> > > > On Mon, Oct 23, 2017, at 10:29, Jason Gustafson wrote:
> > > > > Thanks for the KIP. I'm assuming the new behavior only affects
> > > > > ListOffsets requests from the consumer.
> > > >
> > > > That's a very good point.  I will add a caveat that we only apply the
> > > > KIP-207 behavior to requests from clients, not requests from other
> > > > brokers (such as the ones made by ReplicaFetcherThread).
> > > >
> > > > > Might be worth mentioning that in the KIP.
> > > > > Also, does it affect all ListOffsets requests, or only those that
> > > specify
> > > > > the latest offset?
> > > >
> > > > I don't feel great about allowing someone to ask for the offset at time
> > > > T, get back X, and then ask again for the offset at T the next second
> > > > and get back InvalidOffsetException.  So it's probably best just to
> > > > apply the KIP-207 behavior to all ListOffsets requests from consumers.
> > > >
> > > > Thinking about it a bit more, we should disable the KIP-207 behavior
> > > > when unclean leader elections are enabled on the broker.  When unclean
> > > > leader elections are enabled, data loss is possible.  So we cannot
> > > > guarantee that offsets will always go forwards, even in theory, in this
> > > > mode.
> > > >
> > > > I update the kip-- check it out.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Oct 18, 2017 at 9:15 AM, Colin McCabe 
> > > > wrote:
> > > > >
> > > > > > On Wed, Oct 18, 2017, at 04:09, Ismael Juma wrote:
> > > > > > > Thanks for the KIP, +1 (binding). A few comments:
> > > > > > >
> > > > > > > 1. I agree with Jun about LEADER_NOT_AVAILABLE for the error code
> > > for
> > > > > > > older
> > > > > > > versions.
> > > > > > > 2. OffsetNotAvailableException seems clear enough (i.e. we don't
> > > > need the
> > > > > > > "ForPartition" part)
> > > > > >
> > > > > > Yeah, that is shorter and probably clearer.  Changed.
> > > > > >
> > > > > > > 3. The KIP seems to be missing the compatibility section.
> > > > > >
> > > > > > Added.
> > > > > >
> > > > > > > 4. It would be good to mention that it's now possible for a fetch
> > > to
> > > > > > > succeed while list offsets will not for a period of time. And for
> > > > older
> > > > > > > versions, the latter will return LeaderNotAvailable while the
> > > former
> > > > > > > would
> > > > > > > work fine, which is a bit unexpected. Not much we can do about it,
> > > > but
> > > > > > > worth mentioning it in my opinion.
> > > > > >
> > > > > > Fair enough
> > > > > >
> > > > > > cheers,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Tue, Oct 17, 2017 at 9:26 PM, Jun Rao  
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the KIP. +1. Just a minor comment. For the old client
> > > > > > requests,
> > > > > > > > would it be better to return a LEADER_NOT_AVAILABLE error
> > > instead?
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Tue, Oct 17, 2017 at 11:11 AM, Colin McCabe <
> > > cmcc...@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'd like to start the voting process for KIP-207:The  Offsets
> > > > which
> > > > > > > > > ListOffsetsResponse returns should monotonically increase even
> > > > > > during a
> > > > > > > > > partition leader change.
> > > > > > > > >
> > > > > > > > > See
> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > > 207%3A+Offsets+returned+by+ListOffsetsResponse+should+be+
> > > > > > > > > monotonically+increasing+even+during+a+partition+leader+change
> > > > > > > > > for details.
> > > > > > > > >
> > > > > > > > > The voting process will run for at least 72 hours.
> > > > > > > > >
> > > > > > > > > regards,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >

[jira] [Created] (KAFKA-6148) ERROR Commit of WorkerSinkTask{id=bigquery-connector-log-0} offsets threw an unexpected exception: (org.apache.kafka.connect.runtime.WorkerSinkTask:205) java.lang.ClassC

[Eugene Burd (JIRA)]

Eugene Burd created KAFKA-6148:
--

 Summary: ERROR Commit of 
WorkerSinkTask{id=bigquery-connector-log-0} offsets threw an unexpected 
exception:  (org.apache.kafka.connect.runtime.WorkerSinkTask:205) 
java.lang.ClassCastException: 
org.apache.kafka.clients.consumer.OffsetAndMetadata cannot be cast
 Key: KAFKA-6148
 URL: https://issues.apache.org/jira/browse/KAFKA-6148
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Eugene Burd


I am trying to run a com.wepay.kafka.connect.bigquery.BigQuerySinkConnector 
connector, but getting the following exception.  

[2017-10-30 21:48:49,007] ERROR WorkerSinkTask{id=bigquery-connector-log-0} 
Offset commit failed, rewinding to last committed offsets 
(org.apache.kafka.connect.runtime.WorkerSinkTask:311)
java.lang.ClassCastException: 
org.apache.kafka.clients.consumer.OffsetAndMetadata cannot be cast to 
org.apache.kafka.clients.consumer.OffsetAndMetadata
at 
com.wepay.kafka.connect.bigquery.BigQuerySinkTask.updateOffsets(BigQuerySinkTask.java:107)
at 
com.wepay.kafka.connect.bigquery.BigQuerySinkTask.flush(BigQuerySinkTask.java:96)
at org.apache.kafka.connect.sink.SinkTask.preCommit(SinkTask.java:117)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:305)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:164)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:148)
at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
[2017-10-30 21:48:49,012] ERROR Commit of 
WorkerSinkTask{id=bigquery-connector-log-0} offsets threw an unexpected 
exception:  (org.apache.kafka.connect.runtime.WorkerSinkTask:205)
java.lang.ClassCastException: 
org.apache.kafka.clients.consumer.OffsetAndMetadata cannot be cast to 
org.apache.kafka.clients.consumer.OffsetAndMetadata
at 
com.wepay.kafka.connect.bigquery.BigQuerySinkTask.updateOffsets(BigQuerySinkTask.java:107)
at 
com.wepay.kafka.connect.bigquery.BigQuerySinkTask.flush(BigQuerySinkTask.java:96)
at org.apache.kafka.connect.sink.SinkTask.preCommit(SinkTask.java:117)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:305)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:164)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:148)
at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

I have checked the version number of kafka client in the plug in and kafka 
connect itself and they are the same.  
- kafka-clients-0.11.0.0.jar matches

I am still suspecting a type of versioning issue.  Do you have any advice? 

Thanks. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

[Jun Rao]

Hi, Onur,

Thanks for the KIP. +1 from me.

Jun

On Sat, Oct 28, 2017 at 11:48 PM, Manikumar 
wrote:

> +1 (non-binding)
>
> On Fri, Oct 27, 2017 at 10:46 PM, Ted Yu  wrote:
>
> > +1
> >
> > On Fri, Oct 27, 2017 at 10:15 AM, Onur Karaman <
> > onurkaraman.apa...@gmail.com
> > > wrote:
> >
> > > I'd like to start the vote for KIP-214: Add
> > > zookeeper.max.in.flight.requests config to the broker
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 214%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker
> > >
> > > - Onur
> > >
> >
>

Re: [VOTE] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

[Ismael Juma]

Thanks for the KIP, +1 (binding).

On 27 Oct 2017 6:15 pm, "Onur Karaman"  wrote:

> I'd like to start the vote for KIP-214: Add
> zookeeper.max.in.flight.requests config to the broker
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 214%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker
>
> - Onur
>

[GitHub] kafka pull request #4162: [KAFKA-4499] Add "getAllKeys" for querying windowe...

[ConcurrencyPractitioner]

GitHub user ConcurrencyPractitioner opened a pull request:

https://github.com/apache/kafka/pull/4162

[KAFKA-4499]  Add "getAllKeys" for querying windowed KTable stores 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ConcurrencyPractitioner/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4162.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4162


commit 5635d189bb1f0a71f831a44626c206f2360c2f5e
Author: RichardYuSTUG 
Date:   2017-10-30T22:07:27Z

[KAFKA-4499] KafkaKAFKA-4499




---

Re: [VOTE] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

[Jeff Widman]

+1 (non-binding)

Thanks for putting the work in to benchmark various defaults.

On Mon, Oct 30, 2017 at 3:05 PM, Ismael Juma  wrote:

> Thanks for the KIP, +1 (binding).
>
> On 27 Oct 2017 6:15 pm, "Onur Karaman" 
> wrote:
>
> > I'd like to start the vote for KIP-214: Add
> > zookeeper.max.in.flight.requests config to the broker
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 214%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker
> >
> > - Onur
> >
>



-- 

*Jeff Widman*
jeffwidman.com  | 740-WIDMAN-J (943-6265)
<><

[jira] [Created] (KAFKA-6149) LogCleanerManager should include topic partition name when warning of invalid cleaner offset

[Ryan P (JIRA)]

Ryan P created KAFKA-6149:
-

 Summary: LogCleanerManager should include topic partition name 
when warning of invalid cleaner offset 
 Key: KAFKA-6149
 URL: https://issues.apache.org/jira/browse/KAFKA-6149
 Project: Kafka
  Issue Type: Improvement
Reporter: Ryan P


The following message would be a lot more helpful if the topic partition name 
were included.


if (!isCompactAndDelete(log))
  warn(s"Resetting first dirty offset to log start offset 
$logStartOffset since the checkpointed offset $offset is invalid.")





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Jeff Widman]

I support this as the proposed change seems both more intuitive and safer.

Right now we've essentially hacked this at my day job by bumping the offset
retention period really high, but this is a much cleaner solution.

I don't have any use-cases that require custom retention periods on a
per-group basis.

On Mon, Oct 30, 2017 at 10:15 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Bump!
>
>
>
> From:   Vahid S Hashemian/Silicon Valley/IBM
> To: dev 
> Date:   10/18/2017 04:45 PM
> Subject:[DISCUSS] KIP-211: Revise Expiration Semantics of Consumer
> Group Offsets
>
>
> Hi all,
>
> I created a KIP to address the group offset expiration issue reported in
> KAFKA-4682:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets
>
> Your feedback is welcome!
>
> Thanks.
> --Vahid
>
>
>
>


-- 

*Jeff Widman*
jeffwidman.com  | 740-WIDMAN-J (943-6265)
<><

Question about fromLogical method in connect/api/src/main/java/org/apache/kafka/connect/data/Date.java

[CharSyam]

Hi here.

I have a question about fromLogical method in
connect/api/src/main/java/org/apache/kafka/connect/data/Date.java

below code does these step.
1. calendar with UTC timezone
2. set value
3. check hour, minute, second, millisecond are 0. if not throw DataException.

```
public static int fromLogical(Schema schema, java.util.Date value) {
if (schema.name() == null || !(schema.name().equals(LOGICAL_NAME)))
throw new DataException("Requested conversion of Date object
but the schema does not match.");
Calendar calendar = Calendar.getInstance(UTC);
calendar.setTime(value);
if (calendar.get(Calendar.HOUR_OF_DAY) != 0 ||
calendar.get(Calendar.MINUTE) != 0 ||
calendar.get(Calendar.SECOND) != 0 ||
calendar.get(Calendar.MILLISECOND) != 0) {
throw new DataException("Kafka Connect Date type should not
have any time fields set to non-zero values.");
}
long unixMillis = calendar.getTimeInMillis();
return (int) (unixMillis / MILLIS_PER_DAY);
}
```

but, if value is not made by UTC, for if we read value from other
Timezone, the Date can be represented other value, because timezone
will make difference.

so I think this code cause failure when different timezone.

I think revising with timezone info is a way to solve this.
```
public static int fromLogical(Schema schema, java.util.Date value) {
if (schema.name() == null || !(schema.name().equals(LOGICAL_NAME)))
throw new DataException("Requested conversion of Date object
but the schema does not match.");
//Using DefaultTimeZone
Calendar calendar = Calendar.getInstance(TimeZone.getDefault());
calendar.setTime(value);
if (calendar.get(Calendar.HOUR_OF_DAY) != 0 ||
calendar.get(Calendar.MINUTE) != 0 ||
calendar.get(Calendar.SECOND) != 0 ||
calendar.get(Calendar.MILLISECOND) != 0) {
throw new DataException("Kafka Connect Date type should not
have any time fields set to non-zero values.");
}

//Calculate timezone difference as reviseMillis
   long reviseMillis = calendar.get(Calendar.ZONE_OFFSET) +
calendar.get(Calendar.DST_OFFSET) / 60;

//revising with reviseMillis
long unixMillis = calendar.getTimeInMillis() + reviseMillis;
return (int) (unixMillis / MILLIS_PER_DAY);
}

public static java.util.Date toLogical(Schema schema, int value) {
if (schema.name() == null || !(schema.name().equals(LOGICAL_NAME)))
throw new DataException("Requested conversion of Date object
but the schema does not match.");

//Calculate timezone difference as reviseMillis
   Calendar calendar = Calendar.getInstance(TimeZone.getDefault());
long reviseMillis = calendar.get(Calendar.ZONE_OFFSET) +
calendar.get(Calendar.DST_OFFSET) / 60;
//revising with reviseMillis
   return new java.util.Date(value * MILLIS_PER_DAY - reviseMillis);
}
```

But I wonder, is there some reason to force this work?
to check not to have any time fields set to non-zero values?

What do you think guys?

Re: Permission to create a KIP

[Ewen Cheslack-Postava]

Not sure if someone added you and just forgot to reply, but it looks like
you already have permissions on the wiki. Just follow the process described
here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

-Ewen

On Thu, Oct 26, 2017 at 8:40 AM, Elizabeth Bennett <
elizabeth.benn...@stitchfix.com> wrote:

> Hi Kafka Devs,
> I'd like permission to create a KIP. This is following up on a discussion
> from the #connect slack channel regarding overrides for producer/consumer
> configs on a per-connector level.
>
> My confluence id is: elizabeth.bennett
>
>
> Thanks,
> Liz
>

[GitHub] kafka pull request #4072: MINOR: Add HttpMetricsReporter for system tests

[ewencp]

GitHub user ewencp reopened a pull request:

https://github.com/apache/kafka/pull/4072

MINOR: Add HttpMetricsReporter for system tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka http-metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4072.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4072


commit 8d76b41e61dce8c481b21d68f1b6e79b2d1460d6
Author: Ewen Cheslack-Postava 
Date:   2017-09-08T01:11:07Z

MINOR: Add HttpMetricsReporter for system tests

commit cb54f306999f03703c4a263d35b761337992b54e
Author: Ewen Cheslack-Postava 
Date:   2017-10-24T05:24:47Z

Rename to PushHttpMetricsReporter and add unit test

commit 28b1a76c31bba2c8017fb093efa6873658f879db
Author: Ewen Cheslack-Postava 
Date:   2017-10-24T05:26:28Z

Make HttpMetricsCollector stop properly for any Service class that properly 
calls superclass stop() methods

commit 2e2fb75c6b4a64eaeb3b25465785108be1cbbbf1
Author: Ewen Cheslack-Postava 
Date:   2017-10-24T06:27:51Z

Adjust ducker command to only require systemTestLibs

commit b2111d8f7e34495f1b7f567d6175151079a9b137
Author: Ewen Cheslack-Postava 
Date:   2017-10-24T15:37:23Z

Update docs to use systemTestLibs instead of releaseTarGz for system tests

commit 5fd5f1c4e1daae6d9262202635dc2e9fc9f1bd1d
Author: Ewen Cheslack-Postava 
Date:   2017-10-24T18:18:10Z

Use reverse SSH forwarding to ensure requests to driver can always be made

commit 397f511ecad3c82082f73cc052f16cef6dc232be
Author: Ewen Cheslack-Postava 
Date:   2017-10-25T21:55:19Z

Add longer explanation of client configs




---

[GitHub] kafka pull request #4072: MINOR: Add HttpMetricsReporter for system tests

[ewencp]

Github user ewencp closed the pull request at:

https://github.com/apache/kafka/pull/4072


---

Re: [VOTE] 1.0.0 RC4

[Jason Gustafson]

+1. Hope this is the last one.

On Fri, Oct 27, 2017 at 10:28 AM, Guozhang Wang  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fifth candidate for release of Apache Kafka 1.0.0. The main PRs
> that gets merged in after RC3 are the following:
>
> *https://github.com/apache/kafka/commit/def1a768a6301c14ad6611358716ab
> 03de04e76b
>  03de04e76b>*
>
> *https://github.com/apache/kafka/commit/b9fc0f2e6892062efa1fff0c6f7bfc
> 683c8ba7ab
>  683c8ba7ab>*
>
> *https://github.com/apache/kafka/commit/a51fdcd2ee7efbd14857448a2fb7ec
> b71531e1f9
>  b71531e1f9>*
>
> *https://github.com/apache/kafka/commit/109a60c77a56d4afed488c3ba35dc8
> 459fde15ce
>  459fde15ce>*
>
> It's worth noting that starting in this version we are using a different
> version protocol with three digits: *major.minor.bug-fix*
>
> Any and all testing is welcome, but the following areas are worth
> highlighting:
>
> 1. Client developers should verify that their clients can produce/consume
> to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
> 2. Performance and stress testing. Heroku and LinkedIn have helped with
> this in the past (and issues have been found and fixed).
> 3. End users can verify that their apps work correctly with the new
> release.
>
> This is a major version release of Apache Kafka. It includes 29 new KIPs.
> See the release notes and release plan
> (*https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=71764913
>  >*)
> for more details. A few feature highlights:
>
> * Java 9 support with significantly faster TLS and CRC32C implementations
> * JBOD improvements: disk failure only disables failed disk but not the
> broker (KIP-112/KIP-113 part I)
> * Controller improvements: reduced logging change to greatly accelerate
> admin request handling.
> * Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
> KIP-188, KIP-196)
> * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
> and drop compatibility "Evolving" annotations
>
> Release notes for the 1.0.0 release:
> *http://home.apache.org/~guozhang/kafka-1.0.0-rc4/RELEASE_NOTES.html
> *
>
>
>
> *** Please download, test and vote by Tuesday, October 31, 8pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> *http://home.apache.org/~guozhang/kafka-1.0.0-rc4/
> *
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> *http://home.apache.org/~guozhang/kafka-1.0.0-rc4/javadoc/
> *
>
> * Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc4 tag:
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> d4a3919e408e444dde5db5a261c6f912cb8475a2
>
> * Documentation:
> Note the documentation can't be pushed live due to changes that will not go
> live until the release. You can manually verify by downloading
> http://home.apache.org/~guozhang/kafka-1.0.0-rc4/
> kafka_2.11-1.0.0-site-docs.tgz
>
>
> The Jenkins builders for this RC can now be found here:
>
> System test (still running):
> *https://jenkins.confluent.io/job/system-test-kafka-1.0/18/
> *
> Unit test: *https://builds.apache.org/job/kafka-1.0-jdk7/61/
> *
>
>
> /**
>
>
> Thanks,
> -- Guozhang
>

[jira] [Created] (KAFKA-6150) Make Repartition Topics Transient

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-6150:


 Summary: Make Repartition Topics Transient
 Key: KAFKA-6150
 URL: https://issues.apache.org/jira/browse/KAFKA-6150
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang
Assignee: Guozhang Wang


Unlike changelog topics, the repartition topics could just be short-lived. 
Today users have different ways to configure them with short retention such as 
enforce a short retention period or use AppendTime for repartition topics. All 
these would be cumbersome and Streams should just do this for the users.

One way to do it is use the “purgeData” admin API (KIP-107) such that after the 
offset of the input topics are committed, if the input topics are actually 
repartition topics, we would purge the data immediately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-215: Add topic regex support for Connect sinks

[Ewen Cheslack-Postava]

I took a quick pass at the PR, looks good so far. ConfigException would
still be fine in the case you're highlighting as it's inside the framework
anyway and we'd expect a ConfigException from configure() if connectors try
to use their ConfigDef to parse an invalid config. But here I don't feel
strongly about which to use since the error message is clear anyway and
will just end up in logs / the REST API for the user to sort out.

-Ewen

On Fri, Oct 27, 2017 at 6:39 PM, Jeff Klukas  wrote:

> I've updated the KIP to use the topics.regex name and opened a WIP PR with
> an implementation that shows some additional complexity in how the
> configuration option gets passed through, affecting various public function
> signatures.
>
> I would appreciate any eyes on that for feedback on whether more design
> discussion needs to happen in the KIP.
>
> https://github.com/apache/kafka/pull/4151
>
> On Fri, Oct 27, 2017 at 7:50 AM, Jeff Klukas  wrote:
>
> > I added a note in the KIP about ConfigException being thrown. I also
> > changed the proposed default for the new config to empty string rather
> than
> > null.
> >
> > Absent a clear definition of what "common" regex syntax is, it seems an
> > undue burden to ask the user to guess at what Pattern features are safe.
> If
> > we do end up implementing a different regex style, I think it will be
> > necessary to still support the Java Pattern style long-term as an option.
> > If we want to use a different regex style as default down the road, we
> > could require "power users" of Java Pattern to enable an additional
> config
> > option to maintain compatibility.
> >
> > One additional change I might make to the KIP is that 'topics.regex'
> might
> > be a better choice for config name than 'topics.pattern'. That would be
> in
> > keeping with RegexRouter that has a 'regex' configuration option rather
> > than 'pattern'.
> >
> > On Thu, Oct 26, 2017 at 11:00 PM, Ewen Cheslack-Postava <
> e...@confluent.io
> > > wrote:
> >
> >> It's fine to be more detailed, but ConfigException is already implied
> for
> >> all other config issues as well.
> >>
> >> Default could be either null or just empty string. re: alternatives, if
> >> you
> >> wanted to be slightly more detailed (though still a bit vague) re:
> >> supported syntax, you could just say that while Pattern is used, we only
> >> guarantee support for common regular expression syntax. Not sure if
> >> there's
> >> a good way of defining what "common" syntax is.
> >>
> >> Otherwise LGTM, and thanks for helping fill in a longstanding gap!
> >>
> >> -Ewen
> >>
> >> On Thu, Oct 26, 2017 at 7:56 PM, Ted Yu  wrote:
> >>
> >> > bq. Users may specify only one of 'topics' or 'topics.pattern'.
> >> >
> >> > Can you fill in which exception would be thrown if both of them are
> >> > specified
> >> > ?
> >> >
> >> > Cheers
> >> >
> >> > On Thu, Oct 26, 2017 at 6:27 PM, Jeff Klukas  wrote:
> >> >
> >> > > Looking for feedback on
> >> > >
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> >> > >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] KIP-217: Expose a timeout to allow an expired ZK session to be re-created

[Jun Rao]

Hi, Stephane,

Thanks for the reply.

1) Fixing the issue in ZK will be ideal. Not sure when it will happen
though. Once it's fixed, we can probably deprecate this config.

2) That could be useful. Is there a java api to do that at runtime? Also,
invalidating DNS cache doesn't always fix the issue of unresolved host. In
some of the cases, human intervention is needed.

3) The default timeout is infinite though.

Jun


On Sat, Oct 28, 2017 at 11:48 PM, Stephane Maarek <
steph...@simplemachines.com.au> wrote:

> Hi Jun,
>
> I think this is very helpful. Restarting Kafka brokers in case of zookeeper
> host change is not a well known operation.
>
> Few questions:
> 1) would it not be worth fixing the problem at the source ? This has been
> stuck for a while though, maybe a little push would help :
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/ZOOKEEPER-2184
>
> 2) upon recreating the zookeeper object , is it not possible to invalidate
> the DNS cache so that it resolves the new hostname?
>
> 3) could the cluster be down in this situation: one migrates an entire
> zookeeper cluster to new machines (one by one). The quorum is still alive
> without downtime, but now every broker in a cluster can't resolve zookeeper
> at the same time. They all shut down at the same time after the new
> time-out setting.
>
> Thanks !
> Stéphane
>
> On 28 Oct. 2017 9:42 am, "Jun Rao"  wrote:
>
> > Hi, Everyone,
> >
> > We created "KIP-217: Expose a timeout to allow an expired ZK session to
> be
> > re-created".
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 217%3A+Expose+a+timeout+to+allow+an+expired+ZK+session+to+be+re-created
> >
> > Please take a look and provide your feedback.
> >
> > Thanks,
> >
> > Jun
> >
>

Build failed in Jenkins: kafka-trunk-jdk8 #2180

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Code cleanup and JavaDoc improvements for clients and Streams

--
[...truncated 3.79 MB...]
org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalance STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalance PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinAssignment STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinAssignment PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalanceFailedConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalanceFailedConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testHaltCleansUpWorker STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testHaltCleansUpWorker PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorNameConflictsWithWorkerGroupId STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorNameConflictsWithWorkerGroupId PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToLeader STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRequestProcessingOrder STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRequestProcessingOrder PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testUnknownConnectorPaused STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testUnknownConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPausedRunningTaskOnly STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPausedRunningTaskOnly PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumedRunningTaskOnly STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumedRunningTaskOnly PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testTaskConfigAdded STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testTaskConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinLeaderCatchUpFails STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinLeaderCatchUpFails PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testInconsistentConfigs STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testInconsistentConfigs PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnect

MetricsReporter does not get auto-generated client.id

[Kevin Lu]

I have already created a Jira for this, but have not received any comments
so bringing this to email.

https://issues.apache.org/jira/browse/KAFKA-6123


Interceptors and MetricsReporters configured for a client will receive the
user-specified configurations via the configure() method.

When a user does not specify the client.id field, Kafka will automatically
generate a client id for the client (producer-1, producer-2, consumer-1,
consumer-2, etc). This auto-generated client id is given to the
interceptors in the configure method, but it is not given to the metric
reporters configured for the client.

This makes it hard to directly map a client's configured metrics reporters
with the configured interceptors when the client.id field is not specified
(you would have to fetch a MetricName object that has client id as a tag
but that is hacky).

The client.id field is already generated by the time the metrics reporters
are configured.

Is there a particular reason why the MetricsReporter does not also receive
the auto-generated client id?


If there are no issues with this suggestion, then I would love to work on
this as my first commit!

Regards,
Kevin

Mutation/upgrade of kafka message logs

[Feng Xu]

Hi,

I understand in most of cases, we should keep log immutable. However, when
it goes to event sourcing (Ben Stopford
 and Neha Narkhede
 have a series of
posts:  Messaging as the Single Source of Truth
, Using
Apache Kafka as a Scalable, Event-Driven Backbone for Service Architectures

and Event
sourcing, CQRS, stream processing and Apache Kafka: What’s the connection
),
we need to evolve the messages -- upgrade the message from lower version to
newer version. Regarding evolve of the message, Michiel Overeem, etc have
documented the patterns in the paper:
http://files.movereem.nl/2017saner-eventsourcing.pdf

So, there's a strong requirement when kafka is serving as event sourcing.
Is there any plan that kafka will support mutation of logs? If it's too
dangerous to update message online, at least provide a way to upgrade the
messages in batch.

-- 
Regards,
iVen

Re: Question about fromLogical method in connect/api/src/main/java/org/apache/kafka/connect/data/Date.java

[Dongjin Lee]

CharSyam,

As I understood, it seems like Date#fromLogical assumes that the given Data
object is created under UTC timezone - its unit test, DateTest, creates all
related GregorianCalendar objects with TimeZone.getTimeZone("UTC"). So if
all the Date objects given as the parameter are UTC-oriented, it seems like
this implementation makes any problem. So: Have ever experienced the case
of non-UTC Date object? I inspected all classes implementing Transformation
interface[^1] but all the classes related to Date or Time[^2] were using
UTC-oriented Date object. (check the `TimeZone.getTimeZone("UTC")` clause
from the code.) If there is a logical crack, it will worth a fix.

Thanks,
Dongjin

[^1]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Single+Message+Transforms+for+Kafka+Connect
[^2]: TimestampRouter, TimestampConverter.

On Mon, Oct 30, 2017 at 4:45 PM, CharSyam  wrote:

> Hi here.
>
> I have a question about fromLogical method in
> connect/api/src/main/java/org/apache/kafka/connect/data/Date.java
>
> below code does these step.
> 1. calendar with UTC timezone
> 2. set value
> 3. check hour, minute, second, millisecond are 0. if not throw
> DataException.
>
> ```
> public static int fromLogical(Schema schema, java.util.Date value) {
> if (schema.name() == null || !(schema.name().equals(LOGICAL_NAME)))
> throw new DataException("Requested conversion of Date object
> but the schema does not match.");
> Calendar calendar = Calendar.getInstance(UTC);
> calendar.setTime(value);
> if (calendar.get(Calendar.HOUR_OF_DAY) != 0 ||
> calendar.get(Calendar.MINUTE) != 0 ||
> calendar.get(Calendar.SECOND) != 0 ||
> calendar.get(Calendar.MILLISECOND) != 0) {
> throw new DataException("Kafka Connect Date type should not
> have any time fields set to non-zero values.");
> }
> long unixMillis = calendar.getTimeInMillis();
> return (int) (unixMillis / MILLIS_PER_DAY);
> }
> ```
>
> but, if value is not made by UTC, for if we read value from other
> Timezone, the Date can be represented other value, because timezone
> will make difference.
>
> so I think this code cause failure when different timezone.
>
> I think revising with timezone info is a way to solve this.
> ```
> public static int fromLogical(Schema schema, java.util.Date value) {
> if (schema.name() == null || !(schema.name().equals(LOGICAL_NAME)))
> throw new DataException("Requested conversion of Date object
> but the schema does not match.");
> //Using DefaultTimeZone
> Calendar calendar = Calendar.getInstance(TimeZone.getDefault());
> calendar.setTime(value);
> if (calendar.get(Calendar.HOUR_OF_DAY) != 0 ||
> calendar.get(Calendar.MINUTE) != 0 ||
> calendar.get(Calendar.SECOND) != 0 ||
> calendar.get(Calendar.MILLISECOND) != 0) {
> throw new DataException("Kafka Connect Date type should not
> have any time fields set to non-zero values.");
> }
>
> //Calculate timezone difference as reviseMillis
>long reviseMillis = calendar.get(Calendar.ZONE_OFFSET) +
> calendar.get(Calendar.DST_OFFSET) / 60;
>
> //revising with reviseMillis
> long unixMillis = calendar.getTimeInMillis() + reviseMillis;
> return (int) (unixMillis / MILLIS_PER_DAY);
> }
>
> public static java.util.Date toLogical(Schema schema, int value) {
> if (schema.name() == null || !(schema.name().equals(LOGICAL_NAME)))
> throw new DataException("Requested conversion of Date object
> but the schema does not match.");
>
> //Calculate timezone difference as reviseMillis
>Calendar calendar = Calendar.getInstance(TimeZone.getDefault());
> long reviseMillis = calendar.get(Calendar.ZONE_OFFSET) +
> calendar.get(Calendar.DST_OFFSET) / 60;
> //revising with reviseMillis
>return new java.util.Date(value * MILLIS_PER_DAY - reviseMillis);
> }
> ```
>
> But I wonder, is there some reason to force this work?
> to check not to have any time fields set to non-zero values?
>
> What do you think guys?
>



-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*

*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
slideshare:
www.slideshare.net/dongjinleekr
*

[jira] [Created] (KAFKA-6151) kafka broker not starting ; WARN kafka.log.Log: Found a corrupted index file

[Prince Paulson (JIRA)]

Prince Paulson created KAFKA-6151:
-

 Summary: kafka broker not starting ; WARN kafka.log.Log: Found a 
corrupted index file
 Key: KAFKA-6151
 URL: https://issues.apache.org/jira/browse/KAFKA-6151
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.9.0.1
Reporter: Prince Paulson
Priority: Critical


r...@4233106d-cceb-0517-ba7b-4917466b3ede.spherestage-kc.net:/home/SPHERESTAGE/pp023549
 ( koopa_kafkabroker01 )
00:26:30 # service kafka status
Kafka is not running


r...@4233106d-cceb-0517-ba7b-4917466b3ede.spherestage-kc.net:/var/log/kafka ( 
koopa_kafkabroker01 )
00:44:25 # service kafka start
Kafka successfully started
r...@4233106d-cceb-0517-ba7b-4917466b3ede.spherestage-kc.net:/var/log/kafka ( 
koopa_kafkabroker01 )
00:44:32 # service kafka status
Kafka is not running


r...@4233106d-cceb-0517-ba7b-4917466b3ede.spherestage-kc.net:/var/log/kafka ( 
koopa_kafkabroker01 )
00:44:39 # tail -500f /var/log/kafka/server.log

2017-10-30 00:44:33,389 INFO kafka.server.KafkaServer: starting
2017-10-30 00:44:33,392 INFO kafka.server.KafkaServer: Connecting to zookeeper 
on 
42334c1d-3c61-3f57-4a28-09df4a706fa2.spherestage-kc.net:2181,4233668f-b29f-6beb-faab-8a0eda3067ac.spherestage-kc.net:2181,4233c278-fd8c-42b3-2402-ef8be142058b.spherestage-kc.net:2181/kafka/koopa
2017-10-30 00:44:33,497 INFO kafka.server.KafkaServer: Created zookeeper path 
/kafka/koopa
2017-10-30 00:44:34,150 INFO kafka.log.LogManager: Loading logs.
2017-10-30 00:44:34,199 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/KEPLER.deveng_v500.koopa-deveng_v500-aeon-clinicalevent-MillenniumSpout.DEFERRED-0/03650694.index,
 deleting and rebuilding index...
2017-10-30 00:44:34,725 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/KEPLER.1201dep_refrecord_person_4_4-CDH5.1201dep_refrecord_person_4_4-CDH5_updated_recordIds.DEFERRED-0/.index,
 deleting and rebuilding index...
2017-10-30 00:44:34,729 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/KEPLER.v500.koopa-v500-Personnel-Spout-0/.index,
 deleting and rebuilding index...
2017-10-30 00:44:34,733 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/KEPLER.solgm_refrecord_person_4_9-CDH5.solgm_refrecord_person_4_9-CDH5_updated_recordIds.DEFERRED-0/.index,
 deleting and rebuilding index...
2017-10-30 00:44:34,737 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/refrecord_logs-18/00405326.index, 
deleting and rebuilding index...
2017-10-30 00:44:34,841 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/KEPLER.deveng_v500.koopa-deveng_v500-PrivilegeDef-Spout-0/.index,
 deleting and rebuilding index...
2017-10-30 00:44:34,844 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/KEPLER.1501stpdep_refrecord_norm_financial.refRecord-1501stpdep-koopa-1501stpdep_refrecord_norm_financial-CacheUpdaterSpout.DEFERRED-0/.index,
 deleting and rebuilding index...
2017-10-30 00:44:34,847 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/refrecord_logs-1/00404938.index, 
deleting and rebuilding index...
2017-10-30 00:44:34,881 ERROR kafka.log.LogManager: There was an error in one 
of the threads during logs loading: java.lang.InternalError: a fault occurred 
in a recent unsafe memory access operation in compiled Java code
2017-10-30 00:44:34,881 WARN kafka.log.Log: Found a corrupted index file, 
/storage/kafka/00/kafka_data/KEPLER.1201dep_v500.koopa-1201dep_v500-aeon-foundations-MillenniumSpout.DEFERRED-0/00120966.index,
 deleting and rebuilding index...
2017-10-30 00:44:34,882 FATAL kafka.server.KafkaServer: Fatal error during 
KafkaServer startup. Prepare to shutdown
java.lang.InternalError: a fault occurred in a recent unsafe memory access 
operation in compiled Java code
at 
org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:343)
at 
org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:167)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:150)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at 
kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:72)
at 
kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:64)
at 
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:38)
at kafka.log.LogSegment.recov