date:20151012


 [ 
https://issues.apache.org/jira/browse/KAFKA-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2603:
---
Reviewer: Guozhang Wang

Setting [~guozhang] as the reviewer. This change is needed in order to complete 
KAFKA-2581.

> Add timeout to ConsoleConsumer running with new consumer
> 
>
> Key: KAFKA-2603
> URL: https://issues.apache.org/jira/browse/KAFKA-2603
> Project: Kafka
>  Issue Type: Test
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> ConsoleConsumer exits when no messages are received for a timeout period when 
> run with the old consumer since the old consumer had a timeout parameter. 
> This behaviour is not available with the new consumer and hence ducktape 
> tests which rely on the timeout cannot be run with the new consumer. 
> [~granders] has suggested a solution in KAFKA-2581.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1804) Kafka network thread lacks top exception handler

2015-10-12 Thread Vladimir Picka (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953030#comment-14953030
 ] 

Vladimir Picka commented on KAFKA-1804:
---

Hello, I just would like to report that this error was thrown after 4 days run 
on Kafka version: kafka-0.8.2.2-2.11. Unfortunately we were not able to 
reproduce the error yet. I just want to increase the attention to this issue.

{code}
[2015-10-08 14:40:47,176] ERROR Uncaught exception in thread 
'kafka-network-thread-9092-1': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:524)
at kafka.network.AbstractServerThread.close(SocketServer.scala:165)
at kafka.network.AbstractServerThread.close(SocketServer.scala:157)
at kafka.network.Processor.close(SocketServer.scala:374)
at kafka.network.Processor.processNewResponses(SocketServer.scala:406)
at kafka.network.Processor.run(SocketServer.scala:318)
at java.lang.Thread.run(Thread.java:745)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] ERROR Uncaught exception in thread 
'kafka-network-thread-9092-0': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:524)
at kafka.network.AbstractServerThread.close(SocketServer.scala:165)
at kafka.network.AbstractServerThread.close(SocketServer.scala:157)
at kafka.network.Processor.close(SocketServer.scala:374)
at kafka.network.Processor.run(SocketServer.scala:350)
at java.lang.Thread.run(Thread.java:745)
{code}

> Kafka network thread lacks top exception handler
> 
>
> Key: KAFKA-1804
> URL: https://issues.apache.org/jira/browse/KAFKA-1804
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Oleg Golovin
>Priority: Critical
>
> We have faced the problem that some kafka network threads may fail, so that 
> jstack attached to Kafka process showed fewer threads than we had defined in 
> our Kafka configuration. This leads to API requests processed by this thread 
> getting stuck unresponed.
> There were no error messages in the log regarding thread failure.
> We have examined Kafka code to find out there is no top try-catch block in 
> the network thread code, which could at least log possible errors.
> Could you add top-level try-catch block for the network thread, which should 
> recover network thread in case of exception?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1804) Kafka network thread lacks top exception handler

2015-10-12 Thread Vladimir Picka (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953030#comment-14953030
 ] 

Vladimir Picka edited comment on KAFKA-1804 at 10/12/15 11:54 AM:
--

Hello, I just would like to report that this error was thrown after 4 days run 
on Kafka version: kafka-0.8.2.2-2.11. We were not attempting to shutdown. 
Unfortunately we were not able to reproduce the error yet. I just want to 
increase the attention to this issue.

{code}
[2015-10-08 14:40:47,176] ERROR Uncaught exception in thread 
'kafka-network-thread-9092-1': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:524)
at kafka.network.AbstractServerThread.close(SocketServer.scala:165)
at kafka.network.AbstractServerThread.close(SocketServer.scala:157)
at kafka.network.Processor.close(SocketServer.scala:374)
at kafka.network.Processor.processNewResponses(SocketServer.scala:406)
at kafka.network.Processor.run(SocketServer.scala:318)
at java.lang.Thread.run(Thread.java:745)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] ERROR Uncaught exception in thread 
'kafka-network-thread-9092-0': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:524)
at kafka.network.AbstractServerThread.close(SocketServer.scala:165)
at kafka.network.AbstractServerThread.close(SocketServer.scala:157)
at kafka.network.Processor.close(SocketServer.scala:374)
at kafka.network.Processor.run(SocketServer.scala:350)
at java.lang.Thread.run(Thread.java:745)
{code}


was (Author: vladap):
Hello, I just would like to report that this error was thrown after 4 days run 
on Kafka version: kafka-0.8.2.2-2.11. Unfortunately we were not able to 
reproduce the error yet. I just want to increase the attention to this issue.

{code}
[2015-10-08 14:40:47,176] ERROR Uncaught exception in thread 
'kafka-network-thread-9092-1': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:524)
at kafka.network.AbstractServerThread.close(SocketServer.scala:165)
at kafka.network.AbstractServerThread.close(SocketServer.scala:157)
at kafka.network.Processor.close(SocketServer.scala:374)
at kafka.network.Processor.processNewResponses(SocketServer.scala:406)
at kafka.network.Processor.run(SocketServer.scala:318)
at java.lang.Thread.run(Thread.java:745)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] INFO Closing socket connection to /10.33.167.154. 
(kafka.network.Processor)
[2015-10-08 14:40:47,177] ERROR Uncaught exception in thread 
'kafka-network-thread-9092-0': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:524)
at kafka.network.AbstractServerThread.close(SocketServer.scala:165)
at kafka.network.AbstractServerThread.close(SocketServer.scala:157)
at kafka.network.Processor.close(SocketServer.scala:374)
at kafka.network.Processor.run(SocketServer.scala:350)
at java.lang.Thread.run(Thread.java:745)
{code}

> Kafka network thread lacks top exception handler
> 
>
> Key: KAFKA-1804
> URL: https://issues.apache.org/jira/browse/KAFKA-1804
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Oleg Golovin
>Priority: Critical
>
> We have faced the problem that some kafka network threads may fail, so that 
> jstack attached to Kafka process showed fewer threads than we had defined in 
> our Kafka configuration. This leads to API requests processed by this thread 
> getting stuck unresponed.
> There were no error messages in the log regarding thread failure.
> We have examined Kafka code to find out there is no top

[jira] [Commented] (KAFKA-1804) Kafka network thread lacks top exception handler


[ 
https://issues.apache.org/jira/browse/KAFKA-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953062#comment-14953062
 ] 

Ismael Juma commented on KAFKA-1804:


Note that the code in trunk looks different and the following code was removed:

{code}
 catch {
case e: CancelledKeyException => {
  debug("Ignoring response for closed socket.")
  close(key)
}
  } 
{code}

There were also some fixes as part of KAFKA-2614.

Furthermore, `Processor.run` and `Acceptor.run` also have try/catch blocks in 
trunk. All of this code is in `SocketServer.scala`:

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/SocketServer.scala

So, this looks fixed to me. Could one of the people that have pointed out the 
issue in the code verify that my assessment is correct?

> Kafka network thread lacks top exception handler
> 
>
> Key: KAFKA-1804
> URL: https://issues.apache.org/jira/browse/KAFKA-1804
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Oleg Golovin
>Priority: Critical
>
> We have faced the problem that some kafka network threads may fail, so that 
> jstack attached to Kafka process showed fewer threads than we had defined in 
> our Kafka configuration. This leads to API requests processed by this thread 
> getting stuck unresponed.
> There were no error messages in the log regarding thread failure.
> We have examined Kafka code to find out there is no top try-catch block in 
> the network thread code, which could at least log possible errors.
> Could you add top-level try-catch block for the network thread, which should 
> recover network thread in case of exception?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka-no such method exception - custom partitioner

2015-10-12 Thread Willy Lie

i follow the tutorial on
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example,
but got the following error

exception in thread "main" java.lang.ClassNotFoundException:
example.producer.SimplePartitioner
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at kafka.utils.Utils$.createObject(Utils.scala:435)
at kafka.producer.Producer.(Producer.scala:61)
at kafka.javaapi.producer.Producer.(Producer.scala:26)
at TestProducer.run(TestProducer.java:41)
at Main.main(Main.java:11)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)


Here is my producer  and SimplePartitioner
. No solution in the internet related to the
problem above. anything is missing in the tutorial?
Thanks.


Willy

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

2015-10-12 Thread Rajini Sivaram

Ashish,

Thank you for doing this writeup and starting the discussion around
namespaces and multi-tenancy.

We have implemented a namespace solution on top of Kafka trunk and our
motivation to do so matches your description in KIP-37 to a large extent.
But there are some differences in our approach. The main difference is that
we are not exposing namespaces to clients. Our goal is to provide a
multi-tenant interface where each tenant has a view of Kafka that looks and
feels like a single-user Kafka cluster. This is a quick summary of our
approach. We are waiting for the SASL implementation in Kafka to complete
this work and will be happy to share the code once we have integrated with
SASL.

*Current implementation:*

*Goal*: Provide a secure multi-tenant Kafka service that works with
out-of-the-box Kafka 0.9.0.0 clients. We have a requirement to isolate
tenants so that tenants do not have any awareness of other tenants.

*Implementation*:

   - We use a modified version of kafka.server.KafkaApis.scala which
   enables a pluggable interceptor for requests and responses.
   - Our interceptor plugin adds a tenant-specific prefix to topics and
   consumer groups in requests and removes the prefix in responses. At the
   moment, we are using valid topic/group characters in prefixes so that no
   other changes are required in Kafka.
   - Tenants are identified and authenticated based on their security
   principal. We are using clientid to hold an API key temporarily while we
   wait for SASL implementation in Kafka.
   - External clients which connect using SSL+SASL only see the topics
   within their namespace. They have no access to Zookeeper.
   - Our internal clients use PLAINTEXT and do see all topics and also have
   access to Zookeeper.

*Longer term goals:*

   - Even though we started our prototype with hierarchical topics, we
   decided against it for our initial release to avoid changes to clients. But
   for the longer term, it will definitely be useful to use hierarchical
   topics to provide namespaces. In particular, we would like the option to
   use different encryption keys to encrypt data at rest for different
   tenants. If/when hierarchical topics are supported in Kafka, we would use
   the first level of the hierarchy as a tenant prefix.
   - We are currently not using quotas for tenants, but we do have a
   requirement to support quotas.

I think we can make your proposed solution work for us. But I wasn't clear
on why namespaces would be a preferred approach to full-fledged
hierarchical topics. There is no mention of consumer groups which also
requires namespaces to support multi-tenancy. We prefer to have
multi-tenant applications which aren't aware of their namespace as we have
today, but if that is something specific to our environment and the general
consensus is to make tenants namespace-aware, we may be able to work around
it.

Thoughts?

Regards,

Rajini

On Sat, Oct 10, 2015 at 10:03 AM, Magnus Edenhill 
wrote:

> Good write-up Ashish.
>
> Looking at one of the rejected alternatives got me thinking:
> "Modify request/ response formats to take namespace specifically.
>
> Solves the issue of delimiting string required in proposed approach and the
> issue with existing regex consumers.
> This definitely is the cleanest approach. However, will require lots of API
> and protocol changes. This is listed in rejected alternatives, but we
> should totally consider this as an option."
>
>
> I think we could actually achieve this without any protocol change at all
> by moving the namespace token to the common request header's ClientId
> field.
> E.g.: .. RequestHeader { ..., ClientId = "myclient@mynamespace", ... }
>
> That would provide the desired per-request namespacing and in theory the
> existing clients dont even need to be updated as long as the clientId is
> configurable (which it should be).
>
>
> My two cents,
> Magnus
>
>
> 2015-10-10 3:32 GMT+02:00 Ashish Singh :
>
> > Hey Guys,
> >
> > I just created KIP-37 for adding namespaces to Kafka.
> >
> > KIP-37
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> > >
> > tracks the proposal.
> >
> > The idea is to make Kafka support multi-tenancy via namespaces.
> >
> > Feedback and comments are welcome.
> > 
> > --
> >
> > Regards,
> > Ashish
> >
>

[jira] [Assigned] (KAFKA-2633) Default logging from tools to Stderr

2015-10-12 Thread Grant Henke (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke reassigned KAFKA-2633:
--

Assignee: Grant Henke

> Default logging from tools to Stderr
> 
>
> Key: KAFKA-2633
> URL: https://issues.apache.org/jira/browse/KAFKA-2633
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Currently the default logging for tool is stdout, this can make parsing the 
> output  difficult in cases where exceptions are thrown, or even when expected 
> logging messages are output. The most affected tool is the console-consumer 
> but others will have this issue as well. 
> Changing the output to stderr by default allows the user to redirect the 
> output to a log file without effecting the tools output.
> Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to 
> use the configuration of their choice. This Jira is to change the default to 
> be more user friendly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2633) Default logging from tools to Stderr


 [ 
https://issues.apache.org/jira/browse/KAFKA-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2633:

   Resolution: Fixed
Fix Version/s: 0.9.0.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 296
[https://github.com/apache/kafka/pull/296]

> Default logging from tools to Stderr
> 
>
> Key: KAFKA-2633
> URL: https://issues.apache.org/jira/browse/KAFKA-2633
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.0.0
>
>
> Currently the default logging for tool is stdout, this can make parsing the 
> output  difficult in cases where exceptions are thrown, or even when expected 
> logging messages are output. The most affected tool is the console-consumer 
> but others will have this issue as well. 
> Changing the output to stderr by default allows the user to redirect the 
> output to a log file without effecting the tools output.
> Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to 
> use the configuration of their choice. This Jira is to change the default to 
> be more user friendly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2633) Default logging from tools to Stderr


[ 
https://issues.apache.org/jira/browse/KAFKA-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953365#comment-14953365
 ] 

ASF GitHub Bot commented on KAFKA-2633:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/296


> Default logging from tools to Stderr
> 
>
> Key: KAFKA-2633
> URL: https://issues.apache.org/jira/browse/KAFKA-2633
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.0.0
>
>
> Currently the default logging for tool is stdout, this can make parsing the 
> output  difficult in cases where exceptions are thrown, or even when expected 
> logging messages are output. The most affected tool is the console-consumer 
> but others will have this issue as well. 
> Changing the output to stderr by default allows the user to redirect the 
> output to a log file without effecting the tools output.
> Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to 
> use the configuration of their choice. This Jira is to change the default to 
> be more user friendly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2633: Default logging from tools to Stde...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/296


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2633) Default logging from tools to Stderr

2015-10-12 Thread Grant Henke (JIRA)

Grant Henke created KAFKA-2633:
--

 Summary: Default logging from tools to Stderr
 Key: KAFKA-2633
 URL: https://issues.apache.org/jira/browse/KAFKA-2633
 Project: Kafka
  Issue Type: Bug
Reporter: Grant Henke


Currently the default logging for tool is stdout, this can make parsing the 
output  difficult in cases where exceptions are thrown, or even when expected 
logging messages are output. The most affected tool is the console-consumer but 
others will have this issue as well. 

Changing the output to stderr by default allows the user to redirect the output 
to a log file without effecting the tools output.

Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to use 
the configuration of their choice. This Jira is to change the default to be 
more user friendly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2633: Default logging from tools to Stde...

2015-10-12 Thread granthenke

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/296

KAFKA-2633: Default logging from tools to Stderr



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka tools-log4j

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/296.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #296


commit e8dd4a6cc16bace420376845b08dbe8cf949da91
Author: Grant Henke 
Date:   2015-10-12T16:51:54Z

KAFKA-2633: Default logging from tools to Stderr




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1804) Kafka network thread lacks top exception handler

2015-10-12 Thread Patrick J. McNerthney (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953370#comment-14953370
 ] 

Patrick J. McNerthney commented on KAFKA-1804:
--

It looks likely fixed in the trunk, but remains an issue in the 0.8.x line.

What is the intention with the trunk changes? Is that an entirely new major 
version? Is there a road map somewhere explaining what is going on in the trunk 
development that I missed?

> Kafka network thread lacks top exception handler
> 
>
> Key: KAFKA-1804
> URL: https://issues.apache.org/jira/browse/KAFKA-1804
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Oleg Golovin
>Priority: Critical
>
> We have faced the problem that some kafka network threads may fail, so that 
> jstack attached to Kafka process showed fewer threads than we had defined in 
> our Kafka configuration. This leads to API requests processed by this thread 
> getting stuck unresponed.
> There were no error messages in the log regarding thread failure.
> We have examined Kafka code to find out there is no top try-catch block in 
> the network thread code, which could at least log possible errors.
> Could you add top-level try-catch block for the network thread, which should 
> recover network thread in case of exception?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #16

See 

Changes:

[cshapi] KAFKA-2633; Default logging from tools to Stderr

--
[...truncated 859 lines...]
:81:
 warning: no @param for valueSerializer
public MockProducer(boolean autoComplete, Serializer keySerializer, 
Serializer valueSerializer) {
   ^
:39:
 warning: no @return
public int partition(String topic, Object key, byte[] keyBytes, Object 
value, byte[] valueBytes, Cluster cluster);
   ^
:71:
 warning: no @return
public String topic() {
  ^
:78:
 warning: no @return
public K key() {
 ^
:92:
 warning: no @return
public Integer partition() {
   ^
:44:
 warning: no @return
public long offset() {
^
:51:
 warning: no @return
public String topic() {
  ^
:58:
 warning: no @return
public int partition() {
   ^

76 warnings
:kafka-trunk-jdk8:log4j-appender:compileJavawarning: [options] bootstrap class 
path not set in conjunction with -source 1.7
1 warning

:kafka-trunk-jdk8:log4j-appender:processResources UP-TO-DATE
:kafka-trunk-jdk8:log4j-appender:classes
:kafka-trunk-jdk8:log4j-appender:jar
:kafka-trunk-jdk8:core:compileJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileScalaJava HotSpot(TM) 64-Bit Server VM warning: 
ignoring option MaxPermSize=512m; support was removed in 8.0

:78:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP) {

  ^
:259:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
if (offsetAndMetadata.commitTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

  ^
:277:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.uncleanLeaderElectionRate
^
:278:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.leaderElectionTimer
^
:380:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

Hey guys,

I think this is an important feature and one we've talked about for a
while. I really think trying to invent a new nomenclature is going to make
it hard for people to understand, though. As such I recommend we call
namespaces "directories" and denote them with '/'--this will make the
feature 1000x more understandable to people. I think we should inheret the
semantics of normal unix fs in so far as it makes sense.

In this approach we get rid of topics entirely, instead we really just have
partitions which are the equivalent of a file and retain their numeric
names, and the existing topic concept is just the first directory level but
we generalize to allow arbitrarily many more levels of nesting. This allows
categorization of data, such as /datacenter1/user-events/page-views/3 and
you can subscribe, apply configs or permissions at any level of the
hierarchy.

I'm actually not 100% such what the semantics of accessing data in
differing namespaces is in the current proposal, maybe you can clarify
Ashish? Since the point of Kafka is sharing data I think it is really
important that the grouping be just for convenience/permissions/config/etc
and that it remain possible to access multiple directories/namespaces from
the same client.

-Jay

On Fri, Oct 9, 2015 at 6:32 PM, Ashish Singh  wrote:

> Hey Guys,
>
> I just created KIP-37 for adding namespaces to Kafka.
>
> KIP-37
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> >
> tracks the proposal.
>
> The idea is to make Kafka support multi-tenancy via namespaces.
>
> Feedback and comments are welcome.
> 
> --
>
> Regards,
> Ashish
>

[jira] [Commented] (KAFKA-1804) Kafka network thread lacks top exception handler


[ 
https://issues.apache.org/jira/browse/KAFKA-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953391#comment-14953391
 ] 

Ismael Juma commented on KAFKA-1804:


trunk will be released as 0.9.0.0, see 
http://search-hadoop.com/m/uyzND1LUUpN1qRojN1 for the discussion thread where 
the decision was made to name the next release 0.9.0 instead of 0.8.3.

> Kafka network thread lacks top exception handler
> 
>
> Key: KAFKA-1804
> URL: https://issues.apache.org/jira/browse/KAFKA-1804
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Oleg Golovin
>Priority: Critical
>
> We have faced the problem that some kafka network threads may fail, so that 
> jstack attached to Kafka process showed fewer threads than we had defined in 
> our Kafka configuration. This leads to API requests processed by this thread 
> getting stuck unresponed.
> There were no error messages in the log regarding thread failure.
> We have examined Kafka code to find out there is no top try-catch block in 
> the network thread code, which could at least log possible errors.
> Could you add top-level try-catch block for the network thread, which should 
> recover network thread in case of exception?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #673

See 

Changes:

[cshapi] KAFKA-2633; Default logging from tools to Stderr

--
[...truncated 322 lines...]
:kafka-trunk-jdk7:clients:jar UP-TO-DATE
:kafka-trunk-jdk7:clients:javadoc UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:compileJava UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:processResources UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:classes UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:jar UP-TO-DATE
:kafka-trunk-jdk7:core:compileJava UP-TO-DATE
:kafka-trunk-jdk7:core:compileScala UP-TO-DATE
:kafka-trunk-jdk7:core:processResources UP-TO-DATE
:kafka-trunk-jdk7:core:classes UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:javadoc
:kafka-trunk-jdk7:core:javadoc
cache taskArtifacts.bin 
(
 is corrupt. Discarding.
:kafka-trunk-jdk7:core:javadocJar
:kafka-trunk-jdk7:core:scaladoc
[ant:scaladoc] Element 
' 
does not exist.
[ant:scaladoc] 
:277:
 warning: a pure expression does nothing in statement position; you may be 
omitting necessary parentheses
[ant:scaladoc] ControllerStats.uncleanLeaderElectionRate
[ant:scaladoc] ^
[ant:scaladoc] 
:278:
 warning: a pure expression does nothing in statement position; you may be 
omitting necessary parentheses
[ant:scaladoc] ControllerStats.leaderElectionTimer
[ant:scaladoc] ^
[ant:scaladoc] warning: there were 14 feature warning(s); re-run with -feature 
for details
[ant:scaladoc] 
:72:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#offer".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
:32:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#offer".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
:137:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#poll".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
:120:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#poll".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
:97:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#put".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
:152:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#take".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 9 warnings found
:kafka-trunk-jdk7:core:scaladocJar
:kafka-trunk-jdk7:core:docsJar
:docsJar_2_11_7
Building project 'core' with Scala version 2.11.7
:kafka-trunk-jdk7:clients:compileJavaNote: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

:kafka-trunk-jdk7:clients:processResources UP-TO-DATE
:kafka-trunk-jdk7:clients:classes
:kafka-trunk-jdk7:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk7:clients:createVersionFile
:kafka-trunk-jdk7:clients:jar
:kafka-trunk-jdk7:clients:javadoc
:kafka-trunk-jdk7:log4j-appender:compileJava
:kafka-trunk-jdk7:log4j-appender:processResources UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:classes
:kafka-trunk-jdk7:log4j-appender:jar
:kafka-trunk-jdk7:core:compileJava UP-TO-DATE
:kafka-trunk-jdk7:core:compileScala
:78:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^

[jira] [Commented] (KAFKA-2633) Default logging from tools to Stderr


[ 
https://issues.apache.org/jira/browse/KAFKA-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953349#comment-14953349
 ] 

ASF GitHub Bot commented on KAFKA-2633:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/296

KAFKA-2633: Default logging from tools to Stderr



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka tools-log4j

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/296.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #296


commit e8dd4a6cc16bace420376845b08dbe8cf949da91
Author: Grant Henke 
Date:   2015-10-12T16:51:54Z

KAFKA-2633: Default logging from tools to Stderr




> Default logging from tools to Stderr
> 
>
> Key: KAFKA-2633
> URL: https://issues.apache.org/jira/browse/KAFKA-2633
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>
> Currently the default logging for tool is stdout, this can make parsing the 
> output  difficult in cases where exceptions are thrown, or even when expected 
> logging messages are output. The most affected tool is the console-consumer 
> but others will have this issue as well. 
> Changing the output to stderr by default allows the user to redirect the 
> output to a log file without effecting the tools output.
> Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to 
> use the configuration of their choice. This Jira is to change the default to 
> be more user friendly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2633) Default logging from tools to Stderr

2015-10-12 Thread Grant Henke (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-2633:
---
Status: Patch Available  (was: Open)

> Default logging from tools to Stderr
> 
>
> Key: KAFKA-2633
> URL: https://issues.apache.org/jira/browse/KAFKA-2633
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Currently the default logging for tool is stdout, this can make parsing the 
> output  difficult in cases where exceptions are thrown, or even when expected 
> logging messages are output. The most affected tool is the console-consumer 
> but others will have this issue as well. 
> Changing the output to stderr by default allows the user to redirect the 
> output to a log file without effecting the tools output.
> Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to 
> use the configuration of their choice. This Jira is to change the default to 
> be more user friendly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2634) KafkaProducer hangs on future.get()

2015-10-12 Thread Jason Kania (JIRA)

Jason Kania created KAFKA-2634:
--

 Summary: KafkaProducer hangs on future.get()
 Key: KAFKA-2634
 URL: https://issues.apache.org/jira/browse/KAFKA-2634
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1, 0.8.2.2
 Environment: Windows 8 PC connecting to local ubuntu VM running kafka 
in docker image with topic set up having 1 partition and a replication factor 
of 1
Reporter: Jason Kania
Priority: Blocker


When running the Kafka Producer at the end of this description, the following 
problems occur:

1) If the ip address of the server running the kafka queue is incorrect or 
unavailable, the producer times out without ever indicating that the node could 
not be found. The time out exception should indicate if the node could not be 
connected to in the first place or if communication timed out during 
communication with the node.

2) In my situation, the client connects to the kafka queue according to server 
logs, but never returns from the get() call on the producer side. This get call 
too should time out indicating what the producer was waiting on when it timed 
out. At present there are no details to indicate how to debug.

The issue is a blocker in my view because there are no error details that allow 
debugging of the issue even when the log4j properties of the server are set as 
detailed as possible.

public class BrokenProducer
{
public static void main(String[] args)
{
Properties properties = new Properties();
properties.put("bootstrap.servers", ":9092");
properties.put("key.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
properties.put("value.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer producer = new KafkaProducer( 
properties );
try
{
producer.send(new ProducerRecord("myQueue", 
"aString")).get();
}
catch (Exception e)
{
e.printStackTrace();
}
producer.close();
}
}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2487) change kafka.examples.Consumer to use the new java consumer


[ 
https://issues.apache.org/jira/browse/KAFKA-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953516#comment-14953516
 ] 

ASF GitHub Bot commented on KAFKA-2487:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/297

KAFKA-2487: change kafka.examples.Consumer to use the new java consumer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-2487

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/297.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #297


commit 97fbdc78ceb9e48e32ea6108f6a4852da25cc91f
Author: Ashish Singh 
Date:   2015-09-11T22:35:47Z

KAFKA-2487: change kafka.examples.Consumer to use the new java consumer




> change kafka.examples.Consumer to use the new java consumer
> ---
>
> Key: KAFKA-2487
> URL: https://issues.apache.org/jira/browse/KAFKA-2487
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Ashish K Singh
>Priority: Blocker
> Fix For: 0.9.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2487: change kafka.examples.Consumer to ...

2015-10-12 Thread SinghAsDev

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/297

KAFKA-2487: change kafka.examples.Consumer to use the new java consumer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-2487

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/297.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #297


commit 97fbdc78ceb9e48e32ea6108f6a4852da25cc91f
Author: Ashish Singh 
Date:   2015-09-11T22:35:47Z

KAFKA-2487: change kafka.examples.Consumer to use the new java consumer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2443 Expose windowSize on Rate; KAFKA-25...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/213


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #674

See 

Changes:

[jjkoshy] KAFKA-2443 KAFKA-2567; Expose windowSize on Rate; - Throttle time 
should

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-2 (docker Ubuntu ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 9fde92f9f1f1e40ceb6b3d5ff9676ba590385825 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9fde92f9f1f1e40ceb6b3d5ff9676ba590385825
 > git rev-list fe4818e0944a9146059b02fde3fea4cc15676bf6 # timeout=10
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson3956668990009867668.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.1/userguide/gradle_daemon.html.

FAILURE: Build failed with an exception.

* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the user guide chapter on the daemon at 
http://gradle.org/docs/2.1/userguide/gradle_daemon.html
Please read below process output to find out more:
---
19:34:53.648 [main] DEBUG o.g.l.daemon.bootstrap.DaemonMain - Assuming the 
daemon was started with following jvm opts: [-XX:MaxPermSize=512m, -Xss2m, 
-Xmx1024m, -Dfile.encoding=ISO-8859-1, -Duser.country=US, -Duser.language=en, 
-Duser.variant]
19:34:54.021 [main] DEBUG o.g.l.daemon.server.DaemonServices - Creating daemon 
context with opts: [-XX:MaxPermSize=512m, -Xss2m, -Xmx1024m, 
-Dfile.encoding=ISO-8859-1, -Duser.country=US, -Duser.language=en, 
-Duser.variant]
19:34:54.082 [INFO] [org.gradle.launcher.daemon.server.Daemon] start() called 
on daemon - 
DefaultDaemonContext[uid=97cf57d0-dcae-48bf-986f-67e6f5a29bef,javaHome=/home/jenkins/tools/java/jdk1.7.0_25-32,daemonRegistryDir=/home/jenkins/.gradle/daemon,pid=4911,idleTimeout=12,daemonOpts=-XX:MaxPermSize=512m,-Xss2m,-Xmx1024m,-Dfile.encoding=ISO-8859-1,-Duser.country=US,-Duser.language=en,-Duser.variant]
19:34:54.088 [DEBUG] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] 
updating lastActivityAt to 1444678494088
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x92866654, pid=4911, tid=4136323904
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) Server VM (23.25-b01 mixed mode linux-x86 )
# Problematic frame:
# C  [libnet.so+0x14654]  _fini+0x1d0c
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/jenkins/.gradle/daemon/2.1/hs_err_pid4911.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7
ERROR: Publisher 'Publish JUnit test result report' failed: Test reports were 
found but none of them are new. Did tests run? 
For example, 

 is 6 days 23 hr old

Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7

[jira] [Updated] (KAFKA-2259) port offset_management_testsuite


 [ 
https://issues.apache.org/jira/browse/KAFKA-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2259:
---
Priority: Major  (was: Blocker)

> port offset_management_testsuite
> 
>
> Key: KAFKA-2259
> URL: https://issues.apache.org/jira/browse/KAFKA-2259
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
> Fix For: 0.9.0.0
>
>
> Port to run on ducktape



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2258) Port mirrormaker_testsuite


 [ 
https://issues.apache.org/jira/browse/KAFKA-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2258:
---
Priority: Major  (was: Blocker)

> Port mirrormaker_testsuite
> --
>
> Key: KAFKA-2258
> URL: https://issues.apache.org/jira/browse/KAFKA-2258
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
> Fix For: 0.9.0.0
>
>
> Port mirrormaker_testsuite to run on ducktape



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2257) port replication_testsuite


 [ 
https://issues.apache.org/jira/browse/KAFKA-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2257:
---
Priority: Major  (was: Blocker)

> port replication_testsuite
> --
>
> Key: KAFKA-2257
> URL: https://issues.apache.org/jira/browse/KAFKA-2257
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
> Fix For: 0.9.0.0
>
>
> Port subset of replication_testsuite to run on ducktape. Details to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2203) Get gradle build to work with Java 8


 [ 
https://issues.apache.org/jira/browse/KAFKA-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2203:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 147
[https://github.com/apache/kafka/pull/147]

> Get gradle build to work with Java 8
> 
>
> Key: KAFKA-2203
> URL: https://issues.apache.org/jira/browse/KAFKA-2203
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.1.1
>Reporter: Gaju Bhat
>Assignee: Gwen Shapira
>Priority: Minor
> Fix For: 0.9.0.0
>
> Attachments: 0001-Special-case-java-8-and-javadoc-handling.patch
>
>
> The gradle build halts because javadoc in java 8 is a lot stricter about 
> valid html.
> It might be worthwhile to special case java 8 as described 
> [here|http://blog.joda.org/2014/02/turning-off-doclint-in-jdk-8-javadoc.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2203: Getting Java8 to relax about javad...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/147


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2203) Get gradle build to work with Java 8


[ 
https://issues.apache.org/jira/browse/KAFKA-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953687#comment-14953687
 ] 

ASF GitHub Bot commented on KAFKA-2203:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/147


> Get gradle build to work with Java 8
> 
>
> Key: KAFKA-2203
> URL: https://issues.apache.org/jira/browse/KAFKA-2203
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.1.1
>Reporter: Gaju Bhat
>Assignee: Gwen Shapira
>Priority: Minor
> Fix For: 0.9.0.0
>
> Attachments: 0001-Special-case-java-8-and-javadoc-handling.patch
>
>
> The gradle build halts because javadoc in java 8 is a lot stricter about 
> valid html.
> It might be worthwhile to special case java 8 as described 
> [here|http://blog.joda.org/2014/02/turning-off-doclint-in-jdk-8-javadoc.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka 0.9.0 release branch

2015-10-12 Thread Jun Rao

Hi, Everyone,

As we are getting closer to the 0.9.0 release, we plan to cut an 0.9.0
release branch in about two weeks from now. In the meantime, we will try to
resolve most if not all 0.9.0 blockers listed below.

https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC

Are there any concerns? We will also discuss the 0.9.0 release in
tomorrow's KIP meeting.

Thanks,

Jun

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover


[ 
https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953683#comment-14953683
 ] 

Guozhang Wang commented on KAFKA-2017:
--

[~junrao] I agree this should be in 0.9.0 release, could you take a look at the 
current patch's ZK format then?

> Persist Coordinator State for Coordinator Failover
> --
>
> Key: KAFKA-2017
> URL: https://issues.apache.org/jira/browse/KAFKA-2017
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Onur Karaman
>Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2017.patch, KAFKA-2017_2015-05-20_09:13:39.patch, 
> KAFKA-2017_2015-05-21_19:02:47.patch
>
>
> When a coordinator fails, the group membership protocol tries to failover to 
> a new coordinator without forcing all the consumers rejoin their groups. This 
> is possible if the coordinator persists its state so that the state can be 
> transferred during coordinator failover. This state consists of most of the 
> information in GroupRegistry and ConsumerRegistry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2295) Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer


[ 
https://issues.apache.org/jira/browse/KAFKA-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953689#comment-14953689
 ] 

Guozhang Wang commented on KAFKA-2295:
--

Pinging [~tdas] once again for validation, or if someone else and verify the 
issue was reproducible and now fixed with the patch, we can go ahead and check 
it in.

> Dynamically loaded classes (encoders, etc.) may not be found by Kafka 
> Producer 
> ---
>
> Key: KAFKA-2295
> URL: https://issues.apache.org/jira/browse/KAFKA-2295
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Reporter: Tathagata Das
>Assignee: Manikumar Reddy
>Priority: Blocker
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2295.patch, KAFKA-2295_2015-07-06_11:32:58.patch, 
> KAFKA-2295_2015-08-20_17:44:56.patch
>
>
> Kafka Producer (via CoreUtils.createObject) effectively uses Class.forName to 
> load encoder classes. Class.forName is by design finds classes only in the 
> defining classloader of the enclosing class (which is often the bootstrap 
> class loader). It does not use the current thread context class loader. This 
> can lead to problems in environments where classes are dynamically loaded and 
> therefore may not be present in the bootstrap classloader.
> This leads to ClassNotFound Exceptions in environments like Spark where 
> classes are loaded dynamically using custom classloaders. Issues like this 
> have reported. E.g. - 
> https://www.mail-archive.com/user@spark.apache.org/msg30951.html
> Other references regarding this issue with Class.forName 
> http://stackoverflow.com/questions/21749741/though-my-class-was-loaded-class-forname-throws-classnotfoundexception
> This is a problem we have faced repeatedly in Apache Spark and we solved it 
> by explicitly specifying the class loader to use. See 
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L178



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #17

See

Kafka KIP meeting Oct 13 at 11:00am PST

2015-10-12 Thread Jun Rao

Hi, Everyone,

We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to
attend but haven't received an invite, please let me know. The following is
the agenda.

Agenda:
1. Kafka 0.9.0 release

and if time permits,
2. KIP-32: Add CreateTime and LogAppendTime to Kafka message
3. KIP-33: Add a time based log index
4. KIP-36: Add rack-aware support

Thanks,

Jun

[jira] [Updated] (KAFKA-2515) handle oversized messages properly in new consumer


 [ 
https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2515:
---
Priority: Blocker  (was: Major)

> handle oversized messages properly in new consumer
> --
>
> Key: KAFKA-2515
> URL: https://issues.apache.org/jira/browse/KAFKA-2515
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Reporter: Jun Rao
>Assignee: Onur Karaman
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> When there is an oversized message in the broker, it seems that the new 
> consumer just silently gets stuck. We should at least log an error when this 
> happens.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2443) Expose windowSize on Rate


[ 
https://issues.apache.org/jira/browse/KAFKA-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953597#comment-14953597
 ] 

ASF GitHub Bot commented on KAFKA-2443:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/213


> Expose windowSize on Rate
> -
>
> Key: KAFKA-2443
> URL: https://issues.apache.org/jira/browse/KAFKA-2443
> Project: Kafka
>  Issue Type: Task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
> Fix For: 0.9.0.0
>
>
> Currently, we dont have a means to measure the size of the metric window 
> since the final sample can be incomplete.
> Expose windowSize(now) on Rate



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2567) throttle-time shouldn't be NaN

2015-10-12 Thread Joel Koshy (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy resolved KAFKA-2567.
---
Resolution: Fixed

Issue resolved by pull request 213
[https://github.com/apache/kafka/pull/213]

> throttle-time shouldn't be NaN
> --
>
> Key: KAFKA-2567
> URL: https://issues.apache.org/jira/browse/KAFKA-2567
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
>Assignee: Aditya Auradkar
>Priority: Minor
>  Labels: quotas
> Fix For: 0.9.0.0
>
>
> Currently, if throttling never happens, we get the NaN for throttle-time. It 
> seems it's better to default to 0.
> "kafka.server:client-id=eventsimgroup200343,type=Fetch" : { "byte-rate": 0.0, 
> "throttle-time": NaN }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2443) Expose windowSize on Rate

2015-10-12 Thread Joel Koshy (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy resolved KAFKA-2443.
---
   Resolution: Fixed
Fix Version/s: 0.9.0.0

Issue resolved by pull request 213
[https://github.com/apache/kafka/pull/213]

> Expose windowSize on Rate
> -
>
> Key: KAFKA-2443
> URL: https://issues.apache.org/jira/browse/KAFKA-2443
> Project: Kafka
>  Issue Type: Task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
> Fix For: 0.9.0.0
>
>
> Currently, we dont have a means to measure the size of the metric window 
> since the final sample can be incomplete.
> Expose windowSize(now) on Rate



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2631) Expose data volume via JMX

2015-10-12 Thread Eno Thereska (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska resolved KAFKA-2631.
-
Resolution: Implemented

> Expose data volume via JMX
> --
>
> Key: KAFKA-2631
> URL: https://issues.apache.org/jira/browse/KAFKA-2631
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8.2.1
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.8.2.1
>
>
> It would be good to expose the volume of data stored on a broker. This must 
> include only the volume created by a client and must not include the volume 
> of data created due to replication. In practice this means only a leader 
> needs to keep track of the volume of the data received by a producer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

2015-10-12 Thread Mike Yoder (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953542#comment-14953542
]

Mike Yoder commented on KAFKA-2629:
---

Hi - I work with [~singhashish] so I thought I'd jump in here.

{quote}
While flexibility is generally a good thing, in my opinion the security
consequences in real-world of this approach needs to be carefully evaluated.
{quote}
Appreciate the feedback. You're right, security is hard.

{quote}
My question is that why this approach provides more security? Why trusting the
operating system kernel is insufficient in this case? Why bringing in yet
another another application to the trusted computing base? How does it widen
the attack surfaces? What are the kinds of attacks that this approach prevents,
but putting permissions on top of the file fails to do so?
{quote}
All reasonable questions. First, let's step back and realize that this is only
an _interface_ to getting a password, not the implementation behind said
interface. Many of your questions really deserve to be asked of the
implementations.

That being said, running an executable is a little different than ordinary file
access. However I don't believe that it adds risk to the overall security of
the application.

Yes, the specification of the executable to run is in a config file - but if an
attacker controls the config file itself, we have far bigger problems than them
being able to run some program. Likewise, if an attacker can control the
executable that's being run, then bad things can happen - but that's bad
because the implementation has somehow been screwed up, or because the attacker
has control of things that defeat other security measures as well.

I think that the benefit of this approach is that it _enables_ things that
aren't possible with a password simply sitting in a config file. Primarily it
enables a use case where a centralized password store is contacted, and a
password dispensed. The benefit with this is that the password store can audit
usage, rotate passwords, and provide centralized management.

{quote}
Hadoop implemented something called the CredentialProvider specifically for the
purpose of encrypting passwords. See
https://issues.apache.org/jira/browse/HADOOP-10607. This functionality is now
supported by other projects, including Hive, HBase, etc.
I believe the Hadoop jira is irrelevant. This is specifically tailored to hook
with services like Apache Ranger and Apache Sentry – all of which for user
authentication / authorization.
{quote}
I disagree. The implementation of the CredentialProvider is hooked into all the
"sensitive" parameters of Hadoop. It's not just for the authentication of
users.

One example is here:
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java#L155
This is for core hadoop ssl's keystore password. The getPassword() call goes to
the CredentialProvider interface for passwords.

The CredentialProvider is an interface; the implementation can do pretty much
anything it wants, including something like what is described in this jira. The
popular implementation gets a password from a jceks file.

An alternative implementation of this jira would be to build something like the
CredentialProvider for Kafka. That's a bigger project... but accomplishes
similar goals.

{quote}
8.2.1 Using strong cryptography, render all authentication credentials (such as
passwords/phrases) unreadable during transmission and storage on all system
components.

Please correct me if I'm wrong, it looks to me that you misread the standard?
The statement you specifically refers to user authentication services. The
password in SSL keystore serves as a master keyphrase to protect the private
key stored in the keystore.
{quote}
That's an interesting distinction that I have not seen others make. I admit
that differentiating between passwords used for human authentication and
passwords used to encrypt things in a keystore is not one that occurred to me.
In a way, you're right, they are somewhat different use cases, and I could be
accused of misreading the standard.

HOWEVER, I know many organizations who interpreted the standard like I did: a
password is a password, and shall not be suffered to exist in plain text on a
disk. There are corporations out there with quite rigid blanket policies
regarding this. Auditors, in particular, do not have a sense of humor at all
about such things. So many others could be accused of misreading the standard,
too.

{quote}
I'd like to point out that the other side of the argument is that the system
needs to trust the your customized application to be secure.
{quote}
Yes.

{quote}
Unfortunately building a secure application is often tricky \[1,2,3,4\].
{quote}
No argument here. Remember, though, this is an _interface_ to such a

[jira] [Updated] (KAFKA-2017) Persist Coordinator State for Coordinator Failover


 [ 
https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2017:
---
Affects Version/s: (was: 0.10.0.0)
   0.9.0.0
Fix Version/s: (was: 0.10.0.0)
   0.9.0.0

An impact is that if we don't persist the coordinator state, during the 
coordinator failover, existing consumers won't be able to commit offsets during 
rebalances since the new coordinator doesn't know existing members of the 
group. This can potentially introduce duplicates in the new consumer, which is 
a degradation from the old consumer.

> Persist Coordinator State for Coordinator Failover
> --
>
> Key: KAFKA-2017
> URL: https://issues.apache.org/jira/browse/KAFKA-2017
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Onur Karaman
>Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2017.patch, KAFKA-2017_2015-05-20_09:13:39.patch, 
> KAFKA-2017_2015-05-21_19:02:47.patch
>
>
> When a coordinator fails, the group membership protocol tries to failover to 
> a new coordinator without forcing all the consumers rejoin their groups. This 
> is possible if the coordinator persists its state so that the state can be 
> transferred during coordinator failover. This state consists of most of the 
> information in GroupRegistry and ConsumerRegistry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2635) After deleting the topic, you can still send and receive messages but kafka-topic --list does not show the topic.

2015-10-12 Thread Mayuresh Gharat (JIRA)

Mayuresh Gharat created KAFKA-2635:
--

 Summary: After deleting the topic, you can still send and receive 
messages but kafka-topic --list does not show the topic.
 Key: KAFKA-2635
 URL: https://issues.apache.org/jira/browse/KAFKA-2635
 Project: Kafka
  Issue Type: Bug
Reporter: Mayuresh Gharat
Assignee: Mayuresh Gharat


As per [~mingfang]'s comment on 
https://issues.apache.org/jira/browse/KAFKA-2094, it is possible to send and 
receive data for a deleted topic but the kafka-topic --list does not show that 
topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2094) Kafka does not create topic automatically after deleting the topic.

2015-10-12 Thread Mayuresh Gharat (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953697#comment-14953697
 ] 

Mayuresh Gharat commented on KAFKA-2094:


[~mingfang] We should create a separate Jira for this. I have created a ticket 
:https://issues.apache.org/jira/browse/KAFKA-2635.
Can you can put some context on that about what you tried and what you saw. I 
can than start working on it.
PS :
When one of the topic on our Linkedin cluster was deleted recently we saw 
LeaderNotAvailableException while producing to that topic.

> Kafka does not create topic automatically after deleting the topic.
> ---
>
> Key: KAFKA-2094
> URL: https://issues.apache.org/jira/browse/KAFKA-2094
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.8.2.0
> Environment: Ubuntu 14.04 LTS
>Reporter: Hyukjin Kwon
>Priority: Critical
> Fix For: 0.8.2.0
>
>
> After I create a topic and then remove it (and wait for enough time to 
> eventually delete it), it does not create a topic emitting errors even though 
>  auto-create topic option is true. It works okay when I manually create a 
> topic after deleting it.
> Here is the command I run.
> ./bin/kafka-topics.sh --list  --zookeeper 192.168.0.190:2181
> test
> ./bin/kafka-topics.sh --delete --zookeeper 192.168.0.190:2181 --topic test
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2015-04-06 20:51:44,542] WARN Property topic is not valid 
> (kafka.utils.VerifiableProperties)
> test
> [2015-04-06 20:51:49,896] WARN Error while fetching metadata [{TopicMetadata 
> for topic test ->
> No partition metadata for topic test due to 
> kafka.common.LeaderNotAvailableException}] for topic [test]: class 
> kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
> [2015-04-06 20:51:49,904] WARN Error while fetching metadata [{TopicMetadata 
> for topic test->
> No partition metadata for topic testdue to 
> kafka.common.LeaderNotAvailableException}] for topic [test]: class 
> kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
> [2015-04-06 20:51:49,905] ERROR Failed to collate messages by topic, 
> partition due to: Failed to fetch topic metadata for topic: 
> test(kafka.producer.async.DefaultEventHandler)
> ^Csnowdrop@master:~/workspace/kafka/kafka_2.10-0.8.2.0$ 
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2015-04-06 20:52:08,733] WARN Property topic is not valid 
> (kafka.utils.VerifiableProperties)
> Server side, I got this error
> [2015-04-06 21:41:08,491] ERROR [Replica Manager on Broker 0]: Error when 
> processing fetch request for partition [test,0] offset 1687 from consumer 
> with correlation id 5. Possible cause: Request for offset 1687 but we only 
> have log segments in the range 0 to 107. (kafka.server.ReplicaManager)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Jenkins build is back to normal : kafka-trunk-jdk8 #17

2015-10-12 Thread Ismael Juma

Our first successful Jenkins JDK 8 build. :)

Ismael
On 12 Oct 2015 21:40, "Apache Jenkins Server" 
wrote:

> See 
>
>

[jira] [Commented] (KAFKA-2209) Change client quotas dynamically using DynamicConfigManager


[ 
https://issues.apache.org/jira/browse/KAFKA-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953856#comment-14953856
 ] 

ASF GitHub Bot commented on KAFKA-2209:
---

GitHub user auradkar opened a pull request:

https://github.com/apache/kafka/pull/298

KAFKA-2209 - Change quotas dynamically using DynamicConfigManager

Changes in this patch are:
1. ClientIdConfigHandler now passes through the config changes to the quota 
manager.
2. Removed static KafkaConfigs for quota overrides. These are no longer 
needed since we can override configs through ZooKeeper.
3. Added testcases to verify that the config changes are propogated from ZK 
(written using AdminTools) to the actual Metric objects.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/auradkar/kafka K-2209

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 4a1914d86e37f32a5efae2b1840015f4dea4c139
Author: Aditya Auradkar 
Date:   2015-10-12T18:39:20Z

KAFKA-2209 - Dynamically change quotas per clientId

commit 42e8412f82d1199289ac2fa327502de43eb831d7
Author: Aditya Auradkar 
Date:   2015-10-12T22:08:39Z

Minor fixes

commit dfe35e2941ef76e5eb47b355296d3aa8c4a0e3f0
Author: Aditya Auradkar 
Date:   2015-10-12T22:11:38Z

Minor change




> Change client quotas dynamically using DynamicConfigManager
> ---
>
> Key: KAFKA-2209
> URL: https://issues.apache.org/jira/browse/KAFKA-2209
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Two quick pieces of feedback:
1. The use of a version of -1 as magical entry dividing between deprecated
versions is a bit hacky. What about instead having an array of supported
versions and a separate array of deprecated versions. The deprecated
versions would always be a subset of the supported versions. Or,
alternately, since deprecation has no functional impact and is just a
message to developers, we could just leave it out of the protocol and just
have it in release notes etc.
2. I think including the api name may cause some problems. Currently the
api key is the primary key that we keep consistent but we have actually
evolved the english description of the apis as they have changed. The only
use I can think of for the name would be if people used the logical name
and tried to resolve the api key, but that would be wrong. Not sure if we
actually need the english name, if there is a use case I guess we'll just
have to be very clear that the name is just documentation and can change
any time.

-Jay

On Fri, Sep 25, 2015 at 2:53 PM, Magnus Edenhill  wrote:

> Good evening,
>
> KIP-35 was created to address current and future broker-client
> compatibility.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
>
> Summary:
>  * allow clients to retrieve the broker's protocol version
>  * make broker handle unknown protocol requests gracefully
>
> Feedback and comments welcome!
>
> Regards,
> Magnus
>

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

Great. I definitely would strongly favor carrying over user's intuition
from FS unless we think we need a very different model. The minor details
like the seperator and namespace term will help with that.

Follow-up question, say I have a layout like
   /chicago-datacenter/user-events/pageviews
Can I subscribe to
   /chicago-datacenter/user-events
to get the full firehose of user events from chicago? Can I subscribe to
   /*/user-events
to get user events originating from all datacenters?

(Assuming, for now, that these are all in the same cluster...)

Also, just to confirm, it sounds from the proposal like config overrides
would become fully hierarchical so you can override config at any directory
point. This will add complexity in implementation but I think will likely
be much more operator friendly.

There are about a thousand details to discuss in terms of how this would
impact the metadata request, various zk entries, and various other aspects,
but probably it makes sense to first agree on how we would want it to work
and then start to dive into how to implement that.

-Jay

On Mon, Oct 12, 2015 at 5:28 PM, Ashish Singh  wrote:

> Hey Jay, thanks for reviewing the proposal. Answers inline.
>
> On Mon, Oct 12, 2015 at 10:53 AM, Jay Kreps  wrote:
>
> > Hey guys,
> >
> > I think this is an important feature and one we've talked about for a
> > while. I really think trying to invent a new nomenclature is going to
> make
> > it hard for people to understand, though. As such I recommend we call
> > namespaces "directories" and denote them with '/'--this will make the
> > feature 1000x more understandable to people.
>
> Essentially you are suggesting two things here.
> 1. Use "Directory" instead of "Namespace" as it is more intuitive. I agree.
> 2. Make '/' as delimiter instead of ':'. Fine with me and I agree if we
> call these directories, '/' is the way to go.
>
> I think we should inheret the
> > semantics of normal unix fs in so far as it makes sense.
> >
> > In this approach we get rid of topics entirely, instead we really just
> have
> > partitions which are the equivalent of a file and retain their numeric
> > names, and the existing topic concept is just the first directory level
> but
> > we generalize to allow arbitrarily many more levels of nesting. This
> allows
> > categorization of data, such as /datacenter1/user-events/page-views/3 and
> > you can subscribe, apply configs or permissions at any level of the
> > hierarchy.
> >
> +1. This actually requires just a minor change to existing proposal, i.e.,
> "some:namespace:topic" becomes "some/namespace/topic".
>
> >
> > I'm actually not 100% such what the semantics of accessing data in
> > differing namespaces is in the current proposal, maybe you can clarify
> > Ashish?
>
> I will add more info to KIP on this, however I think a client should be
> able to access data in any namespace as long as following conditions are
> satisfied.
>
> 1. Namespace, the client is trying to access, exists.
> 2. The client has sufficient permissions on the namespace for type of
> operation the client is trying to perform on a topic within that namespace.
> 3. The client has sufficient permissions on the topic for type of operation
> the client is trying to perform on that topic.
>
> If we choose to go with what you suggested earlier that just have hierarchy
> of directories, then step 3 will actually be covered in step 2.
>
> In the current proposal, consumers will subscribe to a topic in a namespace
> by specifying : as the topic name. They can subscribe to
> topics from multiple namespaces.
>
> Let me know if I totally missed your question.
>
> Since the point of Kafka is sharing data I think it is really
> > important that the grouping be just for
> convenience/permissions/config/etc
> > and that it remain possible to access multiple directories/namespaces
> from
> > the same client.
> >
> Totally agree with you.
>
> >
> > -Jay
> >
> > On Fri, Oct 9, 2015 at 6:32 PM, Ashish Singh 
> wrote:
> >
> > > Hey Guys,
> > >
> > > I just created KIP-37 for adding namespaces to Kafka.
> > >
> > > KIP-37
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> > > >
> > > tracks the proposal.
> > >
> > > The idea is to make Kafka support multi-tenancy via namespaces.
> > >
> > > Feedback and comments are welcome.
> > > 
> > > --
> > >
> > > Regards,
> > > Ashish
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
>

[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds


[ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954205#comment-14954205
 ] 

Gwen Shapira commented on KAFKA-2613:
-

Committed and updated the Jenkins builds.

Waiting to make sure tests pass before closing this one.

> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #20

See 

Changes:

[cshapi] KAFKA-2613; Make maxParallelForks configurable via Gradle config so it

--
[...truncated 962 lines...]
kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
PASSED

kafka.common.TopicTest > testInvalidTopicNames PASSED

kafka.common.TopicTest > testTopicHasCollision PASSED

kafka.common.TopicTest > testTopicHasCollisionChars PASSED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.RollingBounceTest > testRollingBounce PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.SslTopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.SslTopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SslTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SslTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SslTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeralRecursive PASSED

kafka.zk.ZKEphemeralTest > testOverlappingSessions PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral PASSED

kafka.zk.ZKEphemeralTest > testSameSession PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialExists PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathExists PASSED

kafka.zk.ZKPathTest > testCreatePersistentPath PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExistsThrowsException PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentPathThrowsException PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExists PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.MessageWriterTest > testWithNoCompressionAttribute PASSED

kafka.message.MessageWriterTest > testWithCompressionAttribute PASSED

kafka.message.MessageWriterTest > testBufferingOutputStream PASSED

kafka.message.MessageWriterTest > testWithKey PASSED

kafka.message.ByteBufferMessageSetTest > testOffsetAssignment PASSED

[jira] [Commented] (KAFKA-2598) Add Test with authorizer for producer and consumer


[ 
https://issues.apache.org/jira/browse/KAFKA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953976#comment-14953976
 ] 

ASF GitHub Bot commented on KAFKA-2598:
---

GitHub user Parth-Brahmbhatt opened a pull request:

https://github.com/apache/kafka/pull/300

KAFKA-2598: Adding integration test for the authorizer at API level. …

…Some bug fixes that I encountered while running the tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2598

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/300.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #300


commit 70d5b6a47085587fd5c30d3696e5dd1c6f740b19
Author: Parth Brahmbhatt 
Date:   2015-10-12T22:46:59Z

KAFKA-2598: Adding integration test for the authorizer at API level. Some 
bug fixes that I encountered while running the tests.




> Add Test with authorizer for producer and consumer
> --
>
> Key: KAFKA-2598
> URL: https://issues.apache.org/jira/browse/KAFKA-2598
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security, unit tests
>Affects Versions: 0.8.2.2
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Now that we have all the authorizer code merged into trunk we should add a 
> test that enables authorizer and tests that only authorized users can 
> produce/consume from topics or issue cluster actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2598: Adding integration test for the au...

2015-10-12 Thread Parth-Brahmbhatt

GitHub user Parth-Brahmbhatt opened a pull request:

https://github.com/apache/kafka/pull/300

KAFKA-2598: Adding integration test for the authorizer at API level. â¦

â¦Some bug fixes that I encountered while running the tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2598

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/300.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #300


commit 70d5b6a47085587fd5c30d3696e5dd1c6f740b19
Author: Parth Brahmbhatt 
Date:   2015-10-12T22:46:59Z

KAFKA-2598: Adding integration test for the authorizer at API level. Some 
bug fixes that I encountered while running the tests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds


[ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954217#comment-14954217
 ] 

Ismael Juma commented on KAFKA-2613:


OK, seems sensible to go for stability in the run-up to 0.9.0.0.

> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #675

See

[GitHub] kafka pull request: KAFKA-2581: Run some existing ducktape tests w...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/271


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2581) Run all existing ducktape tests with SSL-enabled clients and brokers


 [ 
https://issues.apache.org/jira/browse/KAFKA-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2581:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 271
[https://github.com/apache/kafka/pull/271]

> Run all existing ducktape tests with SSL-enabled clients and brokers
> 
>
> Key: KAFKA-2581
> URL: https://issues.apache.org/jira/browse/KAFKA-2581
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> New ducktape tests for testing SSL are being added under KAFKA-2417. This 
> task will enable existing ducktape tests to be run with SSL-enabled brokers 
> and clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2581) Run all existing ducktape tests with SSL-enabled clients and brokers


[ 
https://issues.apache.org/jira/browse/KAFKA-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954107#comment-14954107
 ] 

ASF GitHub Bot commented on KAFKA-2581:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/271


> Run all existing ducktape tests with SSL-enabled clients and brokers
> 
>
> Key: KAFKA-2581
> URL: https://issues.apache.org/jira/browse/KAFKA-2581
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> New ducktape tests for testing SSL are being added under KAFKA-2417. This 
> task will enable existing ducktape tests to be run with SSL-enabled brokers 
> and clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

2015-10-12 Thread Ashish Singh

Hey Jay, thanks for reviewing the proposal. Answers inline.

On Mon, Oct 12, 2015 at 10:53 AM, Jay Kreps  wrote:

> Hey guys,
>
> I think this is an important feature and one we've talked about for a
> while. I really think trying to invent a new nomenclature is going to make
> it hard for people to understand, though. As such I recommend we call
> namespaces "directories" and denote them with '/'--this will make the
> feature 1000x more understandable to people.

Essentially you are suggesting two things here.
1. Use "Directory" instead of "Namespace" as it is more intuitive. I agree.
2. Make '/' as delimiter instead of ':'. Fine with me and I agree if we
call these directories, '/' is the way to go.

I think we should inheret the
> semantics of normal unix fs in so far as it makes sense.
>
> In this approach we get rid of topics entirely, instead we really just have
> partitions which are the equivalent of a file and retain their numeric
> names, and the existing topic concept is just the first directory level but
> we generalize to allow arbitrarily many more levels of nesting. This allows
> categorization of data, such as /datacenter1/user-events/page-views/3 and
> you can subscribe, apply configs or permissions at any level of the
> hierarchy.
>
+1. This actually requires just a minor change to existing proposal, i.e.,
"some:namespace:topic" becomes "some/namespace/topic".

>
> I'm actually not 100% such what the semantics of accessing data in
> differing namespaces is in the current proposal, maybe you can clarify
> Ashish?

I will add more info to KIP on this, however I think a client should be
able to access data in any namespace as long as following conditions are
satisfied.

1. Namespace, the client is trying to access, exists.
2. The client has sufficient permissions on the namespace for type of
operation the client is trying to perform on a topic within that namespace.
3. The client has sufficient permissions on the topic for type of operation
the client is trying to perform on that topic.

If we choose to go with what you suggested earlier that just have hierarchy
of directories, then step 3 will actually be covered in step 2.

In the current proposal, consumers will subscribe to a topic in a namespace
by specifying : as the topic name. They can subscribe to
topics from multiple namespaces.

Let me know if I totally missed your question.

Since the point of Kafka is sharing data I think it is really
> important that the grouping be just for convenience/permissions/config/etc
> and that it remain possible to access multiple directories/namespaces from
> the same client.
>
Totally agree with you.

>
> -Jay
>
> On Fri, Oct 9, 2015 at 6:32 PM, Ashish Singh  wrote:
>
> > Hey Guys,
> >
> > I just created KIP-37 for adding namespaces to Kafka.
> >
> > KIP-37
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> > >
> > tracks the proposal.
> >
> > The idea is to make Kafka support multi-tenancy via namespaces.
> >
> > Feedback and comments are welcome.
> > 
> > --
> >
> > Regards,
> > Ashish
> >
>

-- 

Regards,
Ashish

Build failed in Jenkins: kafka-trunk-jdk7 #676

See 

Changes:

[wangguoz] KAFKA-2581: Run some existing ducktape tests with SSL

--
[...truncated 324 lines...]
:kafka-trunk-jdk7:log4j-appender:compileJava UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:processResources UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:classes UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:jar UP-TO-DATE
:kafka-trunk-jdk7:core:compileJava UP-TO-DATE
:kafka-trunk-jdk7:core:compileScala UP-TO-DATE
:kafka-trunk-jdk7:core:processResources UP-TO-DATE
:kafka-trunk-jdk7:core:classes UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:javadoc
:kafka-trunk-jdk7:core:javadoc
cache taskArtifacts.bin 
(/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/.gradle/2.4/taskArtifacts/taskArtifacts.bin)
 is corrupt. Discarding.
:kafka-trunk-jdk7:core:javadocJar
:kafka-trunk-jdk7:core:scaladoc
[ant:scaladoc] Element 
'/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/build/resources/main'
 does not exist.
[ant:scaladoc] 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/server/KafkaServer.scala:277:
 warning: a pure expression does nothing in statement position; you may be 
omitting necessary parentheses
[ant:scaladoc] ControllerStats.uncleanLeaderElectionRate
[ant:scaladoc] ^
[ant:scaladoc] 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/server/KafkaServer.scala:278:
 warning: a pure expression does nothing in statement position; you may be 
omitting necessary parentheses
[ant:scaladoc] ControllerStats.leaderElectionTimer
[ant:scaladoc] ^
[ant:scaladoc] warning: there were 14 feature warning(s); re-run with -feature 
for details
[ant:scaladoc] 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/utils/ByteBoundedBlockingQueue.scala:72:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#offer".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/utils/ByteBoundedBlockingQueue.scala:32:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#offer".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/utils/ByteBoundedBlockingQueue.scala:137:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#poll".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/utils/ByteBoundedBlockingQueue.scala:120:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#poll".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/utils/ByteBoundedBlockingQueue.scala:97:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#put".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/utils/ByteBoundedBlockingQueue.scala:152:
 warning: Could not find any member to link for 
"java.util.concurrent.BlockingQueue#take".
[ant:scaladoc]   /**
[ant:scaladoc]   ^
[ant:scaladoc] 9 warnings found
:kafka-trunk-jdk7:core:scaladocJar
:kafka-trunk-jdk7:core:docsJar
:docsJar_2_11_7
Building project 'core' with Scala version 2.11.7
:kafka-trunk-jdk7:clients:compileJavaNote: 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

:kafka-trunk-jdk7:clients:processResources UP-TO-DATE
:kafka-trunk-jdk7:clients:classes
:kafka-trunk-jdk7:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk7:clients:createVersionFile
:kafka-trunk-jdk7:clients:jar
:kafka-trunk-jdk7:clients:javadoc
:kafka-trunk-jdk7:log4j-appender:compileJava
:kafka-trunk-jdk7:log4j-appender:processResources UP-TO-DATE
:kafka-trunk-jdk7:log4j-appender:classes
:kafka-trunk-jdk7:log4j-appender:jar
:kafka-trunk-jdk7:core:compileJava UP-TO-DATE
:kafka-trunk-jdk7:core:compileScala
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/api/OffsetCommitRequest.scala:78:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk7/core/src/main/scala/kafka/common/OffsetMetadataAndError.scala:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is

[jira] [Commented] (KAFKA-2581) Run all existing ducktape tests with SSL-enabled clients and brokers


[ 
https://issues.apache.org/jira/browse/KAFKA-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954162#comment-14954162
 ] 

Ismael Juma commented on KAFKA-2581:


Thanks for your contribution Rajini and thanks for merging Guozhang. Rajini, 
can you please update the title and description of the JIRA ticket to match 
what was actually in PR 271? And can you please file another ticket for the SSL 
ducktape tests that depend on the ConsoleConsumer timeout JIRA being merged?

> Run all existing ducktape tests with SSL-enabled clients and brokers
> 
>
> Key: KAFKA-2581
> URL: https://issues.apache.org/jira/browse/KAFKA-2581
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> New ducktape tests for testing SSL are being added under KAFKA-2417. This 
> task will enable existing ducktape tests to be run with SSL-enabled brokers 
> and clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds


[ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954214#comment-14954214
 ] 

Ismael Juma commented on KAFKA-2613:


Thanks Gwen and Ewen. What value did you use for the Jenkins jobs?

> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2209 - Change quotas dynamically using D...

2015-10-12 Thread auradkar

GitHub user auradkar opened a pull request:

https://github.com/apache/kafka/pull/298

KAFKA-2209 - Change quotas dynamically using DynamicConfigManager

Changes in this patch are:
1. ClientIdConfigHandler now passes through the config changes to the quota 
manager.
2. Removed static KafkaConfigs for quota overrides. These are no longer 
needed since we can override configs through ZooKeeper.
3. Added testcases to verify that the config changes are propogated from ZK 
(written using AdminTools) to the actual Metric objects.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/auradkar/kafka K-2209

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 4a1914d86e37f32a5efae2b1840015f4dea4c139
Author: Aditya Auradkar 
Date:   2015-10-12T18:39:20Z

KAFKA-2209 - Dynamically change quotas per clientId

commit 42e8412f82d1199289ac2fa327502de43eb831d7
Author: Aditya Auradkar 
Date:   2015-10-12T22:08:39Z

Minor fixes

commit dfe35e2941ef76e5eb47b355296d3aa8c4a0e3f0
Author: Aditya Auradkar 
Date:   2015-10-12T22:11:38Z

Minor change




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds


[ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954201#comment-14954201
 ] 

ASF GitHub Bot commented on KAFKA-2613:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/293


> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2613: Make maxParallelForks configurable...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/293


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #677

See 

Changes:

[cshapi] KAFKA-2613; Make maxParallelForks configurable via Gradle config so it

--
[...truncated 207 lines...]
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP) {

  ^
:259:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
if (offsetAndMetadata.commitTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

  ^
:277:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.uncleanLeaderElectionRate
^
:278:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.leaderElectionTimer
^
:380:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
if (value.expireTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

  ^
:115:
 value METADATA_FETCH_TIMEOUT_CONFIG in object ProducerConfig is deprecated: 
see corresponding Javadoc for more information.
props.put(ProducerConfig.METADATA_FETCH_TIMEOUT_CONFIG, 
config.metadataFetchTimeoutMs.toString)
 ^
:117:
 value TIMEOUT_CONFIG in object ProducerConfig is deprecated: see corresponding 
Javadoc for more information.
props.put(ProducerConfig.TIMEOUT_CONFIG, config.requestTimeoutMs.toString)
 ^
:121:
 value BLOCK_ON_BUFFER_FULL_CONFIG in object ProducerConfig is deprecated: see 
corresponding Javadoc for more information.
  props.put(ProducerConfig.BLOCK_ON_BUFFER_FULL_CONFIG, "false")
   ^
:75:
 value BLOCK_ON_BUFFER_FULL_CONFIG in object ProducerConfig is deprecated: see 
corresponding Javadoc for more information.
producerProps.put(ProducerConfig.BLOCK_ON_BUFFER_FULL_CONFIG, "true")
 ^
:185:
 value BLOCK_ON_BUFFER_FULL_CONFIG in object ProducerConfig is deprecated: see 
corresponding Javadoc for more information.
maybeSetDefaultProperty(producerProps, 
ProducerConfig.BLOCK_ON_BUFFER_FULL_CONFIG, "true")
  ^
:389:
 class BrokerEndPoint in object UpdateMetadataRequest is deprecated: see 
corresponding Javadoc for more information.
  new UpdateMetadataRequest.BrokerEndPoint(brokerEndPoint.id, 
brokerEndPoint.host, brokerEndPoint.port)
^
:391:
 constructor UpdateMetadataRequest in class

[GitHub] kafka pull request: KAFKA-2490: support new consumer in ConsumerGr...

2015-10-12 Thread SinghAsDev

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/299

KAFKA-2490: support new consumer in ConsumerGroupCommand



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-2490

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/299.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #299


commit 73ad826fb653c975dd4f8983219e611045d87ed7
Author: Ashish Singh 
Date:   2015-09-21T17:46:47Z

KAFKA-2490: support new consumer in ConsumerGroupCommand




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds


 [ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2613:

   Resolution: Fixed
Fix Version/s: 0.9.0.0
   Status: Resolved  (was: Patch Available)

> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #678

See

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

Okay this is similar to what I think we have talked about before. Let me
elaborate on the idea that I think has been floating around--it's pretty
similar with a few differences.

I think what you are calling the "default namespace" is basically what I
would call the "current working directory" with paths not beginning with
'/' being interpreted relative to this directory as in the fs.

One thing you have to work out is what levels in this hierarchy you can
actually subscribe to. I think you are assuming only what we currently
consider a "topic", i.e. the first level of directories but not the
partitions or parent dirs, would be subscribable. If you think about it,
though, that constraint is a bit arbitrary.

I'd propose instead the semantics that:
- Subscribing to /a/b/c/0 means subscribing to the 0th partition of topic
"c" in directory /a/b
- Subscribing to /a/b/c means subscribing to all partitions in
topic/directory "c"
- Subscribing to /a/b means subscribing to all partitions in all
topics/subdirectories under a/b recursively

Effectively the concept of topics goes away entirely--you just have
partitions/logs and directories. In this respect rather than adding new
concepts this new feature would actually just generalizes what we have
(which I think is a good thing).

-Jay

On Mon, Oct 12, 2015 at 6:24 PM, Ashish Singh  wrote:

> On Mon, Oct 12, 2015 at 5:42 PM, Jay Kreps  wrote:
>
> > Great. I definitely would strongly favor carrying over user's intuition
> > from FS unless we think we need a very different model. The minor details
> > like the seperator and namespace term will help with that.
> >
> > Follow-up question, say I have a layout like
> >/chicago-datacenter/user-events/pageviews
> > Can I subscribe to
> >/chicago-datacenter/user-events
> >
> Yes, however they will have need a regex like
> /chicago-datacenter/user-events/*
>
> > to get the full firehose of user events from chicago? Can I subscribe to
> >/*/user-events
> > to get user events originating from all datacenters?
> >
> Yes, however they will have need a regex like
> /chicago-datacenter/user-events/*
> Yes
>
> >
> > (Assuming, for now, that these are all in the same cluster...)
> >
> > Also, just to confirm, it sounds from the proposal like config overrides
> > would become fully hierarchical so you can override config at any
> directory
> > point. This will add complexity in implementation but I think will likely
> > be much more operator friendly.
> >
> Yes, that is the idea.
>
> >
> > There are about a thousand details to discuss in terms of how this would
> > impact the metadata request, various zk entries, and various other
> aspects,
> > but probably it makes sense to first agree on how we would want it to
> work
> > and then start to dive into how to implement that.
> >
> Agreed.
>
> >
> > -Jay
> >
> > On Mon, Oct 12, 2015 at 5:28 PM, Ashish Singh 
> wrote:
> >
> > > Hey Jay, thanks for reviewing the proposal. Answers inline.
> > >
> > > On Mon, Oct 12, 2015 at 10:53 AM, Jay Kreps  wrote:
> > >
> > > > Hey guys,
> > > >
> > > > I think this is an important feature and one we've talked about for a
> > > > while. I really think trying to invent a new nomenclature is going to
> > > make
> > > > it hard for people to understand, though. As such I recommend we call
> > > > namespaces "directories" and denote them with '/'--this will make the
> > > > feature 1000x more understandable to people.
> > >
> > > Essentially you are suggesting two things here.
> > > 1. Use "Directory" instead of "Namespace" as it is more intuitive. I
> > agree.
> > > 2. Make '/' as delimiter instead of ':'. Fine with me and I agree if we
> > > call these directories, '/' is the way to go.
> > >
> > > I think we should inheret the
> > > > semantics of normal unix fs in so far as it makes sense.
> > > >
> > > > In this approach we get rid of topics entirely, instead we really
> just
> > > have
> > > > partitions which are the equivalent of a file and retain their
> numeric
> > > > names, and the existing topic concept is just the first directory
> level
> > > but
> > > > we generalize to allow arbitrarily many more levels of nesting. This
> > > allows
> > > > categorization of data, such as /datacenter1/user-events/page-views/3
> > and
> > > > you can subscribe, apply configs or permissions at any level of the
> > > > hierarchy.
> > > >
> > > +1. This actually requires just a minor change to existing proposal,
> > i.e.,
> > > "some:namespace:topic" becomes "some/namespace/topic".
> > >
> > > >
> > > > I'm actually not 100% such what the semantics of accessing data in
> > > > differing namespaces is in the current proposal, maybe you can
> clarify
> > > > Ashish?
> > >
> > > I will add more info to KIP on this, however I think a client should be
> > > able to access data in any namespace as long as following conditions
> are
> > > satisfied.
> > >
> > > 1. Namespace,

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

2015-10-12 Thread Ashish Singh

Rajini, thanks for the review. Answers inline.

On Mon, Oct 12, 2015 at 8:01 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Ashish,
>
> Thank you for doing this writeup and starting the discussion around
> namespaces and multi-tenancy.
>
> We have implemented a namespace solution on top of Kafka trunk and our
> motivation to do so matches your description in KIP-37 to a large extent.
> But there are some differences in our approach. The main difference is that
> we are not exposing namespaces to clients. Our goal is to provide a
> multi-tenant interface where each tenant has a view of Kafka that looks and
> feels like a single-user Kafka cluster. This is a quick summary of our
> approach. We are waiting for the SASL implementation in Kafka to complete
> this work and will be happy to share the code once we have integrated with
> SASL.
>
 As Jay later suggested, and I agree, clients should be able to talk to any
namespace as long as they have the permission to access that namespace and
the namespace exists.

I think below is a client should be able to do.

1. Get a list of namespaces it has permissions to.
2. Get a list of topics in namespaces it has permissions to.
3. Produce/ Consume to any topic in any namespace as long as that namespace
exists, the client has permissions to the namespace and the topic.

I think as long as we satisfy the above mentioned reqs, your requirement
should by met. Let me know if I am missing something.

>
> *Current implementation:*
>
> *Goal*: Provide a secure multi-tenant Kafka service that works with
> out-of-the-box Kafka 0.9.0.0 clients. We have a requirement to isolate
> tenants so that tenants do not have any awareness of other tenants.
>
I think one the advantages of current proposal is that existing and
previous clients will also be able to work seamlessly with the proposed
changes.

>
> *Implementation*:
>
>- We use a modified version of kafka.server.KafkaApis.scala which
>enables a pluggable interceptor for requests and responses.
>- Our interceptor plugin adds a tenant-specific prefix to topics and
>consumer groups in requests and removes the prefix in responses. At the
>moment, we are using valid topic/group characters in prefixes so that no
>other changes are required in Kafka.
>- Tenants are identified and authenticated based on their security
>principal. We are using clientid to hold an API key temporarily while we
>wait for SASL implementation in Kafka.
>- External clients which connect using SSL+SASL only see the topics
>within their namespace. They have no access to Zookeeper.
>- Our internal clients use PLAINTEXT and do see all topics and also have
>access to Zookeeper.
>
> *Longer term goals:*
>
>- Even though we started our prototype with hierarchical topics, we
>decided against it for our initial release to avoid changes to clients.
> But
>for the longer term, it will definitely be useful to use hierarchical
>topics to provide namespaces. In particular, we would like the option to
>use different encryption keys to encrypt data at rest for different
>tenants. If/when hierarchical topics are supported in Kafka, we would
> use
>the first level of the hierarchy as a tenant prefix.
>- We are currently not using quotas for tenants, but we do have a
>requirement to support quotas.
>
> I think we can make your proposed solution work for us. But I wasn't clear
> on why namespaces would be a preferred approach to full-fledged
> hierarchical topics.

What do you mean by full-fledged hierarchical topics? I think full-fledged
hierarchical topics is a way to implement namespace. Let me know if I am
missing something.


> There is no mention of consumer groups which also
> requires namespaces to support multi-tenancy.

Not sure what you mean by namespaces for consumer groups. As long as
consumers in consumer group have permissions to interact with a namespace,
they should be able to work they work right now. Let me know if I am
missing something.

We prefer to have
> multi-tenant applications which aren't aware of their namespace as we have
> today, but if that is something specific to our environment and the general
> consensus is to make tenants namespace-aware, we may be able to work around
> it.
>
> Thoughts?
>
>
> Regards,
>
> Rajini
>
> On Sat, Oct 10, 2015 at 10:03 AM, Magnus Edenhill 
> wrote:
>
> > Good write-up Ashish.
> >
> > Looking at one of the rejected alternatives got me thinking:
> > "Modify request/ response formats to take namespace specifically.
> >
> > Solves the issue of delimiting string required in proposed approach and
> the
> > issue with existing regex consumers.
> > This definitely is the cleanest approach. However, will require lots of
> API
> > and protocol changes. This is listed in rejected alternatives, but we
> > should totally consider this as an option."
> >
> >
> > I think we could actually achieve this without

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

2015-10-12 Thread Ashish Singh

On Mon, Oct 12, 2015 at 5:42 PM, Jay Kreps  wrote:

> Great. I definitely would strongly favor carrying over user's intuition
> from FS unless we think we need a very different model. The minor details
> like the seperator and namespace term will help with that.
>
> Follow-up question, say I have a layout like
>/chicago-datacenter/user-events/pageviews
> Can I subscribe to
>/chicago-datacenter/user-events
>
Yes, however they will have need a regex like
/chicago-datacenter/user-events/*

> to get the full firehose of user events from chicago? Can I subscribe to
>/*/user-events
> to get user events originating from all datacenters?
>
Yes, however they will have need a regex like
/chicago-datacenter/user-events/*
Yes

>
> (Assuming, for now, that these are all in the same cluster...)
>
> Also, just to confirm, it sounds from the proposal like config overrides
> would become fully hierarchical so you can override config at any directory
> point. This will add complexity in implementation but I think will likely
> be much more operator friendly.
>
Yes, that is the idea.

>
> There are about a thousand details to discuss in terms of how this would
> impact the metadata request, various zk entries, and various other aspects,
> but probably it makes sense to first agree on how we would want it to work
> and then start to dive into how to implement that.
>
Agreed.

>
> -Jay
>
> On Mon, Oct 12, 2015 at 5:28 PM, Ashish Singh  wrote:
>
> > Hey Jay, thanks for reviewing the proposal. Answers inline.
> >
> > On Mon, Oct 12, 2015 at 10:53 AM, Jay Kreps  wrote:
> >
> > > Hey guys,
> > >
> > > I think this is an important feature and one we've talked about for a
> > > while. I really think trying to invent a new nomenclature is going to
> > make
> > > it hard for people to understand, though. As such I recommend we call
> > > namespaces "directories" and denote them with '/'--this will make the
> > > feature 1000x more understandable to people.
> >
> > Essentially you are suggesting two things here.
> > 1. Use "Directory" instead of "Namespace" as it is more intuitive. I
> agree.
> > 2. Make '/' as delimiter instead of ':'. Fine with me and I agree if we
> > call these directories, '/' is the way to go.
> >
> > I think we should inheret the
> > > semantics of normal unix fs in so far as it makes sense.
> > >
> > > In this approach we get rid of topics entirely, instead we really just
> > have
> > > partitions which are the equivalent of a file and retain their numeric
> > > names, and the existing topic concept is just the first directory level
> > but
> > > we generalize to allow arbitrarily many more levels of nesting. This
> > allows
> > > categorization of data, such as /datacenter1/user-events/page-views/3
> and
> > > you can subscribe, apply configs or permissions at any level of the
> > > hierarchy.
> > >
> > +1. This actually requires just a minor change to existing proposal,
> i.e.,
> > "some:namespace:topic" becomes "some/namespace/topic".
> >
> > >
> > > I'm actually not 100% such what the semantics of accessing data in
> > > differing namespaces is in the current proposal, maybe you can clarify
> > > Ashish?
> >
> > I will add more info to KIP on this, however I think a client should be
> > able to access data in any namespace as long as following conditions are
> > satisfied.
> >
> > 1. Namespace, the client is trying to access, exists.
> > 2. The client has sufficient permissions on the namespace for type of
> > operation the client is trying to perform on a topic within that
> namespace.
> > 3. The client has sufficient permissions on the topic for type of
> operation
> > the client is trying to perform on that topic.
> >
> > If we choose to go with what you suggested earlier that just have
> hierarchy
> > of directories, then step 3 will actually be covered in step 2.
> >
> > In the current proposal, consumers will subscribe to a topic in a
> namespace
> > by specifying : as the topic name. They can subscribe
> to
> > topics from multiple namespaces.
> >
> > Let me know if I totally missed your question.
> >
> > Since the point of Kafka is sharing data I think it is really
> > > important that the grouping be just for
> > convenience/permissions/config/etc
> > > and that it remain possible to access multiple directories/namespaces
> > from
> > > the same client.
> > >
> > Totally agree with you.
> >
> > >
> > > -Jay
> > >
> > > On Fri, Oct 9, 2015 at 6:32 PM, Ashish Singh 
> > wrote:
> > >
> > > > Hey Guys,
> > > >
> > > > I just created KIP-37 for adding namespaces to Kafka.
> > > >
> > > > KIP-37
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> > > > >
> > > > tracks the proposal.
> > > >
> > > > The idea is to make Kafka support multi-tenancy via namespaces.
> > > >
> > > > Feedback and comments are welcome.
> > > > 
> > > > --
> > > >
> > > >

[jira] [Created] (KAFKA-2636) Producer connectivity obscured connection failure logging

2015-10-12 Thread Jason Kania (JIRA)

Jason Kania created KAFKA-2636:
--

 Summary: Producer connectivity obscured connection failure logging
 Key: KAFKA-2636
 URL: https://issues.apache.org/jira/browse/KAFKA-2636
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1, 0.8.2.2
 Environment: Windows 8 running java implementation of Kafka Producer
Reporter: Jason Kania


The Kafka Producer does not generate a visible exception when a connection 
cannot be made. Instead DEBUG settings are required to observe the problem as 
shown below:

[2015-10-12 21:23:20,335] DEBUG Error connecting to node 0 at 
482f4769eed1:9092: (org.apache.kafka.clients.NetworkClient)
java.io.IOException: Can't resolve address: 482f4769eed1:9092
at org.apache.kafka.common.network.Selector.connect(Selector.java:138)
at 
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:417)
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:116)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:165)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.kafka.common.network.Selector.connect(Selector.java:135)
... 5 more
[2015-10-12 21:23:20,358] DEBUG Initiating connection to node 0 at 
482f4769eed1:9092. (org.apache.kafka.clients.NetworkClient)

Secondly, the errors do not identify the node by IP address making error 
investigation more difficult especially when learning to use Kafka.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds


[ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954215#comment-14954215
 ] 

Gwen Shapira commented on KAFKA-2613:
-

1 :)


> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2015-10-12 Thread Jiangjie Qin

I am thinking instead of returning an empty response, it would be better to
return an explicit UnsupportedVersionException code.

Today KafkaApis handles the error in the following way:
1. For requests/responses using old Scala classes, KafkaApis uses
RequestOrResponse.handleError() to return an error response.
2. For requests/response using Java classes (only JoinGroupRequest and
Heartbeat now), KafkaApis calls AbstractRequest.getErrorResponse() to
return an error response.

In KAFKA-2512, I am returning an UnsupportedVersionException for case [1]
when see an unsupported version. This will put the error code per topic or
partition for most of the requests, but might not work all the time. e.g.
TopicMetadataRequest with an empty topic set.

Case [2] does not quite work for unsupported version, because we will
thrown an uncaught exception when version is not recognized (BTW this is a
bug). Part of the reason is that for some response types, error code is not
part of the response level field.

Maybe it worth checking how each response is dealing with error code today.
A scan of the response formats gives the following result:
1. TopicMetadataResponse - per topic error code, does not work when the
topic set is empty in the request.
2. ProduceResonse - per partition error code.
3. OffsetCommitResponse - per partition.
4. OffsetFetchResponse - per partition.
5. OffsetResponse - per partition.
6. FetchResponse - per partition
7. ConsumerMetadataResponse - response level
8. ControlledShutdownResponse - response level
9. JoinGroupResponse - response level
10. HearbeatResponse - response level
11. LeaderAndIsrResponse - response level
12. StopReplicaResponse - response level
13. UpdateMetadataResponse - response level

So from the list above it looks for each response we are actually able to
return an error code, as long as we make sure the topic or partition won't
be empty when the error code is at topic or partition level. Luckily in the
above list we only need to worry about TopicMetadataResponse.

Maybe error handling is out of the scope of this KIP, but I prefer we think
through how to deal with error code for the requests, because there are
more request types to be added in KAFKA-2464 and future patches.

Thanks,

Jiangjie (Becket) Qin

On Mon, Oct 12, 2015 at 6:04 PM, Jay Kreps  wrote:

> Two quick pieces of feedback:
> 1. The use of a version of -1 as magical entry dividing between deprecated
> versions is a bit hacky. What about instead having an array of supported
> versions and a separate array of deprecated versions. The deprecated
> versions would always be a subset of the supported versions. Or,
> alternately, since deprecation has no functional impact and is just a
> message to developers, we could just leave it out of the protocol and just
> have it in release notes etc.
> 2. I think including the api name may cause some problems. Currently the
> api key is the primary key that we keep consistent but we have actually
> evolved the english description of the apis as they have changed. The only
> use I can think of for the name would be if people used the logical name
> and tried to resolve the api key, but that would be wrong. Not sure if we
> actually need the english name, if there is a use case I guess we'll just
> have to be very clear that the name is just documentation and can change
> any time.
>
> -Jay
>
> On Fri, Sep 25, 2015 at 2:53 PM, Magnus Edenhill 
> wrote:
>
> > Good evening,
> >
> > KIP-35 was created to address current and future broker-client
> > compatibility.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >
> > Summary:
> >  * allow clients to retrieve the broker's protocol version
> >  * make broker handle unknown protocol requests gracefully
> >
> > Feedback and comments welcome!
> >
> > Regards,
> > Magnus
> >
>

[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds


[ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954250#comment-14954250
 ] 

Gwen Shapira commented on KAFKA-2613:
-

Well, I'm not sure we are any stabler now, but I didn't break Jenkins so I'm 
closing this one.

> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

2015-10-12 Thread Gwen Shapira

This works really nicely from the consumer side, but what about the
producer? If there are no more topics,do we allow producing to a directory
and have the Partitioner hash-partition messages between all partitions in
the multiple levels in a directory?

Also, I think we want to preserve the consumer terminology of "subscribe"
to topics / directories, but "assign" partitions - since the consumer
behavior is different in those cases.

On Mon, Oct 12, 2015 at 7:16 PM, Jay Kreps  wrote:

> Okay this is similar to what I think we have talked about before. Let me
> elaborate on the idea that I think has been floating around--it's pretty
> similar with a few differences.
>
> I think what you are calling the "default namespace" is basically what I
> would call the "current working directory" with paths not beginning with
> '/' being interpreted relative to this directory as in the fs.
>
> One thing you have to work out is what levels in this hierarchy you can
> actually subscribe to. I think you are assuming only what we currently
> consider a "topic", i.e. the first level of directories but not the
> partitions or parent dirs, would be subscribable. If you think about it,
> though, that constraint is a bit arbitrary.
>
> I'd propose instead the semantics that:
> - Subscribing to /a/b/c/0 means subscribing to the 0th partition of topic
> "c" in directory /a/b
> - Subscribing to /a/b/c means subscribing to all partitions in
> topic/directory "c"
> - Subscribing to /a/b means subscribing to all partitions in all
> topics/subdirectories under a/b recursively
>
> Effectively the concept of topics goes away entirely--you just have
> partitions/logs and directories. In this respect rather than adding new
> concepts this new feature would actually just generalizes what we have
> (which I think is a good thing).
>
> -Jay
>
> On Mon, Oct 12, 2015 at 6:24 PM, Ashish Singh  wrote:
>
> > On Mon, Oct 12, 2015 at 5:42 PM, Jay Kreps  wrote:
> >
> > > Great. I definitely would strongly favor carrying over user's intuition
> > > from FS unless we think we need a very different model. The minor
> details
> > > like the seperator and namespace term will help with that.
> > >
> > > Follow-up question, say I have a layout like
> > >/chicago-datacenter/user-events/pageviews
> > > Can I subscribe to
> > >/chicago-datacenter/user-events
> > >
> > Yes, however they will have need a regex like
> > /chicago-datacenter/user-events/*
> >
> > > to get the full firehose of user events from chicago? Can I subscribe
> to
> > >/*/user-events
> > > to get user events originating from all datacenters?
> > >
> > Yes, however they will have need a regex like
> > /chicago-datacenter/user-events/*
> > Yes
> >
> > >
> > > (Assuming, for now, that these are all in the same cluster...)
> > >
> > > Also, just to confirm, it sounds from the proposal like config
> overrides
> > > would become fully hierarchical so you can override config at any
> > directory
> > > point. This will add complexity in implementation but I think will
> likely
> > > be much more operator friendly.
> > >
> > Yes, that is the idea.
> >
> > >
> > > There are about a thousand details to discuss in terms of how this
> would
> > > impact the metadata request, various zk entries, and various other
> > aspects,
> > > but probably it makes sense to first agree on how we would want it to
> > work
> > > and then start to dive into how to implement that.
> > >
> > Agreed.
> >
> > >
> > > -Jay
> > >
> > > On Mon, Oct 12, 2015 at 5:28 PM, Ashish Singh 
> > wrote:
> > >
> > > > Hey Jay, thanks for reviewing the proposal. Answers inline.
> > > >
> > > > On Mon, Oct 12, 2015 at 10:53 AM, Jay Kreps 
> wrote:
> > > >
> > > > > Hey guys,
> > > > >
> > > > > I think this is an important feature and one we've talked about
> for a
> > > > > while. I really think trying to invent a new nomenclature is going
> to
> > > > make
> > > > > it hard for people to understand, though. As such I recommend we
> call
> > > > > namespaces "directories" and denote them with '/'--this will make
> the
> > > > > feature 1000x more understandable to people.
> > > >
> > > > Essentially you are suggesting two things here.
> > > > 1. Use "Directory" instead of "Namespace" as it is more intuitive. I
> > > agree.
> > > > 2. Make '/' as delimiter instead of ':'. Fine with me and I agree if
> we
> > > > call these directories, '/' is the way to go.
> > > >
> > > > I think we should inheret the
> > > > > semantics of normal unix fs in so far as it makes sense.
> > > > >
> > > > > In this approach we get rid of topics entirely, instead we really
> > just
> > > > have
> > > > > partitions which are the equivalent of a file and retain their
> > numeric
> > > > > names, and the existing topic concept is just the first directory
> > level
> > > > but
> > > > > we generalize to allow arbitrarily many more levels of nesting.

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I wonder if we need to solve the error problem? I think this KIP gives a
descent work around.

Probably we should have included an error in the response header, but we
debated it at the time decided not to and now it is pretty hard to add
because the headers aren't versioned (d'oh).

It seems like any other solution is going to be kind of a hack, right?
Sending malformed responses back seems like not a clean solution...

(Not sure if I was pro- having a top-level error or not, but in any case
the rationale for the decision was that so many of the requests were
per-partition or per-topic or whatever and hence fail or succeed at that
level and this makes it hard to know what the right top-level error code is
and hard for the client to figure out what to do with the top level error
if some of the partitions succeed but there is a top-level error).

I think actually this new API actually gives a way to handle this
gracefully on the client side by just having clients that want to be
graceful check for support for their version. Clients that do that will
have a graceful message.

At some point if we're ever reworking the headers we should really consider
(a) versioning them and (b) adding a top-level error code in the response.
But given this would be a big breaking change and this is really just to
give a nicer error message seems like it probably isn't worth it to try to
do something now.

-Jay



On Mon, Oct 12, 2015 at 8:11 PM, Jiangjie Qin 
wrote:

> I am thinking instead of returning an empty response, it would be better to
> return an explicit UnsupportedVersionException code.
>
> Today KafkaApis handles the error in the following way:
> 1. For requests/responses using old Scala classes, KafkaApis uses
> RequestOrResponse.handleError() to return an error response.
> 2. For requests/response using Java classes (only JoinGroupRequest and
> Heartbeat now), KafkaApis calls AbstractRequest.getErrorResponse() to
> return an error response.
>
> In KAFKA-2512, I am returning an UnsupportedVersionException for case [1]
> when see an unsupported version. This will put the error code per topic or
> partition for most of the requests, but might not work all the time. e.g.
> TopicMetadataRequest with an empty topic set.
>
> Case [2] does not quite work for unsupported version, because we will
> thrown an uncaught exception when version is not recognized (BTW this is a
> bug). Part of the reason is that for some response types, error code is not
> part of the response level field.
>
> Maybe it worth checking how each response is dealing with error code today.
> A scan of the response formats gives the following result:
> 1. TopicMetadataResponse - per topic error code, does not work when the
> topic set is empty in the request.
> 2. ProduceResonse - per partition error code.
> 3. OffsetCommitResponse - per partition.
> 4. OffsetFetchResponse - per partition.
> 5. OffsetResponse - per partition.
> 6. FetchResponse - per partition
> 7. ConsumerMetadataResponse - response level
> 8. ControlledShutdownResponse - response level
> 9. JoinGroupResponse - response level
> 10. HearbeatResponse - response level
> 11. LeaderAndIsrResponse - response level
> 12. StopReplicaResponse - response level
> 13. UpdateMetadataResponse - response level
>
> So from the list above it looks for each response we are actually able to
> return an error code, as long as we make sure the topic or partition won't
> be empty when the error code is at topic or partition level. Luckily in the
> above list we only need to worry about TopicMetadataResponse.
>
> Maybe error handling is out of the scope of this KIP, but I prefer we think
> through how to deal with error code for the requests, because there are
> more request types to be added in KAFKA-2464 and future patches.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Oct 12, 2015 at 6:04 PM, Jay Kreps  wrote:
>
> > Two quick pieces of feedback:
> > 1. The use of a version of -1 as magical entry dividing between
> deprecated
> > versions is a bit hacky. What about instead having an array of supported
> > versions and a separate array of deprecated versions. The deprecated
> > versions would always be a subset of the supported versions. Or,
> > alternately, since deprecation has no functional impact and is just a
> > message to developers, we could just leave it out of the protocol and
> just
> > have it in release notes etc.
> > 2. I think including the api name may cause some problems. Currently the
> > api key is the primary key that we keep consistent but we have actually
> > evolved the english description of the apis as they have changed. The
> only
> > use I can think of for the name would be if people used the logical name
> > and tried to resolve the api key, but that would be wrong. Not sure if we
> > actually need the english name, if there is a use case I guess we'll just
> > have to be very clear that the name is just documentation and can

Jenkins build is back to normal : kafka-trunk-jdk8 #21

See

[GitHub] kafka pull request: let kafka support "commit offset after consumi...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/206


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #680