date:20160324

[jira] [Created] (KAFKA-3456) In-house KafkaMetric misreports metrics when periodically observed

2016-03-24 Thread The Data Lorax (JIRA)

The Data Lorax created KAFKA-3456:
-

 Summary: In-house KafkaMetric misreports metrics when periodically 
observed
 Key: KAFKA-3456
 URL: https://issues.apache.org/jira/browse/KAFKA-3456
 Project: Kafka
  Issue Type: Bug
  Components: consumer, core, producer 
Affects Versions: 0.9.0.1, 0.9.0.0, 0.10.0.0
Reporter: The Data Lorax
Assignee: Neha Narkhede
Priority: Minor


The metrics captured by Kafka through the in-house {{SampledStat}} suffer from 
misreporting metrics if observed in a periodic manner.

Consider a {{Rate}} metric that is using the default 2 samples and 30 second 
sample window i.e. the {{Rate}} is capturing 60 seconds worth of data.  So, to 
report this metric to some external system we might poll it every 60 seconds to 
observe the current value. Using a shorter period would, in the case of a 
{{Rate}}, lead to smoothing of the plotted data, and worse, in the case of a 
{{Count}}, would lead to double counting - so 60 seconds is the only period at 
which we can poll the metrics if we are to report accurate metrics.

To demonstrate the issue consider the following somewhat extreme case:

The {{Rate}}  is capturing data from a system which alternates between a 999 
per sec rate and a 1 per sec rate every 30 seconds, with the different rates 
aligned with the sample boundaries within the {{Rate}} instance i.e. after 60 
seconds the first sample within the {{Rate}} instance will have a rate of 999 
per sec, and the second 1 per sec. 

If we were to as the metric for its value at this 60 second boundary it would 
correctly report 500 per sec. However, if we asked it again 1 millisecond later 
it would report 1 per sec, as the first sample window has been aged out. 
Depending on how retarded into the 60 sec period of the metric our periodic 
poll of the metric was, we would observe a constant rate somewhere the range of 
1 to 500 per second, most likely around the 250 mark. 

Other metrics based off of the {{SampledStat}} type suffer from the same issue 
e.g. the {{Count}} metric, given a constant rate of 1 per second, will report a 
constant count somewhere between 30 and 60, rather than the correct 60.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-493) High CPU usage on inactive server

2016-03-24 Thread Li Jinyu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210084#comment-15210084
 ] 

Li Jinyu commented on KAFKA-493:


I have the same problem with 0.8.2.2 cluster, the CPU usage is really high, 

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
40442 root  20   0 11.3g 606m  16m S 1040.0  3.8  36:42.34 java

this cluster was upgraded from 0.8.1.1, as some issues found. leaders changed 
frequently, and sometimes one node was alive but zookeeper cannot see it.

but in another 0.8.2.2 cluster, everything is fine. I tried to delete all data 
on one node as I thought it's caused by corrupted message data, but it didn't 
work.

> High CPU usage on inactive server
> -
>
> Key: KAFKA-493
> URL: https://issues.apache.org/jira/browse/KAFKA-493
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
> Attachments: Kafka-2014-11-10.snapshot.zip, Kafka-sampling1.zip, 
> Kafka-sampling2.zip, Kafka-sampling3.zip, Kafka-trace1.zip, Kafka-trace2.zip, 
> Kafka-trace3.zip, backtraces.txt, stacktrace.txt
>
>
> > I've been playing with the 0.8 branch of Kafka and noticed that idle CPU 
> > usage is fairly high (13% of a 
> > core). Is that to be expected? I did look at the stack, but didn't see 
> > anything obvious. A background 
> > task?
> > I wanted to mention how I am getting into this state. I've set up two 
> > machines with the latest 0.8 
> > code base and am using a replication factor of 2. On starting the brokers 
> > there is no idle CPU activity. 
> > Then I run a test that essential does 10k publish operations followed by 
> > immediate consume operations 
> > (I was measuring latency). Once this has run the kafka nodes seem to 
> > consistently be consuming CPU 
> > essentially forever.
> hprof results:
> THREAD START (obj=53ae, id = 24, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 25, name="RMI TCP Accept-", 
> group="system")
> THREAD START (obj=53ae, id = 26, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 21, name="main", group="main")
> THREAD START (obj=53ae, id = 27, name="Thread-2", group="main")
> THREAD START (obj=53ae, id = 28, name="Thread-3", group="main")
> THREAD START (obj=53ae, id = 29, name="kafka-processor-9092-0", 
> group="main")
> THREAD START (obj=53ae, id = 200010, name="kafka-processor-9092-1", 
> group="main")
> THREAD START (obj=53ae, id = 200011, name="kafka-acceptor", group="main")
> THREAD START (obj=574b, id = 200012, 
> name="ZkClient-EventThread-20-localhost:2181", group="main")
> THREAD START (obj=576e, id = 200014, name="main-SendThread()", 
> group="main")
> THREAD START (obj=576d, id = 200013, name="main-EventThread", 
> group="main")
> THREAD START (obj=53ae, id = 200015, name="metrics-meter-tick-thread-1", 
> group="main")
> THREAD START (obj=53ae, id = 200016, name="metrics-meter-tick-thread-2", 
> group="main")
> THREAD START (obj=53ae, id = 200017, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200018, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200019, name="kafka-request-handler-0", 
> group="main")
> THREAD START (obj=53ae, id = 200020, name="kafka-request-handler-1", 
> group="main")
> THREAD START (obj=53ae, id = 200021, name="Thread-6", group="main")
> THREAD START (obj=53ae, id = 200022, name="Thread-7", group="main")
> THREAD START (obj=5899, id = 200023, name="ReplicaFetcherThread-0-2 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200024, name="ReplicaFetcherThread-0-3 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200025, name="ReplicaFetcherThread-0-0 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200026, name="ReplicaFetcherThread-0-1 on 
> broker 1, ", group="main")
> THREAD START (obj=53ae, id = 200028, name="SIGINT handler", 
> group="system")
> THREAD START (obj=53ae, id = 200029, name="Thread-5", group="main")
> THREAD START (obj=574b, id = 200030, name="Thread-1", group="main")
> THREAD START (obj=574b, id = 200031, name="Thread-0", group="main")
> THREAD END (id = 200031)
> THREAD END (id = 200029)
> THREAD END (id = 200020)
> THREAD END (id = 200019)
> THREAD END (id = 28)
> THREAD END (id = 200021)
> THREAD END (id = 27)
> THREAD END (id = 200022)
> THREAD END (id = 200018)
> THREAD END (id = 200017)
> THREAD END (id = 200012)
> THREAD END (id = 200013)
> THREAD END (id = 200014)
> THREAD END (id = 200025)
> THREAD END (id = 200023)
> THREAD END (id = 200026)
> THREAD END (id = 200024)
> THREAD END (id = 200011)
> THREAD END (id = 29)
>

[jira] [Updated] (KAFKA-3456) In-house KafkaMetric misreports metrics when periodically observed

2016-03-24 Thread The Data Lorax (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

The Data Lorax updated KAFKA-3456:
--
Description: 
The metrics captured by Kafka through the in-house {{SampledStat}} suffer from 
misreporting metrics if observed in a periodic manner.

Consider a {{Rate}} metric that is using the default 2 samples and 30 second 
sample window i.e. the {{Rate}} is capturing 60 seconds worth of data.  So, to 
report this metric to some external system we might poll it every 60 seconds to 
observe the current value. Using a shorter period would, in the case of a 
{{Rate}}, lead to smoothing of the plotted data, and worse, in the case of a 
{{Count}}, would lead to double counting - so 60 seconds is the only period at 
which we can poll the metrics if we are to report accurate metrics.

To demonstrate the issue consider the following somewhat extreme case:

The {{Rate}}  is capturing data from a system which alternates between a 999 
per sec rate and a 1 per sec rate every 30 seconds, with the different rates 
aligned with the sample boundaries within the {{Rate}} instance i.e. after 60 
seconds the first sample within the {{Rate}} instance will have a rate of 999 
per sec, and the second 1 per sec. 

If we were to ask the metric for its value at this 60 second boundary it would 
correctly report 500 per sec. However, if we asked it again 1 millisecond later 
it would report 1 per sec, as the first sample window has been aged out. 
Depending on how retarded into the 60 sec period of the metric our periodic 
poll of the metric was, we would observe a constant rate somewhere in the range 
of 1 to 500 per second, most likely around the 250 mark. 

Other metrics based off of the {{SampledStat}} type suffer from the same issue 
e.g. the {{Count}} metric, given a constant rate of 1 per second, will report a 
constant count somewhere between 30 and 60, rather than the correct 60.

This can be seen in the following test code:

{code:java}
public class MetricsTest {
private MetricConfig metricsConfig;

@Before
public void setUp() throws Exception {
metricsConfig = new MetricConfig();
}

private long t(final int bucket) {
return metricsConfig.timeWindowMs() * bucket;
}

@Test
public void testHowRateDropsMetrics() throws Exception {
Rate rate = new Rate();
metricsConfig.samples(2);
metricsConfig.timeWindow(30, TimeUnit.SECONDS);

// First sample window from t0 -> (t1 -1), with rate 999 per second:
for (long time = t(0); time != t(1); time += 1000) {
rate.record(metricsConfig, 999, time);
}

// Second sample window from t1 -> (t2 -1), with rate 1 per second:
for (long time = t(1); time != t(2); time += 1000) {
rate.record(metricsConfig, 1, time);
}

// Measure at bucket boundary, (though same issue exists all periodic 
measurements)
final double m1 = rate.measure(metricsConfig, t(2));// m1 = 1.0

// Third sample window from t2 -> (t3 -1), with rate 999 per second:
for (long time = t(2); time != t(3); time += 1000) {
rate.record(metricsConfig, 999, time);
}

// Second sample window from t3 -> (t4 -1), with rate 1 per second:
for (long time = t(3); time != t(4); time += 1000) {
rate.record(metricsConfig, 1, time);
}

// Measure second pair of samples:
final double m2 = rate.measure(metricsConfig, t(4));// m2 = 1.0

assertEquals("Measurement of the rate over the first two samples", 
500.0, m1, 2.0);
assertEquals("Measurement of the rate over the last two samples", 
500.0, m2, 2.0);
}

@Test
public void testHowRateDropsMetricsWithRetardedObservations() throws 
Exception {
final long retardation = 1000;

Rate rate = new Rate();
metricsConfig.samples(2);
metricsConfig.timeWindow(30, TimeUnit.SECONDS);

// First sample window from t0 -> (t1 -1), with rate 999 per second:
for (long time = t(0); time != t(1); time += 1000) {
rate.record(metricsConfig, 999, time);
}

// Second sample window from t1 -> (t2 -1), with rate 1 per second:
for (long time = t(1); time != t(2); time += 1000) {
rate.record(metricsConfig, 1, time);
}

double m1 = 0.0;

// Third sample window from t2 -> (t3 -1), with rate 999 per second:
for (long time = t(2); time != t(3); time += 1000) {
rate.record(metricsConfig, 999, time);

if (time == t(2) + retardation) {
m1 = rate.measure(metricsConfig, time); // // m1 = 65.something
}
}

// Second sample window from t3 -> (t4 -1), with rate 1 per second:
for (long time = t(3); time != t(4); time += 1000) {

[jira] [Commented] (KAFKA-493) High CPU usage on inactive server

2016-03-24 Thread Cosmin Marginean (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210092#comment-15210092
 ] 

Cosmin Marginean commented on KAFKA-493:


I can confirm that we're been running 0.9.0.1 for over a week now and none of 
the CPU issues are present anymore (See 0.9.0.1-upgrade.png. I will make sure 
to update this if it regresses.

> High CPU usage on inactive server
> -
>
> Key: KAFKA-493
> URL: https://issues.apache.org/jira/browse/KAFKA-493
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
> Attachments: Kafka-2014-11-10.snapshot.zip, Kafka-sampling1.zip, 
> Kafka-sampling2.zip, Kafka-sampling3.zip, Kafka-trace1.zip, Kafka-trace2.zip, 
> Kafka-trace3.zip, backtraces.txt, stacktrace.txt
>
>
> > I've been playing with the 0.8 branch of Kafka and noticed that idle CPU 
> > usage is fairly high (13% of a 
> > core). Is that to be expected? I did look at the stack, but didn't see 
> > anything obvious. A background 
> > task?
> > I wanted to mention how I am getting into this state. I've set up two 
> > machines with the latest 0.8 
> > code base and am using a replication factor of 2. On starting the brokers 
> > there is no idle CPU activity. 
> > Then I run a test that essential does 10k publish operations followed by 
> > immediate consume operations 
> > (I was measuring latency). Once this has run the kafka nodes seem to 
> > consistently be consuming CPU 
> > essentially forever.
> hprof results:
> THREAD START (obj=53ae, id = 24, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 25, name="RMI TCP Accept-", 
> group="system")
> THREAD START (obj=53ae, id = 26, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 21, name="main", group="main")
> THREAD START (obj=53ae, id = 27, name="Thread-2", group="main")
> THREAD START (obj=53ae, id = 28, name="Thread-3", group="main")
> THREAD START (obj=53ae, id = 29, name="kafka-processor-9092-0", 
> group="main")
> THREAD START (obj=53ae, id = 200010, name="kafka-processor-9092-1", 
> group="main")
> THREAD START (obj=53ae, id = 200011, name="kafka-acceptor", group="main")
> THREAD START (obj=574b, id = 200012, 
> name="ZkClient-EventThread-20-localhost:2181", group="main")
> THREAD START (obj=576e, id = 200014, name="main-SendThread()", 
> group="main")
> THREAD START (obj=576d, id = 200013, name="main-EventThread", 
> group="main")
> THREAD START (obj=53ae, id = 200015, name="metrics-meter-tick-thread-1", 
> group="main")
> THREAD START (obj=53ae, id = 200016, name="metrics-meter-tick-thread-2", 
> group="main")
> THREAD START (obj=53ae, id = 200017, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200018, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200019, name="kafka-request-handler-0", 
> group="main")
> THREAD START (obj=53ae, id = 200020, name="kafka-request-handler-1", 
> group="main")
> THREAD START (obj=53ae, id = 200021, name="Thread-6", group="main")
> THREAD START (obj=53ae, id = 200022, name="Thread-7", group="main")
> THREAD START (obj=5899, id = 200023, name="ReplicaFetcherThread-0-2 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200024, name="ReplicaFetcherThread-0-3 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200025, name="ReplicaFetcherThread-0-0 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200026, name="ReplicaFetcherThread-0-1 on 
> broker 1, ", group="main")
> THREAD START (obj=53ae, id = 200028, name="SIGINT handler", 
> group="system")
> THREAD START (obj=53ae, id = 200029, name="Thread-5", group="main")
> THREAD START (obj=574b, id = 200030, name="Thread-1", group="main")
> THREAD START (obj=574b, id = 200031, name="Thread-0", group="main")
> THREAD END (id = 200031)
> THREAD END (id = 200029)
> THREAD END (id = 200020)
> THREAD END (id = 200019)
> THREAD END (id = 28)
> THREAD END (id = 200021)
> THREAD END (id = 27)
> THREAD END (id = 200022)
> THREAD END (id = 200018)
> THREAD END (id = 200017)
> THREAD END (id = 200012)
> THREAD END (id = 200013)
> THREAD END (id = 200014)
> THREAD END (id = 200025)
> THREAD END (id = 200023)
> THREAD END (id = 200026)
> THREAD END (id = 200024)
> THREAD END (id = 200011)
> THREAD END (id = 29)
> THREAD END (id = 200010)
> THREAD END (id = 200030)
> THREAD END (id = 200028)
> TRACE 301281:
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown 
> line)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>

[jira] [Updated] (KAFKA-3456) In-house KafkaMetric misreports metrics when periodically observed

2016-03-24 Thread The Data Lorax (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

The Data Lorax updated KAFKA-3456:
--
Description: 
The metrics captured by Kafka through the in-house {{SampledStat}} suffer from 
misreporting metrics if observed in a periodic manner.

Consider a {{Rate}} metric that is using the default 2 samples and 30 second 
sample window i.e. the {{Rate}} is capturing 60 seconds worth of data.  So, to 
report this metric to some external system we might poll it every 60 seconds to 
observe the current value. Using a shorter period would, in the case of a 
{{Rate}}, lead to smoothing of the plotted data, and worse, in the case of a 
{{Count}}, would lead to double counting - so 60 seconds is the only period at 
which we can poll the metrics if we are to report accurate metrics.

To demonstrate the issue consider the following somewhat extreme case:

The {{Rate}}  is capturing data from a system which alternates between a 999 
per sec rate and a 1 per sec rate every 30 seconds, with the different rates 
aligned with the sample boundaries within the {{Rate}} instance i.e. after 60 
seconds the first sample within the {{Rate}} instance will have a rate of 999 
per sec, and the second 1 per sec. 

If we were to ask the metric for its value at this 60 second boundary it would 
correctly report 500 per sec. However, if we asked it again 1 millisecond later 
it would report 1 per sec, as the first sample window has been aged out. 
Depending on how retarded into the 60 sec period of the metric our periodic 
poll of the metric was, we would observe a constant rate somewhere in the range 
of 1 to 500 per second, most likely around the 250 mark. 

Other metrics based off of the {{SampledStat}} type suffer from the same issue 
e.g. the {{Count}} metric, given a constant rate of 1 per second, will report a 
constant count somewhere between 30 and 60, rather than the correct 60.

This can be seen in the following test code:

{code:java}
public class MetricsTest {
private MetricConfig metricsConfig;

@Before
public void setUp() throws Exception {
metricsConfig = new MetricConfig();
}

private long t(final int bucket) {
return metricsConfig.timeWindowMs() * bucket;
}

@Test
public void testHowRateDropsMetrics() throws Exception {
Rate rate = new Rate();
metricsConfig.samples(2);
metricsConfig.timeWindow(30, TimeUnit.SECONDS);

// First sample window from t0 -> (t1 -1), with rate 999 per second:
for (long time = t(0); time != t(1); time += 1000) {
rate.record(metricsConfig, 999, time);
}

// Second sample window from t1 -> (t2 -1), with rate 1 per second:
for (long time = t(1); time != t(2); time += 1000) {
rate.record(metricsConfig, 1, time);
}

// Measure at bucket boundary, (though same issue exists all periodic 
measurements)
final double m1 = rate.measure(metricsConfig, t(2));// m1 = 1.0

// Third sample window from t2 -> (t3 -1), with rate 999 per second:
for (long time = t(2); time != t(3); time += 1000) {
rate.record(metricsConfig, 999, time);
}

// Second sample window from t3 -> (t4 -1), with rate 1 per second:
for (long time = t(3); time != t(4); time += 1000) {
rate.record(metricsConfig, 1, time);
}

// Measure second pair of samples:
final double m2 = rate.measure(metricsConfig, t(4));// m2 = 1.0

assertEquals("Measurement of the rate over the first two samples", 
500.0, m1, 2.0);
assertEquals("Measurement of the rate over the last two samples", 
500.0, m2, 2.0);
}

@Test
public void testHowRateDropsMetricsWithRetardedObservations() throws 
Exception {
final long retardation = 1000;

Rate rate = new Rate();
metricsConfig.samples(2);
metricsConfig.timeWindow(30, TimeUnit.SECONDS);

// First sample window from t0 -> (t1 -1), with rate 999 per second:
for (long time = t(0); time != t(1); time += 1000) {
rate.record(metricsConfig, 999, time);
}

// Second sample window from t1 -> (t2 -1), with rate 1 per second:
for (long time = t(1); time != t(2); time += 1000) {
rate.record(metricsConfig, 1, time);
}

double m1 = 0.0;

// Third sample window from t2 -> (t3 -1), with rate 999 per second:
for (long time = t(2); time != t(3); time += 1000) {
rate.record(metricsConfig, 999, time);

if (time == t(2) + retardation) {
m1 = rate.measure(metricsConfig, time); // // m1 = 65.something
}
}

// Second sample window from t3 -> (t4 -1), with rate 1 per second:
for (long time = t(3); time != t(4); time += 1000) {

[GitHub] kafka pull request: Update Sender.java

2016-03-24 Thread SoyeeDst

GitHub user SoyeeDst opened a pull request:

https://github.com/apache/kafka/pull/1130

Update Sender.java

Avoid expired RecordBatch to be sent out to network

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SoyeeDst/kafka soyee

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1130.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1130


commit d589f4e286697dd11d274c3fa9c895946b3303ba
Author: SoyeeDeng 
Date:   2016-03-24T11:18:51Z

Update Sender.java

Avoid expired RecordBatch to be sent out to network




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-493) High CPU usage on inactive server

2016-03-24 Thread Cosmin Marginean (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210092#comment-15210092
 ] 

Cosmin Marginean edited comment on KAFKA-493 at 3/24/16 11:19 AM:
--

I can confirm that we're been running 0.9.0.1 for over a week now and none of 
the CPU issues are present anymore (See 0.9.0.1-upgrade.png). I will make sure 
to update this if it regresses.


was (Author: cosmin.marginean):
I can confirm that we're been running 0.9.0.1 for over a week now and none of 
the CPU issues are present anymore (See 0.9.0.1-upgrade.png. I will make sure 
to update this if it regresses.

> High CPU usage on inactive server
> -
>
> Key: KAFKA-493
> URL: https://issues.apache.org/jira/browse/KAFKA-493
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
> Attachments: Kafka-2014-11-10.snapshot.zip, Kafka-sampling1.zip, 
> Kafka-sampling2.zip, Kafka-sampling3.zip, Kafka-trace1.zip, Kafka-trace2.zip, 
> Kafka-trace3.zip, backtraces.txt, stacktrace.txt
>
>
> > I've been playing with the 0.8 branch of Kafka and noticed that idle CPU 
> > usage is fairly high (13% of a 
> > core). Is that to be expected? I did look at the stack, but didn't see 
> > anything obvious. A background 
> > task?
> > I wanted to mention how I am getting into this state. I've set up two 
> > machines with the latest 0.8 
> > code base and am using a replication factor of 2. On starting the brokers 
> > there is no idle CPU activity. 
> > Then I run a test that essential does 10k publish operations followed by 
> > immediate consume operations 
> > (I was measuring latency). Once this has run the kafka nodes seem to 
> > consistently be consuming CPU 
> > essentially forever.
> hprof results:
> THREAD START (obj=53ae, id = 24, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 25, name="RMI TCP Accept-", 
> group="system")
> THREAD START (obj=53ae, id = 26, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 21, name="main", group="main")
> THREAD START (obj=53ae, id = 27, name="Thread-2", group="main")
> THREAD START (obj=53ae, id = 28, name="Thread-3", group="main")
> THREAD START (obj=53ae, id = 29, name="kafka-processor-9092-0", 
> group="main")
> THREAD START (obj=53ae, id = 200010, name="kafka-processor-9092-1", 
> group="main")
> THREAD START (obj=53ae, id = 200011, name="kafka-acceptor", group="main")
> THREAD START (obj=574b, id = 200012, 
> name="ZkClient-EventThread-20-localhost:2181", group="main")
> THREAD START (obj=576e, id = 200014, name="main-SendThread()", 
> group="main")
> THREAD START (obj=576d, id = 200013, name="main-EventThread", 
> group="main")
> THREAD START (obj=53ae, id = 200015, name="metrics-meter-tick-thread-1", 
> group="main")
> THREAD START (obj=53ae, id = 200016, name="metrics-meter-tick-thread-2", 
> group="main")
> THREAD START (obj=53ae, id = 200017, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200018, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200019, name="kafka-request-handler-0", 
> group="main")
> THREAD START (obj=53ae, id = 200020, name="kafka-request-handler-1", 
> group="main")
> THREAD START (obj=53ae, id = 200021, name="Thread-6", group="main")
> THREAD START (obj=53ae, id = 200022, name="Thread-7", group="main")
> THREAD START (obj=5899, id = 200023, name="ReplicaFetcherThread-0-2 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200024, name="ReplicaFetcherThread-0-3 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200025, name="ReplicaFetcherThread-0-0 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200026, name="ReplicaFetcherThread-0-1 on 
> broker 1, ", group="main")
> THREAD START (obj=53ae, id = 200028, name="SIGINT handler", 
> group="system")
> THREAD START (obj=53ae, id = 200029, name="Thread-5", group="main")
> THREAD START (obj=574b, id = 200030, name="Thread-1", group="main")
> THREAD START (obj=574b, id = 200031, name="Thread-0", group="main")
> THREAD END (id = 200031)
> THREAD END (id = 200029)
> THREAD END (id = 200020)
> THREAD END (id = 200019)
> THREAD END (id = 28)
> THREAD END (id = 200021)
> THREAD END (id = 27)
> THREAD END (id = 200022)
> THREAD END (id = 200018)
> THREAD END (id = 200017)
> THREAD END (id = 200012)
> THREAD END (id = 200013)
> THREAD END (id = 200014)
> THREAD END (id = 200025)
> THREAD END (id = 200023)
> THREAD END (id = 200026)
> THREAD END (id = 200024)
> THREAD END (id = 200011)
> THREAD END (id = 29)
> THREAD END (id = 200010)
> THREAD END (id = 200030)

[jira] [Updated] (KAFKA-493) High CPU usage on inactive server

2016-03-24 Thread Cosmin Marginean (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cosmin Marginean updated KAFKA-493:
---
Attachment: 0.9.0.1-upgrade.png

> High CPU usage on inactive server
> -
>
> Key: KAFKA-493
> URL: https://issues.apache.org/jira/browse/KAFKA-493
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
> Attachments: 0.9.0.1-upgrade.png, Kafka-2014-11-10.snapshot.zip, 
> Kafka-sampling1.zip, Kafka-sampling2.zip, Kafka-sampling3.zip, 
> Kafka-trace1.zip, Kafka-trace2.zip, Kafka-trace3.zip, backtraces.txt, 
> stacktrace.txt
>
>
> > I've been playing with the 0.8 branch of Kafka and noticed that idle CPU 
> > usage is fairly high (13% of a 
> > core). Is that to be expected? I did look at the stack, but didn't see 
> > anything obvious. A background 
> > task?
> > I wanted to mention how I am getting into this state. I've set up two 
> > machines with the latest 0.8 
> > code base and am using a replication factor of 2. On starting the brokers 
> > there is no idle CPU activity. 
> > Then I run a test that essential does 10k publish operations followed by 
> > immediate consume operations 
> > (I was measuring latency). Once this has run the kafka nodes seem to 
> > consistently be consuming CPU 
> > essentially forever.
> hprof results:
> THREAD START (obj=53ae, id = 24, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 25, name="RMI TCP Accept-", 
> group="system")
> THREAD START (obj=53ae, id = 26, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 21, name="main", group="main")
> THREAD START (obj=53ae, id = 27, name="Thread-2", group="main")
> THREAD START (obj=53ae, id = 28, name="Thread-3", group="main")
> THREAD START (obj=53ae, id = 29, name="kafka-processor-9092-0", 
> group="main")
> THREAD START (obj=53ae, id = 200010, name="kafka-processor-9092-1", 
> group="main")
> THREAD START (obj=53ae, id = 200011, name="kafka-acceptor", group="main")
> THREAD START (obj=574b, id = 200012, 
> name="ZkClient-EventThread-20-localhost:2181", group="main")
> THREAD START (obj=576e, id = 200014, name="main-SendThread()", 
> group="main")
> THREAD START (obj=576d, id = 200013, name="main-EventThread", 
> group="main")
> THREAD START (obj=53ae, id = 200015, name="metrics-meter-tick-thread-1", 
> group="main")
> THREAD START (obj=53ae, id = 200016, name="metrics-meter-tick-thread-2", 
> group="main")
> THREAD START (obj=53ae, id = 200017, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200018, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200019, name="kafka-request-handler-0", 
> group="main")
> THREAD START (obj=53ae, id = 200020, name="kafka-request-handler-1", 
> group="main")
> THREAD START (obj=53ae, id = 200021, name="Thread-6", group="main")
> THREAD START (obj=53ae, id = 200022, name="Thread-7", group="main")
> THREAD START (obj=5899, id = 200023, name="ReplicaFetcherThread-0-2 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200024, name="ReplicaFetcherThread-0-3 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200025, name="ReplicaFetcherThread-0-0 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200026, name="ReplicaFetcherThread-0-1 on 
> broker 1, ", group="main")
> THREAD START (obj=53ae, id = 200028, name="SIGINT handler", 
> group="system")
> THREAD START (obj=53ae, id = 200029, name="Thread-5", group="main")
> THREAD START (obj=574b, id = 200030, name="Thread-1", group="main")
> THREAD START (obj=574b, id = 200031, name="Thread-0", group="main")
> THREAD END (id = 200031)
> THREAD END (id = 200029)
> THREAD END (id = 200020)
> THREAD END (id = 200019)
> THREAD END (id = 28)
> THREAD END (id = 200021)
> THREAD END (id = 27)
> THREAD END (id = 200022)
> THREAD END (id = 200018)
> THREAD END (id = 200017)
> THREAD END (id = 200012)
> THREAD END (id = 200013)
> THREAD END (id = 200014)
> THREAD END (id = 200025)
> THREAD END (id = 200023)
> THREAD END (id = 200026)
> THREAD END (id = 200024)
> THREAD END (id = 200011)
> THREAD END (id = 29)
> THREAD END (id = 200010)
> THREAD END (id = 200030)
> THREAD END (id = 200028)
> TRACE 301281:
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown 
> line)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 
>

[jira] [Commented] (KAFKA-3453) Transient test failures due to MiniKDC port allocation strategy

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210176#comment-15210176
 ] 

ASF GitHub Bot commented on KAFKA-3453:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1131

KAFKA-3453; Transient test failures due to MiniKDC port allocation strategy

Temporarily copy fixed `MiniKdc` class until `MiniKdc` 2.8.0 is released.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3453-transient-test-failures-mini-kdc-port

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1131.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1131


commit ee5890c4a15fffc7c1d155e938d2541fcf8fd790
Author: Ismael Juma 
Date:   2016-03-24T12:01:46Z

KAFKA-3453; Transient test failures due to MiniKDC port allocation strategy

Temporarily copy fixed `MiniKdc` class until `MiniKdc` 2.8.0 is released.




> Transient test failures due to MiniKDC port allocation strategy
> ---
>
> Key: KAFKA-3453
> URL: https://issues.apache.org/jira/browse/KAFKA-3453
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
> Fix For: 0.10.0.1
>
>
> A number of tests, especially our consumer tests, fail transiently because 
> MiniKDC allocates ports by creating a socket, getting its port, then closing 
> it. As previously addressed in our own code, this causes problems because 
> that port can be reallocated before the process has a chance to bind a new 
> socket -- whether due to another test running in parallel or another process 
> simply binding the port first. This results in errors like this in the tests:
> {quote}
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
>   at 
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {quote}
> This is an ongoing issue that Confluent sees in its Jenkins builds, which is 
> the reason for this ticket. The real issue is actually in MiniKDC (we pass in 
> "0" for the port, but then it uses this other port allocation strategy), but 
> we either need to a) figure out a workaround or b) get a fix in upstream and 
> then update to a newer MiniKDC version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3453; Transient test failures due to Min...

2016-03-24 Thread ijuma

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1131

KAFKA-3453; Transient test failures due to MiniKDC port allocation strategy

Temporarily copy fixed `MiniKdc` class until `MiniKdc` 2.8.0 is released.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3453-transient-test-failures-mini-kdc-port

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1131.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1131


commit ee5890c4a15fffc7c1d155e938d2541fcf8fd790
Author: Ismael Juma 
Date:   2016-03-24T12:01:46Z

KAFKA-3453; Transient test failures due to MiniKDC port allocation strategy

Temporarily copy fixed `MiniKdc` class until `MiniKdc` 2.8.0 is released.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3456) In-house KafkaMetric misreports metrics when periodically observed

2016-03-24 Thread The Data Lorax (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210060#comment-15210060
 ] 

The Data Lorax commented on KAFKA-3456:
---

[~aauradkar], I see you've made changes in this area - would be interested to 
hear your thoughts...

> In-house KafkaMetric misreports metrics when periodically observed
> --
>
> Key: KAFKA-3456
> URL: https://issues.apache.org/jira/browse/KAFKA-3456
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, core, producer 
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0
>Reporter: The Data Lorax
>Assignee: Neha Narkhede
>Priority: Minor
>
> The metrics captured by Kafka through the in-house {{SampledStat}} suffer 
> from misreporting metrics if observed in a periodic manner.
> Consider a {{Rate}} metric that is using the default 2 samples and 30 second 
> sample window i.e. the {{Rate}} is capturing 60 seconds worth of data.  So, 
> to report this metric to some external system we might poll it every 60 
> seconds to observe the current value. Using a shorter period would, in the 
> case of a {{Rate}}, lead to smoothing of the plotted data, and worse, in the 
> case of a {{Count}}, would lead to double counting - so 60 seconds is the 
> only period at which we can poll the metrics if we are to report accurate 
> metrics.
> To demonstrate the issue consider the following somewhat extreme case:
> The {{Rate}}  is capturing data from a system which alternates between a 999 
> per sec rate and a 1 per sec rate every 30 seconds, with the different rates 
> aligned with the sample boundaries within the {{Rate}} instance i.e. after 60 
> seconds the first sample within the {{Rate}} instance will have a rate of 999 
> per sec, and the second 1 per sec. 
> If we were to ask the metric for its value at this 60 second boundary it 
> would correctly report 500 per sec. However, if we asked it again 1 
> millisecond later it would report 1 per sec, as the first sample window has 
> been aged out. Depending on how retarded into the 60 sec period of the metric 
> our periodic poll of the metric was, we would observe a constant rate 
> somewhere in the range of 1 to 500 per second, most likely around the 250 
> mark. 
> Other metrics based off of the {{SampledStat}} type suffer from the same 
> issue e.g. the {{Count}} metric, given a constant rate of 1 per second, will 
> report a constant count somewhere between 30 and 60, rather than the correct 
> 60.
> This can be seen in the following test code:
> {code:java}
> public class MetricsTest {
> private MetricConfig metricsConfig;
> @Before
> public void setUp() throws Exception {
> metricsConfig = new MetricConfig();
> }
> private long t(final int bucket) {
> return metricsConfig.timeWindowMs() * bucket;
> }
> @Test
> public void testHowRateDropsMetrics() throws Exception {
> Rate rate = new Rate();
> metricsConfig.samples(2);
> metricsConfig.timeWindow(30, TimeUnit.SECONDS);
> // First sample window from t0 -> (t1 -1), with rate 999 per second:
> for (long time = t(0); time != t(1); time += 1000) {
> rate.record(metricsConfig, 999, time);
> }
> // Second sample window from t1 -> (t2 -1), with rate 1 per second:
> for (long time = t(1); time != t(2); time += 1000) {
> rate.record(metricsConfig, 1, time);
> }
> // Measure at bucket boundary, (though same issue exists all periodic 
> measurements)
> final double m1 = rate.measure(metricsConfig, t(2));// m1 = 1.0
> // Third sample window from t2 -> (t3 -1), with rate 999 per second:
> for (long time = t(2); time != t(3); time += 1000) {
> rate.record(metricsConfig, 999, time);
> }
> // Second sample window from t3 -> (t4 -1), with rate 1 per second:
> for (long time = t(3); time != t(4); time += 1000) {
> rate.record(metricsConfig, 1, time);
> }
> // Measure second pair of samples:
> final double m2 = rate.measure(metricsConfig, t(4));// m2 = 1.0
> assertEquals("Measurement of the rate over the first two samples", 
> 500.0, m1, 2.0);
> assertEquals("Measurement of the rate over the last two samples", 
> 500.0, m2, 2.0);
> }
> @Test
> public void testHowRateDropsMetricsWithRetardedObservations() throws 
> Exception {
> final long retardation = 1000;
> Rate rate = new Rate();
> metricsConfig.samples(2);
> metricsConfig.timeWindow(30, TimeUnit.SECONDS);
> // First sample window from t0 -> (t1 -1), with rate 999 per second:
> for (long time = t(0); time != t(1); time += 1000) {
>

[jira] [Commented] (KAFKA-493) High CPU usage on inactive server

2016-03-24 Thread Ismael Juma (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210094#comment-15210094
 ] 

Ismael Juma commented on KAFKA-493:
---

Thanks for reporting back Cosmin.

> High CPU usage on inactive server
> -
>
> Key: KAFKA-493
> URL: https://issues.apache.org/jira/browse/KAFKA-493
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
> Attachments: 0.9.0.1-upgrade.png, Kafka-2014-11-10.snapshot.zip, 
> Kafka-sampling1.zip, Kafka-sampling2.zip, Kafka-sampling3.zip, 
> Kafka-trace1.zip, Kafka-trace2.zip, Kafka-trace3.zip, backtraces.txt, 
> stacktrace.txt
>
>
> > I've been playing with the 0.8 branch of Kafka and noticed that idle CPU 
> > usage is fairly high (13% of a 
> > core). Is that to be expected? I did look at the stack, but didn't see 
> > anything obvious. A background 
> > task?
> > I wanted to mention how I am getting into this state. I've set up two 
> > machines with the latest 0.8 
> > code base and am using a replication factor of 2. On starting the brokers 
> > there is no idle CPU activity. 
> > Then I run a test that essential does 10k publish operations followed by 
> > immediate consume operations 
> > (I was measuring latency). Once this has run the kafka nodes seem to 
> > consistently be consuming CPU 
> > essentially forever.
> hprof results:
> THREAD START (obj=53ae, id = 24, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 25, name="RMI TCP Accept-", 
> group="system")
> THREAD START (obj=53ae, id = 26, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 21, name="main", group="main")
> THREAD START (obj=53ae, id = 27, name="Thread-2", group="main")
> THREAD START (obj=53ae, id = 28, name="Thread-3", group="main")
> THREAD START (obj=53ae, id = 29, name="kafka-processor-9092-0", 
> group="main")
> THREAD START (obj=53ae, id = 200010, name="kafka-processor-9092-1", 
> group="main")
> THREAD START (obj=53ae, id = 200011, name="kafka-acceptor", group="main")
> THREAD START (obj=574b, id = 200012, 
> name="ZkClient-EventThread-20-localhost:2181", group="main")
> THREAD START (obj=576e, id = 200014, name="main-SendThread()", 
> group="main")
> THREAD START (obj=576d, id = 200013, name="main-EventThread", 
> group="main")
> THREAD START (obj=53ae, id = 200015, name="metrics-meter-tick-thread-1", 
> group="main")
> THREAD START (obj=53ae, id = 200016, name="metrics-meter-tick-thread-2", 
> group="main")
> THREAD START (obj=53ae, id = 200017, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200018, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200019, name="kafka-request-handler-0", 
> group="main")
> THREAD START (obj=53ae, id = 200020, name="kafka-request-handler-1", 
> group="main")
> THREAD START (obj=53ae, id = 200021, name="Thread-6", group="main")
> THREAD START (obj=53ae, id = 200022, name="Thread-7", group="main")
> THREAD START (obj=5899, id = 200023, name="ReplicaFetcherThread-0-2 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200024, name="ReplicaFetcherThread-0-3 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200025, name="ReplicaFetcherThread-0-0 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200026, name="ReplicaFetcherThread-0-1 on 
> broker 1, ", group="main")
> THREAD START (obj=53ae, id = 200028, name="SIGINT handler", 
> group="system")
> THREAD START (obj=53ae, id = 200029, name="Thread-5", group="main")
> THREAD START (obj=574b, id = 200030, name="Thread-1", group="main")
> THREAD START (obj=574b, id = 200031, name="Thread-0", group="main")
> THREAD END (id = 200031)
> THREAD END (id = 200029)
> THREAD END (id = 200020)
> THREAD END (id = 200019)
> THREAD END (id = 28)
> THREAD END (id = 200021)
> THREAD END (id = 27)
> THREAD END (id = 200022)
> THREAD END (id = 200018)
> THREAD END (id = 200017)
> THREAD END (id = 200012)
> THREAD END (id = 200013)
> THREAD END (id = 200014)
> THREAD END (id = 200025)
> THREAD END (id = 200023)
> THREAD END (id = 200026)
> THREAD END (id = 200024)
> THREAD END (id = 200011)
> THREAD END (id = 29)
> THREAD END (id = 200010)
> THREAD END (id = 200030)
> THREAD END (id = 200028)
> TRACE 301281:
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown 
> line)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>

[GitHub] kafka pull request: KAFKA-3445

2016-03-24 Thread rnpridgeon

GitHub user rnpridgeon opened a pull request:

https://github.com/apache/kafka/pull/1132

KAFKA-3445

Currently the property TASKS_MAX_CONFIG is not validated against 
nonsensical values such as 0. This patch leverages the Range.atLeast() method 
to ensure value is at least 1. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rnpridgeon/kafka KAFKA-3445

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1132


commit 8617218300d5af70a4dc62ac4de77f443291b5ed
Author: Ryan P 
Date:   2016-03-24T14:56:11Z

KAFKA-3445
add validator to TASKS_MAX_CONFIG




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Kafka Connect ++ Kafka Streams

2016-03-24 Thread Michal Hariš

Hello Kafka people!

Great to see Kafka Streams coming along, the design validates (and in many
way supersedes) my own findings from working with various stream processing
systems/frameworks and eventually ending-up using just a small custom
library built directly around Kafka.

I have set out yesterday to translate Hello Samza (the wikipedia feed
example) into Kafka Streams application. Now because this workflow starts
by polling wikipedia IRC and publishes to a topic from which the stream
processors pick-up it would be nice to have this first part done by Kafka
Connect but:

1. IRC channels are not seekable and Kafka Connect architecture claims that
all sources must be seekable - is this still suitable ? (I guess yes as
FileStreamSourceTask can read from stdin which is similar)

2. I would like to have ConnectEmbedded (as opposed to ConnectStandalone or
ConnectDistributed) which is similar to ConnectDistributed, just without
the rest server - i.e. say I have the WikipediaFeedConnector and I want to
launch it programatically from all the instances along-side the Kafka
Streams - but reusing the connect distributed coordination so that only one
instance actually reads the IRC data but another instance picks up work if
that one dies - does it sound like a bad idea for some design reason ? -
the only problem I see is rather technical that the coordination process
uses the rest server for some actions.

Cheers,
Michal

Reg : kafka consumer to dynamically detect topics added

2016-03-24 Thread Siddu Shebannavar

  Hi Team,




I'm using KafkaConsumer to consume messages from Kafka server (topics)..
* It works fine for topics created before starting Consumer code...
* But the problem is, it will not work if the topics created 
dynamically(i mean to say after consumer code started), but the API says it 
will support dynamic topic creation..



Kafka version used : 0.9.0.1

Here is the link for your reference..

https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html



Here is the JAVA code...

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put("enable.auto.commit", "false");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "3");

props.put("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer");

props.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer consumer = new KafkaConsumer<>(props);
Pattern r = Pattern.compile("siddu(\\d)*");

consumer.subscribe(r, new HandleRebalance());
try {
 while(true) {
 ConsumerRecords records = 
consumer.poll(Long.MAX_VALUE);
 for (TopicPartition partition : records.partitions()) {
 List> partitionRecords = 
records.records(partition);
 for (ConsumerRecord record : partitionRecords) 
{
 System.out.println(partition.partition()  + ": "  
+record.offset() + ": " + record.value());
 }
 long lastOffset = partitionRecords.get(partitionRecords.size() 
- 1).offset();

 consumer.commitSync(Collections.singletonMap(partition, new 
OffsetAndMetadata(lastOffset + 1)));
 }
 }
 } finally {
   consumer.close();
 }


NOTE: My topic names are matching the Regular Expression.. And if i restart the 
consumer then it will start reading messages pushed to topic...
Any help is really appreciated...
Thanks,
Siddu
FICO Bangalore.
Ph : +91 - 9845234534


This email and any files transmitted with it are confidential, proprietary and 
intended solely for the individual or entity to whom they are addressed. If you 
have received this email in error please delete it immediately.

[jira] [Commented] (KAFKA-3445) ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210382#comment-15210382
 ] 

ASF GitHub Bot commented on KAFKA-3445:
---

GitHub user rnpridgeon opened a pull request:

https://github.com/apache/kafka/pull/1132

KAFKA-3445

Currently the property TASKS_MAX_CONFIG is not validated against 
nonsensical values such as 0. This patch leverages the Range.atLeast() method 
to ensure value is at least 1. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rnpridgeon/kafka KAFKA-3445

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1132


commit 8617218300d5af70a4dc62ac4de77f443291b5ed
Author: Ryan P 
Date:   2016-03-24T14:56:11Z

KAFKA-3445
add validator to TASKS_MAX_CONFIG




> ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit 
> -
>
> Key: KAFKA-3445
> URL: https://issues.apache.org/jira/browse/KAFKA-3445
> Project: Kafka
>  Issue Type: Improvement
>  Components: config
>Reporter: Ryan P
>Priority: Trivial
>  Labels: newbie
> Attachments: KAFKA-3445.patch
>
>
> I'll be the first to admit this is a bit nit picky any property marked with 
> Importance.HIGH should be guarded against nonsensical values. 
> With that said I would like to suggest that TASKS_MAX_CONFIG be validating 
> against a lower bound limit of 1. 
> I do understand this is unlikely to happen and the configuration is 
> nonsensical but there is no penalty for stopping someone from trying it out. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3465) kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode

2016-03-24 Thread BrianLing (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BrianLing updated KAFKA-3465:
-
Description: 
1. When we enable mirrorMake to migrate Kafka event from one to other with 
"new.consumer" mode:

java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
-XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC 
-Djava.awt.headless=true -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka/kafka-app-logs 
-Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
 -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* 
-Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log 
--consumer.config ../config/consumer.properties --new.consumer --num.streams 4 
--producer.config ../config/producer-slca.properties --whitelist risk.*


2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and 
the owner is none.

bin/kafka-run-class.sh  kafka.tools.ConsumerOffsetChecker --broker-info --group 
lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 --topic 

Group   Topic  Pid Offset  logSize  
   Lag Owner
lvs.slca.mirrormaker   0   418578332   418678347   100015   
   none
lvs.slca.mirrormaker  1   418598026   418698338   100312
  none

[Root Cause]
I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka 
internal to store offset & consumer owner information. 

Does it mean we can not use the below command to check new consumer’s 
lag since current lag formula: lag= logSize – offset 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80

https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
 => offSet Fetch from zookeeper instead of from Kafka

  was:
When we enable mirrorMake to migrate Kafka event from one to other with 
"new.consumer" mode:

java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
-XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC 
-Djava.awt.headless=true -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka/kafka-app-logs 
-Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
 -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* 
-Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log 
--consumer.config ../config/consumer.properties --new.consumer --num.streams 4 
--producer.config ../config/producer-slca.properties --whitelist risk.*


When we use ConsumerOffzsetChecker tool, notice the lag won't changed and the 
owner is none.

bin/kafka-run-class.sh  kafka.tools.ConsumerOffsetChecker --broker-info --group 
lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 --topic 
risk.radd.acct_misc01

Group   Topic  Pid Offset  logSize  
   Lag Owner
lvs.slca.mirrormaker   0   418578332   418678347   100015   
   none
lvs.slca.mirrormaker  1   418598026   418698338   100312
  none


I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka 
internal to store offset & consumer owner information. 

Does it mean we can not use the below command to check new consumer’s 
lag since current lag formula: lag= logSize – offset 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80

https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
 => offSet Fetch from zookeeper instead of from Kafka


> kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
> --
>
> Key: KAFKA-3465
> URL: https://issues.apache.org/jira/browse/KAFKA-3465
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: BrianLing
>
> 1. When we enable mirrorMake to migrate Kafka event from one to other with 
> "new.consumer" mode:
> java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC 
> -Djava.awt.headless=true -Dcom.sun.management.jmxremote 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dkafka.logs.dir=/kafka/kafka-app-logs 
> -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
>  -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* 
>

[jira] [Created] (KAFKA-3465) kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode

2016-03-24 Thread BrianLing (JIRA)

BrianLing created KAFKA-3465:


 Summary: kafka.tools.ConsumerOffsetChecker won't align with kafka 
New Consumer mode
 Key: KAFKA-3465
 URL: https://issues.apache.org/jira/browse/KAFKA-3465
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.9.0.0
Reporter: BrianLing


When we enable mirrorMake to migrate Kafka event from one to other with 
"new.consumer" mode:

java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
-XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC 
-Djava.awt.headless=true -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka/kafka-app-logs 
-Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
 -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* 
-Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log 
--consumer.config ../config/consumer.properties --new.consumer --num.streams 4 
--producer.config ../config/producer-slca.properties --whitelist risk.*


When we use ConsumerOffzsetChecker tool, notice the lag won't changed and the 
owner is none.

bin/kafka-run-class.sh  kafka.tools.ConsumerOffsetChecker --broker-info --group 
lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 --topic 
risk.radd.acct_misc01

Group   Topic  Pid Offset  logSize  
   Lag Owner
lvs.slca.mirrormaker   0   418578332   418678347   100015   
   none
lvs.slca.mirrormaker  1   418598026   418698338   100312
  none


I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka 
internal to store offset & consumer owner information. 

Does it mean we can not use the below command to check new consumer’s 
lag since current lag formula: lag= logSize – offset 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80

https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
 => offSet Fetch from zookeeper instead of from Kafka



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3457) KafkaConsumer.committed(...) hangs forever if port number is wrong

2016-03-24 Thread Harald Kirsch (JIRA)

Harald Kirsch created KAFKA-3457:


 Summary: KafkaConsumer.committed(...) hangs forever if port number 
is wrong
 Key: KAFKA-3457
 URL: https://issues.apache.org/jira/browse/KAFKA-3457
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.9.0.1
Reporter: Harald Kirsch


Create a KafkaConsumer with default settings but with a wrong host:port setting 
for bootstrap.servers. Have it in some consumer group, do not subscribe or 
assign partitions.

Then call .committed(...) for a topic/partition combination a few times. It 
will hang on the 2nd or third call forever. In the debug log you will see that 
it repeats connections all over again. I waited many minutes and it never came 
back to throw an Exception.

The connections problems should at least pop out on the WARNING log level. 
Likely the connection problems should throw an exception eventually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3449) Rename filterOut() to filterNot() to achieve better terminology

2016-03-24 Thread Andrea Cosentino (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210492#comment-15210492
 ] 

Andrea Cosentino commented on KAFKA-3449:
-

PR submitted:

https://github.com/apache/kafka/pull/1133

> Rename filterOut() to filterNot() to achieve better terminology
> ---
>
> Key: KAFKA-3449
> URL: https://issues.apache.org/jira/browse/KAFKA-3449
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Michael Noll
> Fix For: 0.10.0.1
>
>
> Currently the Streams DSL has a function called filterOut(), which is the 
> negation of filter().  Instead of using our own name "filterOut", we should 
> rather use the more familiar name "filterNot" (cf. Scala's filter/filterNot 
> functions).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: [KAFKA-3458] Selector should throw InterruptEx...

2016-03-24 Thread sruehl

GitHub user sruehl opened a pull request:

https://github.com/apache/kafka/pull/1135

[KAFKA-3458] Selector should throw InterruptException when interrupted.

https://issues.apache.org/jira/browse/KAFKA-3458

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sruehl/kafka bugfix/throwProperInterrupt

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1135.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1135


commit 73576cce84399afeaebe5dec4e2b8acfb08b94e7
Author: Sebastian RuÌhl 
Date:   2016-03-24T16:28:18Z

[KAFKA-3458] Selector should throw InterruptException when interrupted.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3349: Rename filterOut() to filterNot() ...

2016-03-24 Thread oscerd

Github user oscerd closed the pull request at:

https://github.com/apache/kafka/pull/1133


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3349) Add system tests for upgrade from 0.8.2.x to 0.10.0.0

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210496#comment-15210496
 ] 

ASF GitHub Bot commented on KAFKA-3349:
---

Github user oscerd closed the pull request at:

https://github.com/apache/kafka/pull/1133


> Add system tests for upgrade from 0.8.2.x to 0.10.0.0
> -
>
> Key: KAFKA-3349
> URL: https://issues.apache.org/jira/browse/KAFKA-3349
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Extend existing system upgrade tests to test upgrade from 0.8.2.x to 0.10.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Reg : kafka consumer to dynamically detect topics added

2016-03-24 Thread Jason Gustafson

Hi Siddu,

The consumer supports a configuration option "metadata.max.age.ms" which
basically controls how often topic metadata is fetched. By default, this is
set fairly high (5 minutes), which means it will take up to 5 minutes to
discover new topics matching your regular expression. You can set this
lower to discover topics quicker. If you wait longer than the configured
max age, and still the topic hasn't been discovered, then you may have
found a bug and we'd welcome a JIRA with the details.

Thanks,
Jason

On Wed, Mar 23, 2016 at 11:02 PM, Siddu Shebannavar <
siddushebanna...@fico.com> wrote:

>   Hi Team,
>
>
>
>
> I'm using KafkaConsumer to consume messages from Kafka server (topics)..
> * It works fine for topics created before starting Consumer code...
> * But the problem is, it will not work if the topics created
> dynamically(i mean to say after consumer code started), but the API says it
> will support dynamic topic creation..
>
>
>
> Kafka version used : 0.9.0.1
>
> Here is the link for your reference..
>
>
> https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
>
>
>
> Here is the JAVA code...
>
> Properties props = new Properties();
> props.put("bootstrap.servers", "localhost:9092");
> props.put("group.id", "test");
> props.put("enable.auto.commit", "false");
> props.put("auto.commit.interval.ms", "1000");
> props.put("session.timeout.ms", "3");
>
> props.put("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
>
> props.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
>
> KafkaConsumer consumer = new KafkaConsumer<>(props);
> Pattern r = Pattern.compile("siddu(\\d)*");
>
> consumer.subscribe(r, new HandleRebalance());
> try {
>  while(true) {
>  ConsumerRecords records =
> consumer.poll(Long.MAX_VALUE);
>  for (TopicPartition partition : records.partitions()) {
>  List> partitionRecords =
> records.records(partition);
>  for (ConsumerRecord record :
> partitionRecords) {
>  System.out.println(partition.partition()  + ": "
> +record.offset() + ": " + record.value());
>  }
>  long lastOffset =
> partitionRecords.get(partitionRecords.size() - 1).offset();
>
>  consumer.commitSync(Collections.singletonMap(partition,
> new OffsetAndMetadata(lastOffset + 1)));
>  }
>  }
>  } finally {
>consumer.close();
>  }
>
>
> NOTE: My topic names are matching the Regular Expression.. And if i
> restart the consumer then it will start reading messages pushed to topic...
> Any help is really appreciated...
> Thanks,
> Siddu
> FICO Bangalore.
> Ph : +91 - 9845234534
>
>
> This email and any files transmitted with it are confidential, proprietary
> and intended solely for the individual or entity to whom they are
> addressed. If you have received this email in error please delete it
> immediately.
>

[jira] [Commented] (KAFKA-3296) All consumer reads hang indefinately

2016-03-24 Thread Simon Cooper (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210419#comment-15210419
 ] 

Simon Cooper commented on KAFKA-3296:
-

No, these are the only references to Expired in the logs:

{code}[aric@reg3 ~]$ grep Expired /aric/logs/kafka/*
/aric/logs/kafka/server.log:2016-03-24T14:29:08,207 | INFO | 
k.s.DelayedOperationPurgatory$ExpiredOperationReaper [ExpirationReaper-1] | 
[ExpirationReaper-1], Starting
/aric/logs/kafka/server.log:2016-03-24T14:29:08,208 | INFO | 
k.s.DelayedOperationPurgatory$ExpiredOperationReaper [ExpirationReaper-1] | 
[ExpirationReaper-1], Starting
/aric/logs/kafka/server.log:2016-03-24T14:29:08,360 | INFO | 
k.s.DelayedOperationPurgatory$ExpiredOperationReaper [ExpirationReaper-1] | 
[ExpirationReaper-1], Starting
/aric/logs/kafka/server.log:2016-03-24T14:29:08,369 | INFO | 
k.s.DelayedOperationPurgatory$ExpiredOperationReaper [ExpirationReaper-1] | 
[ExpirationReaper-1], Starting{code}

There are also previous debug logs from the broker and controller already 
attached to the ticket

> All consumer reads hang indefinately
> 
>
> Key: KAFKA-3296
> URL: https://issues.apache.org/jira/browse/KAFKA-3296
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0, 0.9.0.1
>Reporter: Simon Cooper
>Priority: Critical
> Attachments: controller.zip, kafkalogs.zip
>
>
> We've got several integration tests that bring up systems on VMs for testing. 
> We've recently upgraded to 0.9, and very occasionally we occasionally see an 
> issue where every consumer that tries to read from the broker hangs, spamming 
> the following in their logs:
> {code}2016-02-26T12:25:37,856 | DEBUG | o.a.k.c.NetworkClient 
> [pool-10-thread-1] | Sending metadata request 
> ClientRequest(expectResponse=true, callback=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=21905,client_id=consumer-1},
>  body={topics=[Topic1]}), isInitiatedByNetworkClient, 
> createdTimeMs=1456489537856, sendTimeMs=0) to node 1
> 2016-02-26T12:25:37,856 | DEBUG | o.a.k.c.Metadata [pool-10-thread-1] | 
> Updated cluster metadata version 10954 to Cluster(nodes = [Node(1, 
> server.internal, 9092)], partitions = [Partition(topic = Topic1, partition = 
> 0, leader = 1, replicas = [1,], isr = [1,]])
> 2016-02-26T12:25:37,856 | DEBUG | o.a.k.c.c.i.AbstractCoordinator 
> [pool-10-thread-1] | Issuing group metadata request to broker 1
> 2016-02-26T12:25:37,857 | DEBUG | o.a.k.c.c.i.AbstractCoordinator 
> [pool-10-thread-1] | Group metadata response 
> ClientResponse(receivedTimeMs=1456489537857, disconnected=false, 
> request=ClientRequest(expectResponse=true, 
> callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@28edb273,
>  
> request=RequestSend(header={api_key=10,api_version=0,correlation_id=21906,client_id=consumer-1},
>  body={group_id=}), createdTimeMs=1456489537856, sendTimeMs=1456489537856), 
> responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}})
> 2016-02-26T12:25:37,956 | DEBUG | o.a.k.c.NetworkClient [pool-10-thread-1] | 
> Sending metadata request ClientRequest(expectResponse=true, callback=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=21907,client_id=consumer-1},
>  body={topics=[Topic1]}), isInitiatedByNetworkClient, 
> createdTimeMs=1456489537956, sendTimeMs=0) to node 1
> 2016-02-26T12:25:37,956 | DEBUG | o.a.k.c.Metadata [pool-10-thread-1] | 
> Updated cluster metadata version 10955 to Cluster(nodes = [Node(1, 
> server.internal, 9092)], partitions = [Partition(topic = Topic1, partition = 
> 0, leader = 1, replicas = [1,], isr = [1,]])
> 2016-02-26T12:25:37,956 | DEBUG | o.a.k.c.c.i.AbstractCoordinator 
> [pool-10-thread-1] | Issuing group metadata request to broker 1
> 2016-02-26T12:25:37,957 | DEBUG | o.a.k.c.c.i.AbstractCoordinator 
> [pool-10-thread-1] | Group metadata response 
> ClientResponse(receivedTimeMs=1456489537957, disconnected=false, 
> request=ClientRequest(expectResponse=true, 
> callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@40cee8cc,
>  
> request=RequestSend(header={api_key=10,api_version=0,correlation_id=21908,client_id=consumer-1},
>  body={group_id=}), createdTimeMs=1456489537956, sendTimeMs=1456489537956), 
> responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}})
> 2016-02-26T12:25:38,056 | DEBUG | o.a.k.c.NetworkClient [pool-10-thread-1] | 
> Sending metadata request ClientRequest(expectResponse=true, callback=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=21909,client_id=consumer-1},
>  body={topics=[Topic1]}), isInitiatedByNetworkClient, 
> createdTimeMs=1456489538056, sendTimeMs=0) to node 1
> 2016-02-26T12:25:38,056 | DEBUG | o.a.k.c.Metadata

[jira] [Commented] (KAFKA-3445) ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit

2016-03-24 Thread Ryan P (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210423#comment-15210423
 ] 

Ryan P commented on KAFKA-3445:
---

[~ewencp], sorry about that I should have read that beforehand. 

> ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit 
> -
>
> Key: KAFKA-3445
> URL: https://issues.apache.org/jira/browse/KAFKA-3445
> Project: Kafka
>  Issue Type: Improvement
>  Components: config
>Reporter: Ryan P
>Priority: Trivial
>  Labels: newbie
> Attachments: KAFKA-3445.patch
>
>
> I'll be the first to admit this is a bit nit picky any property marked with 
> Importance.HIGH should be guarded against nonsensical values. 
> With that said I would like to suggest that TASKS_MAX_CONFIG be validating 
> against a lower bound limit of 1. 
> I do understand this is unlikely to happen and the configuration is 
> nonsensical but there is no penalty for stopping someone from trying it out. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3349: Rename filterOut() to filterNot() ...

2016-03-24 Thread oscerd

GitHub user oscerd opened a pull request:

https://github.com/apache/kafka/pull/1133

KAFKA-3349: Rename filterOut() to filterNot() to achieve better termiâ¦

â¦nology

Hi all,

This is my first contribution and I hope it will be good.

The PR is related to this issue:
https://issues.apache.org/jira/browse/KAFKA-3449

Thanks a lot,

Andrea

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/oscerd/kafka KAFKA-3349

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1133.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1133


commit 3fd687ed55e213571cab52d54a8ed921e7c1487f
Author: Andrea Cosentino 
Date:   2016-03-24T16:19:32Z

KAFKA-3349: Rename filterOut() to filterNot() to achieve better terminology




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-3449) Rename filterOut() to filterNot() to achieve better terminology

2016-03-24 Thread Andrea Cosentino (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210492#comment-15210492
 ] 

Andrea Cosentino edited comment on KAFKA-3449 at 3/24/16 4:25 PM:
--

PR submitted:

https://github.com/apache/kafka/pull/1134


was (Author: ancosen):
PR submitted:

https://github.com/apache/kafka/pull/1133

> Rename filterOut() to filterNot() to achieve better terminology
> ---
>
> Key: KAFKA-3449
> URL: https://issues.apache.org/jira/browse/KAFKA-3449
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Michael Noll
> Fix For: 0.10.0.1
>
>
> Currently the Streams DSL has a function called filterOut(), which is the 
> negation of filter().  Instead of using our own name "filterOut", we should 
> rather use the more familiar name "filterNot" (cf. Scala's filter/filterNot 
> functions).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3449) Rename filterOut() to filterNot() to achieve better terminology

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210500#comment-15210500
 ] 

ASF GitHub Bot commented on KAFKA-3449:
---

GitHub user oscerd opened a pull request:

https://github.com/apache/kafka/pull/1134

KAFKA-3449: Rename filterOut() to filterNot() to achieve better termi…

…nology

Hi all,

This is my first contribution and I hope it will be good.

The PR is related to this issue:
https://issues.apache.org/jira/browse/KAFKA-3449

Thanks a lot,

Andrea

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/oscerd/kafka KAFKA-3449

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1134.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1134


commit 92f3ddfe9b85a9f8554a4ce0ad298d7487fb411b
Author: Andrea Cosentino 
Date:   2016-03-24T16:19:32Z

KAFKA-3449: Rename filterOut() to filterNot() to achieve better terminology




> Rename filterOut() to filterNot() to achieve better terminology
> ---
>
> Key: KAFKA-3449
> URL: https://issues.apache.org/jira/browse/KAFKA-3449
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Michael Noll
> Fix For: 0.10.0.1
>
>
> Currently the Streams DSL has a function called filterOut(), which is the 
> negation of filter().  Instead of using our own name "filterOut", we should 
> rather use the more familiar name "filterNot" (cf. Scala's filter/filterNot 
> functions).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3349) Add system tests for upgrade from 0.8.2.x to 0.10.0.0

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210490#comment-15210490
 ] 

ASF GitHub Bot commented on KAFKA-3349:
---

GitHub user oscerd opened a pull request:

https://github.com/apache/kafka/pull/1133

KAFKA-3349: Rename filterOut() to filterNot() to achieve better termi…

…nology

Hi all,

This is my first contribution and I hope it will be good.

The PR is related to this issue:
https://issues.apache.org/jira/browse/KAFKA-3449

Thanks a lot,

Andrea

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/oscerd/kafka KAFKA-3349

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1133.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1133


commit 3fd687ed55e213571cab52d54a8ed921e7c1487f
Author: Andrea Cosentino 
Date:   2016-03-24T16:19:32Z

KAFKA-3349: Rename filterOut() to filterNot() to achieve better terminology




> Add system tests for upgrade from 0.8.2.x to 0.10.0.0
> -
>
> Key: KAFKA-3349
> URL: https://issues.apache.org/jira/browse/KAFKA-3349
> Project: Kafka
>  Issue Type: Task
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Extend existing system upgrade tests to test upgrade from 0.8.2.x to 0.10.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3458) Selector should throw InterruptException when interrupted

2016-03-24 Thread JIRA

Sebastian Rühl created KAFKA-3458:
-

 Summary: Selector should throw InterruptException when interrupted
 Key: KAFKA-3458
 URL: https://issues.apache.org/jira/browse/KAFKA-3458
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.9.0.1
Reporter: Sebastian Rühl


Similar to [KAFKA-2704]:

org.apache.kafka.common.network.Selector does not throw InterruptException when 
interrupted



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3458) Selector should throw InterruptException when interrupted

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210507#comment-15210507
 ] 

ASF GitHub Bot commented on KAFKA-3458:
---

GitHub user sruehl opened a pull request:

https://github.com/apache/kafka/pull/1135

[KAFKA-3458] Selector should throw InterruptException when interrupted.

https://issues.apache.org/jira/browse/KAFKA-3458

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sruehl/kafka bugfix/throwProperInterrupt

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1135.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1135


commit 73576cce84399afeaebe5dec4e2b8acfb08b94e7
Author: Sebastian Rühl 
Date:   2016-03-24T16:28:18Z

[KAFKA-3458] Selector should throw InterruptException when interrupted.




> Selector should throw InterruptException when interrupted
> -
>
> Key: KAFKA-3458
> URL: https://issues.apache.org/jira/browse/KAFKA-3458
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Sebastian Rühl
>
> Similar to [KAFKA-2704]:
> org.apache.kafka.common.network.Selector does not throw InterruptException 
> when interrupted



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

consumer group, why commit requests are not considered as effective heartbeats?

2016-03-24 Thread Zaiming Shi

Hi there!

It was probably sent to the wrong list?
Wish I can get some answer here.

-- Forwarded message --
From: Zaiming Shi 
Date: Wed, Mar 23, 2016 at 6:03 PM
Subject: consumer group, why commit requests are not considered as
effective heartbeats?
To: us...@kafka.apache.org

Hi there!

We have noticed that when committing requests are sent intensively, we
receive IllegalGenerationId.
Here is the settings we had problem with: session-timeout: 30 sec,
heartbeat-rate: 3 sec.
Problem resolved by increasing the session timeout to 180 sec.

So I suppose, due to whatever reason (either the client didn't send
heartbeat, or the broker didn't process the heartbeats in time), the
session was considered dead in group coordinator.

My question is: why commit requests can't be taken as an indicator of
member being alive? hence not to kill the session.

Regards
-Zaiming

[jira] [Commented] (KAFKA-2939) Make AbstractConfig.logUnused() tunable for clients

2016-03-24 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210641#comment-15210641
 ] 

Guozhang Wang commented on KAFKA-2939:
--

I feel that the second option, i.e. making sure all streams-specific usages 
must be recorded before constructing producer and consumer clients are quite 
hard to achieve in practice. Personally I would prefer either option 1) or have 
easy way to construct producer / consumer configs from the high-level configs 
(today this is the workaround in streams, where we have to enumerate 
one-by-one).

> Make AbstractConfig.logUnused() tunable for clients
> ---
>
> Key: KAFKA-2939
> URL: https://issues.apache.org/jira/browse/KAFKA-2939
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>  Labels: newbie
> Fix For: 0.10.1.0
>
>
> Today we always log unused configs in KafkaProducer / KafkaConsumer in their 
> constructors, however for some cases like Kafka Streams that make use of 
> these clients, other configs may be passed in to configure Partitioner / 
> Serializer classes, etc. So it would be better to make this function call 
> optional to avoid printing unnecessary and confusing WARN entries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3407) ErrorLoggingCallback trims helpful diagnostic information.

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210671#comment-15210671
 ] 

ASF GitHub Bot commented on KAFKA-3407:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1079


> ErrorLoggingCallback trims helpful diagnostic information.
> --
>
> Key: KAFKA-3407
> URL: https://issues.apache.org/jira/browse/KAFKA-3407
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jeremy Custenborder
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> ErrorLoggingCallback currently only returns the message of the message 
> returned. Any inner exception or callstack is not included. This makes 
> troubleshooting more difficult. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3407 - ErrorLoggingCallback trims helpfu...

2016-03-24 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1079


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-3459) Returning zero task configurations from a connector does not properly clean up existing tasks

2016-03-24 Thread Ewen Cheslack-Postava (JIRA)

Ewen Cheslack-Postava created KAFKA-3459:


 Summary: Returning zero task configurations from a connector does 
not properly clean up existing tasks
 Key: KAFKA-3459
 URL: https://issues.apache.org/jira/browse/KAFKA-3459
 Project: Kafka
  Issue Type: Bug
  Components: copycat
Affects Versions: 0.9.0.1
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava


Instead of deleting existing tasks it just leaves existing tasks in place. If 
you're writing a connector with a variable number of inputs where it may drop 
to zero, this makes it impossible to cleanup existing tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3453; Transient test failures due to Min...

2016-03-24 Thread ijuma

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/1131


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: consumer group, why commit requests are not considered as effective heartbeats?

2016-03-24 Thread Jason Gustafson

Hi Zaiming, I responded on the user list.

-Jason

On Thu, Mar 24, 2016 at 8:31 AM, Zaiming Shi  wrote:

> Hi there!
>
> It was probably sent to the wrong list?
> Wish I can get some answer here.
>
> -- Forwarded message --
> From: Zaiming Shi 
> Date: Wed, Mar 23, 2016 at 6:03 PM
> Subject: consumer group, why commit requests are not considered as
> effective heartbeats?
> To: us...@kafka.apache.org
>
>
> Hi there!
>
> We have noticed that when committing requests are sent intensively, we
> receive IllegalGenerationId.
> Here is the settings we had problem with: session-timeout: 30 sec,
> heartbeat-rate: 3 sec.
> Problem resolved by increasing the session timeout to 180 sec.
>
> So I suppose, due to whatever reason (either the client didn't send
> heartbeat, or the broker didn't process the heartbeats in time), the
> session was considered dead in group coordinator.
>
> My question is: why commit requests can't be taken as an indicator of
> member being alive? hence not to kill the session.
>
> Regards
> -Zaiming
>

[jira] [Commented] (KAFKA-3445) ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210581#comment-15210581
 ] 

ASF GitHub Bot commented on KAFKA-3445:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1132


> ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit 
> -
>
> Key: KAFKA-3445
> URL: https://issues.apache.org/jira/browse/KAFKA-3445
> Project: Kafka
>  Issue Type: Improvement
>  Components: config
>Reporter: Ryan P
>Priority: Trivial
>  Labels: newbie
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-3445.patch
>
>
> I'll be the first to admit this is a bit nit picky any property marked with 
> Importance.HIGH should be guarded against nonsensical values. 
> With that said I would like to suggest that TASKS_MAX_CONFIG be validating 
> against a lower bound limit of 1. 
> I do understand this is unlikely to happen and the configuration is 
> nonsensical but there is no penalty for stopping someone from trying it out. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3434) Add old ConsumerRecord constructor for compatibility

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210580#comment-15210580
 ] 

ASF GitHub Bot commented on KAFKA-3434:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1123


> Add old ConsumerRecord constructor for compatibility
> 
>
> Key: KAFKA-3434
> URL: https://issues.apache.org/jira/browse/KAFKA-3434
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.0, 0.10.0.0
>
>
> After KIP-42, several new fields have been added to ConsumerRecord, all of 
> which are passed through the only constructor. It would be nice to add back 
> the old constructor for compatibility and convenience.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3445

2016-03-24 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1132


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3445) ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit

2016-03-24 Thread Ewen Cheslack-Postava (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3445:
-
   Resolution: Fixed
Fix Version/s: 0.10.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1132
[https://github.com/apache/kafka/pull/1132]

> ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit 
> -
>
> Key: KAFKA-3445
> URL: https://issues.apache.org/jira/browse/KAFKA-3445
> Project: Kafka
>  Issue Type: Improvement
>  Components: config
>Reporter: Ryan P
>Priority: Trivial
>  Labels: newbie
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-3445.patch
>
>
> I'll be the first to admit this is a bit nit picky any property marked with 
> Importance.HIGH should be guarded against nonsensical values. 
> With that said I would like to suggest that TASKS_MAX_CONFIG be validating 
> against a lower bound limit of 1. 
> I do understand this is unlikely to happen and the configuration is 
> nonsensical but there is no penalty for stopping someone from trying it out. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3453) Transient test failures due to MiniKDC port allocation strategy

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210565#comment-15210565
 ] 

ASF GitHub Bot commented on KAFKA-3453:
---

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/1131


> Transient test failures due to MiniKDC port allocation strategy
> ---
>
> Key: KAFKA-3453
> URL: https://issues.apache.org/jira/browse/KAFKA-3453
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
> Fix For: 0.10.0.1
>
>
> A number of tests, especially our consumer tests, fail transiently because 
> MiniKDC allocates ports by creating a socket, getting its port, then closing 
> it. As previously addressed in our own code, this causes problems because 
> that port can be reallocated before the process has a chance to bind a new 
> socket -- whether due to another test running in parallel or another process 
> simply binding the port first. This results in errors like this in the tests:
> {quote}
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
>   at 
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {quote}
> This is an ongoing issue that Confluent sees in its Jenkins builds, which is 
> the reason for this ticket. The real issue is actually in MiniKDC (we pass in 
> "0" for the port, but then it uses this other port allocation strategy), but 
> we either need to a) figure out a workaround or b) get a fix in upstream and 
> then update to a newer MiniKDC version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Consumer Errors...

2016-03-24 Thread vasanth loka

I am running into  ILLEGAL_GENERATION occurred while committing offsets for
group reported by ConsumerCoordinator. I have auto commit enabled.

Can someone give any tips as to what this means? After this error, the
consumer is pretty much rendered useless - unable to consume any more
messages.

Any help appreciated.

Thanks.

Re: [VOTE] KIP-43: Kafka SASL enhancements

2016-03-24 Thread Rajini Sivaram

Gwen,

Thank you. I have pinged Ismael, Harsha and Jun Rao for PR review. If any
of them has time for reviewing the PR, I will update the PR over the
weekend. If you can suggest any other reviewers, I can ping them too.

Many thanks.

On Thu, Mar 24, 2016 at 5:03 PM, Gwen Shapira  wrote:

> This can be discussed in the review.
> If there's good test coverage, is low risk and passes review and gets
> merged before Monday morning...
>
> We won't be doing an extra release candidate just for this though.
>
> Gwen
>
> On Thu, Mar 24, 2016 at 1:21 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Gwen,
> >
> > Is it still possible to include this in 0.10.0.0?
> >
> > Thanks,
> >
> > Rajini
> >
> > On Wed, Mar 23, 2016 at 11:08 PM, Gwen Shapira 
> wrote:
> >
> > > Sorry! Got distracted by the impending release!
> > >
> > > +1 on the current revision of the KIP.
> > >
> > > On Wed, Mar 23, 2016 at 3:33 PM, Harsha  wrote:
> > >
> > > > Any update on this. Gwen since the KIP is adjusted to address the
> > > > pluggable classes we should make a move on this.
> > > >
> > > > Rajini,
> > > >Can you restart the voting thread.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Wed, Mar 16, 2016, at 06:42 AM, Rajini Sivaram wrote:
> > > > > As discussed in the KIP meeting yesterday, the scope of KIP-43 has
> > been
> > > > > reduced so that it can be integrated into 0.10.0.0. The updated KIP
> > is
> > > > > here:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43%3A+Kafka+SASL+enhancements
> > > > > .
> > > > >
> > > > > Can we continue the vote on the updated KIP?
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Thu, Mar 10, 2016 at 2:09 AM, Gwen Shapira 
> > > wrote:
> > > > >
> > > > > > Harsha,
> > > > > >
> > > > > > Since you are clearly in favor of the KIP, do you mind jumping
> into
> > > > > > the discussion thread and help me understand the decision behind
> > the
> > > > > > configuration parameters only allowing a single Login and
> > > > > > CallbackHandler class? This seems too limiting to me, and while
> > > Rajini
> > > > > > is trying hard to convince me otherwise, I remain doubtful.
> Perhaps
> > > > > > (since we have similar experience with Hadoop), you can help me
> see
> > > > > > what I am missing.
> > > > > >
> > > > > > Gwen
> > > > > >
> > > > > > On Wed, Mar 9, 2016 at 12:02 PM, Harsha  wrote:
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > On Tue, Mar 8, 2016, at 02:37 AM, tao xiao wrote:
> > > > > > >> +1 (non-binding)
> > > > > > >>
> > > > > > >> On Tue, 8 Mar 2016 at 05:33 Andrew Schofield <
> > > > > > >> andrew_schofield_j...@outlook.com> wrote:
> > > > > > >>
> > > > > > >> > +1 (non-binding)
> > > > > > >> >
> > > > > > >> > 
> > > > > > >> > > From: ism...@juma.me.uk
> > > > > > >> > > Date: Mon, 7 Mar 2016 19:52:11 +
> > > > > > >> > > Subject: Re: [VOTE] KIP-43: Kafka SASL enhancements
> > > > > > >> > > To: dev@kafka.apache.org
> > > > > > >> > >
> > > > > > >> > > +1 (non-binding)
> > > > > > >> > >
> > > > > > >> > > On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
> > > > > > >> > > rajinisiva...@googlemail.com> wrote:
> > > > > > >> > >
> > > > > > >> > >> I would like to start the voting process for *KIP-43:
> Kafka
> > > > SASL
> > > > > > >> > >> enhancements*. This KIP extends the SASL implementation
> in
> > > > Kafka to
> > > > > > >> > support
> > > > > > >> > >> new SASL mechanisms to enable Kafka to be integrated with
> > > > different
> > > > > > >> > >> authentication servers.
> > > > > > >> > >>
> > > > > > >> > >> The KIP is available here for reference:
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
> > > > > > >> > >>
> > > > > > >> > >> And here's is a link to the discussion on the mailing
> list:
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> >
> > > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> > >> Thank you...
> > > > > > >> > >>
> > > > > > >> > >> Regards,
> > > > > > >> > >>
> > > > > > >> > >> Rajini
> > > > > > >> > >>
> > > > > > >> >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>



-- 
Regards,

Rajini

Build failed in Jenkins: kafka-trunk-jdk8 #479

2016-03-24 Thread Apache Jenkins Server

See 

Changes:

[me] KAFKA-3445: Validate TASKS_MAX_CONFIG's lower bound

--
[...truncated 1637 lines...]

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testLogToClean PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.log.OffsetMapTest > testClear PASSED

kafka.log.OffsetMapTest > testBasicValidation PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupStable PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatIllegalGeneration 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testDescribeGroupWrongCoordinator PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupRebalancing 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaderFailureInSyncGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFromIllegalGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testInvalidGroupId PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesStableGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatDuringRebalanceCausesRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentGroupProtocol PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupEmptyAssignment 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetWithDefaultGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedLeaderShouldRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesRebalancingGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFollowerAfterLeader PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testCommitOffsetInAwaitingSync 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testJoinGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentProtocolType PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetFromUnknownGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testLeaveGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerNewGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedFollowerDoesNotRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest >

Re: [VOTE] KIP-43: Kafka SASL enhancements

2016-03-24 Thread Gwen Shapira

This can be discussed in the review.
If there's good test coverage, is low risk and passes review and gets
merged before Monday morning...

We won't be doing an extra release candidate just for this though.

Gwen

On Thu, Mar 24, 2016 at 1:21 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Gwen,
>
> Is it still possible to include this in 0.10.0.0?
>
> Thanks,
>
> Rajini
>
> On Wed, Mar 23, 2016 at 11:08 PM, Gwen Shapira  wrote:
>
> > Sorry! Got distracted by the impending release!
> >
> > +1 on the current revision of the KIP.
> >
> > On Wed, Mar 23, 2016 at 3:33 PM, Harsha  wrote:
> >
> > > Any update on this. Gwen since the KIP is adjusted to address the
> > > pluggable classes we should make a move on this.
> > >
> > > Rajini,
> > >Can you restart the voting thread.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Wed, Mar 16, 2016, at 06:42 AM, Rajini Sivaram wrote:
> > > > As discussed in the KIP meeting yesterday, the scope of KIP-43 has
> been
> > > > reduced so that it can be integrated into 0.10.0.0. The updated KIP
> is
> > > > here:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43%3A+Kafka+SASL+enhancements
> > > > .
> > > >
> > > > Can we continue the vote on the updated KIP?
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Thu, Mar 10, 2016 at 2:09 AM, Gwen Shapira 
> > wrote:
> > > >
> > > > > Harsha,
> > > > >
> > > > > Since you are clearly in favor of the KIP, do you mind jumping into
> > > > > the discussion thread and help me understand the decision behind
> the
> > > > > configuration parameters only allowing a single Login and
> > > > > CallbackHandler class? This seems too limiting to me, and while
> > Rajini
> > > > > is trying hard to convince me otherwise, I remain doubtful. Perhaps
> > > > > (since we have similar experience with Hadoop), you can help me see
> > > > > what I am missing.
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Wed, Mar 9, 2016 at 12:02 PM, Harsha  wrote:
> > > > > > +1 (binding)
> > > > > >
> > > > > > On Tue, Mar 8, 2016, at 02:37 AM, tao xiao wrote:
> > > > > >> +1 (non-binding)
> > > > > >>
> > > > > >> On Tue, 8 Mar 2016 at 05:33 Andrew Schofield <
> > > > > >> andrew_schofield_j...@outlook.com> wrote:
> > > > > >>
> > > > > >> > +1 (non-binding)
> > > > > >> >
> > > > > >> > 
> > > > > >> > > From: ism...@juma.me.uk
> > > > > >> > > Date: Mon, 7 Mar 2016 19:52:11 +
> > > > > >> > > Subject: Re: [VOTE] KIP-43: Kafka SASL enhancements
> > > > > >> > > To: dev@kafka.apache.org
> > > > > >> > >
> > > > > >> > > +1 (non-binding)
> > > > > >> > >
> > > > > >> > > On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
> > > > > >> > > rajinisiva...@googlemail.com> wrote:
> > > > > >> > >
> > > > > >> > >> I would like to start the voting process for *KIP-43: Kafka
> > > SASL
> > > > > >> > >> enhancements*. This KIP extends the SASL implementation in
> > > Kafka to
> > > > > >> > support
> > > > > >> > >> new SASL mechanisms to enable Kafka to be integrated with
> > > different
> > > > > >> > >> authentication servers.
> > > > > >> > >>
> > > > > >> > >> The KIP is available here for reference:
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
> > > > > >> > >>
> > > > > >> > >> And here's is a link to the discussion on the mailing list:
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> >
> > > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> Thank you...
> > > > > >> > >>
> > > > > >> > >> Regards,
> > > > > >> > >>
> > > > > >> > >> Rajini
> > > > > >> > >>
> > > > > >> >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>

Re: Kafka Connect ++ Kafka Streams

2016-03-24 Thread Ewen Cheslack-Postava

On Thu, Mar 24, 2016 at 4:46 AM, Michal Hariš 
wrote:

> Hello Kafka people!
>
> Great to see Kafka Streams coming along, the design validates (and in many
> way supersedes) my own findings from working with various stream processing
> systems/frameworks and eventually ending-up using just a small custom
> library built directly around Kafka.
>
> I have set out yesterday to translate Hello Samza (the wikipedia feed
> example) into Kafka Streams application. Now because this workflow starts
> by polling wikipedia IRC and publishes to a topic from which the stream
> processors pick-up it would be nice to have this first part done by Kafka
> Connect but:
>
> 1. IRC channels are not seekable and Kafka Connect architecture claims that
> all sources must be seekable - is this still suitable ? (I guess yes as
> FileStreamSourceTask can read from stdin which is similar)
>

They need to be seekable in order to guarantee delivery. If you're fine
with an outage causing you to miss some data, then you don't need the
source to be seekable. However, keep in mind that in distributed mode,
there will be brief periods where work is being rebalanced across workers
and data will not be processed. These are windows where you could easily
lose data if you can't track offsets and recover events that occurred
during the rebalance process. You can of course stick with standalone mode,
but then you lose some of the fault tolerance features.

>
> 2. I would like to have ConnectEmbedded (as opposed to ConnectStandalone or
> ConnectDistributed) which is similar to ConnectDistributed, just without
> the rest server - i.e. say I have the WikipediaFeedConnector and I want to
> launch it programatically from all the instances along-side the Kafka
> Streams - but reusing the connect distributed coordination so that only one
> instance actually reads the IRC data but another instance picks up work if
> that one dies - does it sound like a bad idea for some design reason ? -
> the only problem I see is rather technical that the coordination process
> uses the rest server for some actions.
>

This is planned and is described in the KIP that added Kafka Connect -
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767
However, good embedded support required enough of Kafka Streams to be
defined to ensure good integration. Now that both components are available,
this is a project we'll want to start tackling (but will not be in the next
0.10.0.0 release).

-Ewen

>
> Cheers,
> Michal
>

-- 
Thanks,
Ewen

[jira] [Resolved] (KAFKA-3407) ErrorLoggingCallback trims helpful diagnostic information.

2016-03-24 Thread Ewen Cheslack-Postava (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3407.
--
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1079
[https://github.com/apache/kafka/pull/1079]

> ErrorLoggingCallback trims helpful diagnostic information.
> --
>
> Key: KAFKA-3407
> URL: https://issues.apache.org/jira/browse/KAFKA-3407
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jeremy Custenborder
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> ErrorLoggingCallback currently only returns the message of the message 
> returned. Any inner exception or callstack is not included. This makes 
> troubleshooting more difficult. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #480

2016-03-24 Thread Apache Jenkins Server

See 

Changes:

[me] KAFKA-3407 - ErrorLoggingCallback trims helpful diagnostic information.

--
[...truncated 1631 lines...]

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testLogToClean PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.log.OffsetMapTest > testClear PASSED

kafka.log.OffsetMapTest > testBasicValidation PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupStable PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatIllegalGeneration 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testDescribeGroupWrongCoordinator PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupRebalancing 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaderFailureInSyncGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFromIllegalGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testInvalidGroupId PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesStableGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatDuringRebalanceCausesRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentGroupProtocol PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupEmptyAssignment 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetWithDefaultGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedLeaderShouldRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesRebalancingGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFollowerAfterLeader PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testCommitOffsetInAwaitingSync 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testJoinGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentProtocolType PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetFromUnknownGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testLeaveGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerNewGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedFollowerDoesNotRebalance PASSED

[GitHub] kafka pull request: KAFKA-3460: Remove old 0.7 KafkaMigrationTool

2016-03-24 Thread granthenke

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/1136

KAFKA-3460: Remove old 0.7 KafkaMigrationTool



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka remove-07-migration

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1136.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1136


commit 6f1622cceeef41f80b33c3ff8151156fa73e8dcf
Author: Grant Henke 
Date:   2016-03-24T19:59:21Z

KAFKA-3460: Remove old 0.7 KafkaMigrationTool




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3460) Remove old 0.7 KafkaMigrationTool

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210868#comment-15210868
 ] 

ASF GitHub Bot commented on KAFKA-3460:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/1136

KAFKA-3460: Remove old 0.7 KafkaMigrationTool



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka remove-07-migration

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1136.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1136


commit 6f1622cceeef41f80b33c3ff8151156fa73e8dcf
Author: Grant Henke 
Date:   2016-03-24T19:59:21Z

KAFKA-3460: Remove old 0.7 KafkaMigrationTool




> Remove old 0.7 KafkaMigrationTool
> -
>
> Key: KAFKA-3460
> URL: https://issues.apache.org/jira/browse/KAFKA-3460
> Project: Kafka
>  Issue Type: Task
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.10.0.0
>
>
> Unless we are supporting directly upgrading from 0.7 to 0.10 the 
> KafkaMigrationTool should be cleaned up. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3460) Remove old 0.7 KafkaMigrationTool

2016-03-24 Thread Grant Henke (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3460:
---
Status: Patch Available  (was: Open)

> Remove old 0.7 KafkaMigrationTool
> -
>
> Key: KAFKA-3460
> URL: https://issues.apache.org/jira/browse/KAFKA-3460
> Project: Kafka
>  Issue Type: Task
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.10.0.0
>
>
> Unless we are supporting directly upgrading from 0.7 to 0.10 the 
> KafkaMigrationTool should be cleaned up. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3460) Remove old 0.7 KafkaMigrationTool

2016-03-24 Thread Grant Henke (JIRA)

Grant Henke created KAFKA-3460:
--

 Summary: Remove old 0.7 KafkaMigrationTool
 Key: KAFKA-3460
 URL: https://issues.apache.org/jira/browse/KAFKA-3460
 Project: Kafka
  Issue Type: Task
Affects Versions: 0.9.0.0
Reporter: Grant Henke
Assignee: Grant Henke
 Fix For: 0.10.0.0


Unless we are supporting directly upgrading from 0.7 to 0.10 the 
KafkaMigrationTool should be cleaned up. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Messages corrupted in kafka

2016-03-24 Thread Becket Qin

You mentioned that you saw few corrupted messages, (< 0.1%). If so are you
able to see some corrupted messages if you produce, say, 10M messages?

On Wed, Mar 23, 2016 at 9:40 PM, sunil kalva  wrote:

>  I am using java client and kafka 0.8.2, since events are corrupted in
> kafka broker i cant read and replay them again.
>
> On Thu, Mar 24, 2016 at 9:42 AM, Becket Qin  wrote:
>
> > Hi Sunil,
> >
> > The messages in Kafka has a CRC stored with each of them. When consumer
> > receives a message, it will compute the CRC from the message bytes and
> > compare it to the stored CRC. If the computed CRC and stored CRC does not
> > match, that indicates the message has corrupted. I am not sure in your
> case
> > why the message is corrupted. Corrupted message seems to  be pretty rare
> > because the broker actually validate the CRC before it stores the
> messages
> > on to the disk.
> >
> > Is this problem reproduceable? If so, can you find out the messages that
> > are corrupted? Also, are you using the Java clients or some other
> clients?
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Mar 23, 2016 at 8:28 PM, sunil kalva 
> > wrote:
> >
> > > can some one help me out here.
> > >
> > > On Wed, Mar 23, 2016 at 7:36 PM, sunil kalva 
> > > wrote:
> > >
> > > > Hi
> > > > I am seeing few messages getting corrupted in kafka, It is not
> > happening
> > > > frequently and percentage is also very very less (less than 0.1%).
> > > >
> > > > Basically i am publishing thrift events in byte array format to kafka
> > > > topics(with out encoding like base64), and i also see more events
> than
> > i
> > > > publish (i confirm this by looking at the offset for that topic).
> > > > For example if i publish 100 events and i see 110 as offset for that
> > > topic
> > > > (since it is in production i could not get exact messages which
> causing
> > > > this problem, and we will only realize this problem when we consume
> > > because
> > > > our thrift deserialization fails).
> > > >
> > > > So my question is, is there any magic byte which actually determines
> > the
> > > > boundary of the message which is same as the byte i am sending or or
> > for
> > > > any n/w issues messages get chopped and stores as one message to
> > multiple
> > > > messages on server side ?
> > > >
> > > > tx
> > > > SunilKalva
> > > >
> > >
> >
>

[GitHub] kafka pull request: HOTFIX: set timestamp in SinkNode

2016-03-24 Thread ymatsuda

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/1137

HOTFIX: set timestamp in SinkNode

@guozhangwang 
Setting the timestamp in produced records in SinkNode. This forces the 
producer record's timestamp same as the context's timestamp.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka set_timestamp_in_sinknode

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1137.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1137


commit bd5dd3efae0784261728afddc7acd86612bef610
Author: Yasuhiro Matsuda 
Date:   2016-03-24T22:05:30Z

HOTFIX: set timestamp in SinkNode




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-43: Kafka SASL enhancements

2016-03-24 Thread Gwen Shapira

Rajini,

I think the vote didn't pass yet?
If I can see correctly, Harsha and I are the only committers who voted, so
we are missing a 3rd vote.

Gwen

On Thu, Mar 24, 2016 at 11:24 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Gwen,
>
> Thank you. I have pinged Ismael, Harsha and Jun Rao for PR review. If any
> of them has time for reviewing the PR, I will update the PR over the
> weekend. If you can suggest any other reviewers, I can ping them too.
>
> Many thanks.
>
> On Thu, Mar 24, 2016 at 5:03 PM, Gwen Shapira  wrote:
>
> > This can be discussed in the review.
> > If there's good test coverage, is low risk and passes review and gets
> > merged before Monday morning...
> >
> > We won't be doing an extra release candidate just for this though.
> >
> > Gwen
> >
> > On Thu, Mar 24, 2016 at 1:21 AM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > Gwen,
> > >
> > > Is it still possible to include this in 0.10.0.0?
> > >
> > > Thanks,
> > >
> > > Rajini
> > >
> > > On Wed, Mar 23, 2016 at 11:08 PM, Gwen Shapira 
> > wrote:
> > >
> > > > Sorry! Got distracted by the impending release!
> > > >
> > > > +1 on the current revision of the KIP.
> > > >
> > > > On Wed, Mar 23, 2016 at 3:33 PM, Harsha  wrote:
> > > >
> > > > > Any update on this. Gwen since the KIP is adjusted to address the
> > > > > pluggable classes we should make a move on this.
> > > > >
> > > > > Rajini,
> > > > >Can you restart the voting thread.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Wed, Mar 16, 2016, at 06:42 AM, Rajini Sivaram wrote:
> > > > > > As discussed in the KIP meeting yesterday, the scope of KIP-43
> has
> > > been
> > > > > > reduced so that it can be integrated into 0.10.0.0. The updated
> KIP
> > > is
> > > > > > here:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43%3A+Kafka+SASL+enhancements
> > > > > > .
> > > > > >
> > > > > > Can we continue the vote on the updated KIP?
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Thu, Mar 10, 2016 at 2:09 AM, Gwen Shapira  >
> > > > wrote:
> > > > > >
> > > > > > > Harsha,
> > > > > > >
> > > > > > > Since you are clearly in favor of the KIP, do you mind jumping
> > into
> > > > > > > the discussion thread and help me understand the decision
> behind
> > > the
> > > > > > > configuration parameters only allowing a single Login and
> > > > > > > CallbackHandler class? This seems too limiting to me, and while
> > > > Rajini
> > > > > > > is trying hard to convince me otherwise, I remain doubtful.
> > Perhaps
> > > > > > > (since we have similar experience with Hadoop), you can help me
> > see
> > > > > > > what I am missing.
> > > > > > >
> > > > > > > Gwen
> > > > > > >
> > > > > > > On Wed, Mar 9, 2016 at 12:02 PM, Harsha 
> wrote:
> > > > > > > > +1 (binding)
> > > > > > > >
> > > > > > > > On Tue, Mar 8, 2016, at 02:37 AM, tao xiao wrote:
> > > > > > > >> +1 (non-binding)
> > > > > > > >>
> > > > > > > >> On Tue, 8 Mar 2016 at 05:33 Andrew Schofield <
> > > > > > > >> andrew_schofield_j...@outlook.com> wrote:
> > > > > > > >>
> > > > > > > >> > +1 (non-binding)
> > > > > > > >> >
> > > > > > > >> > 
> > > > > > > >> > > From: ism...@juma.me.uk
> > > > > > > >> > > Date: Mon, 7 Mar 2016 19:52:11 +
> > > > > > > >> > > Subject: Re: [VOTE] KIP-43: Kafka SASL enhancements
> > > > > > > >> > > To: dev@kafka.apache.org
> > > > > > > >> > >
> > > > > > > >> > > +1 (non-binding)
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
> > > > > > > >> > > rajinisiva...@googlemail.com> wrote:
> > > > > > > >> > >
> > > > > > > >> > >> I would like to start the voting process for *KIP-43:
> > Kafka
> > > > > SASL
> > > > > > > >> > >> enhancements*. This KIP extends the SASL implementation
> > in
> > > > > Kafka to
> > > > > > > >> > support
> > > > > > > >> > >> new SASL mechanisms to enable Kafka to be integrated
> with
> > > > > different
> > > > > > > >> > >> authentication servers.
> > > > > > > >> > >>
> > > > > > > >> > >> The KIP is available here for reference:
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
> > > > > > > >> > >>
> > > > > > > >> > >> And here's is a link to the discussion on the mailing
> > list:
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> >
> > > > > > >
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> Thank you...
> > > > > > > >> > >>
> > > > > > > >> > >> Regards,
> > > > > > > >>

Re: [VOTE] KIP-43: Kafka SASL enhancements

2016-03-24 Thread Rajini Sivaram

Gwen,

Ah, I clearly don't know the rules. So it looks like it would not really be
possible to get this into 0.10.0.0 after all.

Rajini

On Thu, Mar 24, 2016 at 8:38 PM, Gwen Shapira  wrote:

> Rajini,
>
> I think the vote didn't pass yet?
> If I can see correctly, Harsha and I are the only committers who voted, so
> we are missing a 3rd vote.
>
> Gwen
>
> On Thu, Mar 24, 2016 at 11:24 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Gwen,
> >
> > Thank you. I have pinged Ismael, Harsha and Jun Rao for PR review. If any
> > of them has time for reviewing the PR, I will update the PR over the
> > weekend. If you can suggest any other reviewers, I can ping them too.
> >
> > Many thanks.
> >
> > On Thu, Mar 24, 2016 at 5:03 PM, Gwen Shapira  wrote:
> >
> > > This can be discussed in the review.
> > > If there's good test coverage, is low risk and passes review and gets
> > > merged before Monday morning...
> > >
> > > We won't be doing an extra release candidate just for this though.
> > >
> > > Gwen
> > >
> > > On Thu, Mar 24, 2016 at 1:21 AM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > > > Gwen,
> > > >
> > > > Is it still possible to include this in 0.10.0.0?
> > > >
> > > > Thanks,
> > > >
> > > > Rajini
> > > >
> > > > On Wed, Mar 23, 2016 at 11:08 PM, Gwen Shapira 
> > > wrote:
> > > >
> > > > > Sorry! Got distracted by the impending release!
> > > > >
> > > > > +1 on the current revision of the KIP.
> > > > >
> > > > > On Wed, Mar 23, 2016 at 3:33 PM, Harsha  wrote:
> > > > >
> > > > > > Any update on this. Gwen since the KIP is adjusted to address the
> > > > > > pluggable classes we should make a move on this.
> > > > > >
> > > > > > Rajini,
> > > > > >Can you restart the voting thread.
> > > > > >
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > > On Wed, Mar 16, 2016, at 06:42 AM, Rajini Sivaram wrote:
> > > > > > > As discussed in the KIP meeting yesterday, the scope of KIP-43
> > has
> > > > been
> > > > > > > reduced so that it can be integrated into 0.10.0.0. The updated
> > KIP
> > > > is
> > > > > > > here:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43%3A+Kafka+SASL+enhancements
> > > > > > > .
> > > > > > >
> > > > > > > Can we continue the vote on the updated KIP?
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Thu, Mar 10, 2016 at 2:09 AM, Gwen Shapira <
> g...@confluent.io
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Harsha,
> > > > > > > >
> > > > > > > > Since you are clearly in favor of the KIP, do you mind
> jumping
> > > into
> > > > > > > > the discussion thread and help me understand the decision
> > behind
> > > > the
> > > > > > > > configuration parameters only allowing a single Login and
> > > > > > > > CallbackHandler class? This seems too limiting to me, and
> while
> > > > > Rajini
> > > > > > > > is trying hard to convince me otherwise, I remain doubtful.
> > > Perhaps
> > > > > > > > (since we have similar experience with Hadoop), you can help
> me
> > > see
> > > > > > > > what I am missing.
> > > > > > > >
> > > > > > > > Gwen
> > > > > > > >
> > > > > > > > On Wed, Mar 9, 2016 at 12:02 PM, Harsha 
> > wrote:
> > > > > > > > > +1 (binding)
> > > > > > > > >
> > > > > > > > > On Tue, Mar 8, 2016, at 02:37 AM, tao xiao wrote:
> > > > > > > > >> +1 (non-binding)
> > > > > > > > >>
> > > > > > > > >> On Tue, 8 Mar 2016 at 05:33 Andrew Schofield <
> > > > > > > > >> andrew_schofield_j...@outlook.com> wrote:
> > > > > > > > >>
> > > > > > > > >> > +1 (non-binding)
> > > > > > > > >> >
> > > > > > > > >> > 
> > > > > > > > >> > > From: ism...@juma.me.uk
> > > > > > > > >> > > Date: Mon, 7 Mar 2016 19:52:11 +
> > > > > > > > >> > > Subject: Re: [VOTE] KIP-43: Kafka SASL enhancements
> > > > > > > > >> > > To: dev@kafka.apache.org
> > > > > > > > >> > >
> > > > > > > > >> > > +1 (non-binding)
> > > > > > > > >> > >
> > > > > > > > >> > > On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
> > > > > > > > >> > > rajinisiva...@googlemail.com> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > >> I would like to start the voting process for *KIP-43:
> > > Kafka
> > > > > > SASL
> > > > > > > > >> > >> enhancements*. This KIP extends the SASL
> implementation
> > > in
> > > > > > Kafka to
> > > > > > > > >> > support
> > > > > > > > >> > >> new SASL mechanisms to enable Kafka to be integrated
> > with
> > > > > > different
> > > > > > > > >> > >> authentication servers.
> > > > > > > > >> > >>
> > > > > > > > >> > >> The KIP is available here for reference:
> > > > > > > > >> > >>
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-43: Kafka SASL enhancements

2016-03-24 Thread Gwen Shapira

I'm afraid it will be a challenge.

I see few options:
1. Jun should be back in the office tomorrow. If he votes +1 and agrees
that the PR is ready to merge and is safe and important enough to
double-commit - this could get in yet.
2. Same as above, but not in time for the Monday release candidate. In this
case, we can get it into 0.10.0.0 if we find other blockers and need to
roll-out another RC.
3. (most likely) We will finish the vote and review but not in time for
0.10.0.0. In this case, 0.10.1.0.0 should be out in around 3 month, and
we'll get it in there. You'll be in good company with KIP-35, KIP-4, KIP-48
and few other things that are close to done, are super critical but are
just not ready in time. Thats why we are trying to release more often.

Gwen

On Thu, Mar 24, 2016 at 2:08 PM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Gwen,
>
> Ah, I clearly don't know the rules. So it looks like it would not really be
> possible to get this into 0.10.0.0 after all.
>
> Rajini
>
> On Thu, Mar 24, 2016 at 8:38 PM, Gwen Shapira  wrote:
>
> > Rajini,
> >
> > I think the vote didn't pass yet?
> > If I can see correctly, Harsha and I are the only committers who voted,
> so
> > we are missing a 3rd vote.
> >
> > Gwen
> >
> > On Thu, Mar 24, 2016 at 11:24 AM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > Gwen,
> > >
> > > Thank you. I have pinged Ismael, Harsha and Jun Rao for PR review. If
> any
> > > of them has time for reviewing the PR, I will update the PR over the
> > > weekend. If you can suggest any other reviewers, I can ping them too.
> > >
> > > Many thanks.
> > >
> > > On Thu, Mar 24, 2016 at 5:03 PM, Gwen Shapira 
> wrote:
> > >
> > > > This can be discussed in the review.
> > > > If there's good test coverage, is low risk and passes review and gets
> > > > merged before Monday morning...
> > > >
> > > > We won't be doing an extra release candidate just for this though.
> > > >
> > > > Gwen
> > > >
> > > > On Thu, Mar 24, 2016 at 1:21 AM, Rajini Sivaram <
> > > > rajinisiva...@googlemail.com> wrote:
> > > >
> > > > > Gwen,
> > > > >
> > > > > Is it still possible to include this in 0.10.0.0?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Wed, Mar 23, 2016 at 11:08 PM, Gwen Shapira 
> > > > wrote:
> > > > >
> > > > > > Sorry! Got distracted by the impending release!
> > > > > >
> > > > > > +1 on the current revision of the KIP.
> > > > > >
> > > > > > On Wed, Mar 23, 2016 at 3:33 PM, Harsha  wrote:
> > > > > >
> > > > > > > Any update on this. Gwen since the KIP is adjusted to address
> the
> > > > > > > pluggable classes we should make a move on this.
> > > > > > >
> > > > > > > Rajini,
> > > > > > >Can you restart the voting thread.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Harsha
> > > > > > >
> > > > > > > On Wed, Mar 16, 2016, at 06:42 AM, Rajini Sivaram wrote:
> > > > > > > > As discussed in the KIP meeting yesterday, the scope of
> KIP-43
> > > has
> > > > > been
> > > > > > > > reduced so that it can be integrated into 0.10.0.0. The
> updated
> > > KIP
> > > > > is
> > > > > > > > here:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43%3A+Kafka+SASL+enhancements
> > > > > > > > .
> > > > > > > >
> > > > > > > > Can we continue the vote on the updated KIP?
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > > On Thu, Mar 10, 2016 at 2:09 AM, Gwen Shapira <
> > g...@confluent.io
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Harsha,
> > > > > > > > >
> > > > > > > > > Since you are clearly in favor of the KIP, do you mind
> > jumping
> > > > into
> > > > > > > > > the discussion thread and help me understand the decision
> > > behind
> > > > > the
> > > > > > > > > configuration parameters only allowing a single Login and
> > > > > > > > > CallbackHandler class? This seems too limiting to me, and
> > while
> > > > > > Rajini
> > > > > > > > > is trying hard to convince me otherwise, I remain doubtful.
> > > > Perhaps
> > > > > > > > > (since we have similar experience with Hadoop), you can
> help
> > me
> > > > see
> > > > > > > > > what I am missing.
> > > > > > > > >
> > > > > > > > > Gwen
> > > > > > > > >
> > > > > > > > > On Wed, Mar 9, 2016 at 12:02 PM, Harsha 
> > > wrote:
> > > > > > > > > > +1 (binding)
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 8, 2016, at 02:37 AM, tao xiao wrote:
> > > > > > > > > >> +1 (non-binding)
> > > > > > > > > >>
> > > > > > > > > >> On Tue, 8 Mar 2016 at 05:33 Andrew Schofield <
> > > > > > > > > >> andrew_schofield_j...@outlook.com> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > +1 (non-binding)
> > > > > > > > > >> >
> > > > > > > > > >> > 
> > > > > > > > > >> > > From:

[jira] [Commented] (KAFKA-2939) Make AbstractConfig.logUnused() tunable for clients

2016-03-24 Thread Ewen Cheslack-Postava (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209825#comment-15209825
 ] 

Ewen Cheslack-Postava commented on KAFKA-2939:
--

Probably worth thinking through how this is going to be impacted by streams and 
connect. Connect already has usages that don't do the streams filtering (and 
where I'm not sure how easy that would be). We have internal producers and 
consumers where we just pass the original configs from the worker. Even cases 
like serializers aren't really handled correctly -- accessing values directly 
works, but iterating over Map entries does not mark them as used. For 
serializers this isn't necessarily an issue, but connectors may more frequently 
have a non-fixed set config keys. In connect we also had to introduce prefixing 
due to potential conflicts because there are just a lot more components to 
configure (thus the newer `AbstractConfig.originalsWithPrefix()` method). And 
connect also has to deal with generic connector configs (things like a `topic` 
list that is managed by the framework) and connector-specific configs (the 
majority of the configs).

Not all of this *has* to be managed via ConfigDef/AbstractConfig and logged, 
but it is today and the current model makes it difficult to account for all 
these different use cases... Overall, I don't think things are actually as 
simple as the original use case accounted for -- I think in the constrained 
context of the clients it probably made sense, but doesn't actually work for 
all the cases we've extended ConfigDef/AbstractConfig to.

Another relevant question is: how often do we think (or know) that this has 
caught real issues? I get the motivation behind it, but I'm not entirely 
convinced of its practical benefit. Especially given that it has not worked as 
expected for at least a couple of releases

> Make AbstractConfig.logUnused() tunable for clients
> ---
>
> Key: KAFKA-2939
> URL: https://issues.apache.org/jira/browse/KAFKA-2939
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>  Labels: newbie
> Fix For: 0.10.1.0
>
>
> Today we always log unused configs in KafkaProducer / KafkaConsumer in their 
> constructors, however for some cases like Kafka Streams that make use of 
> these clients, other configs may be passed in to configure Partitioner / 
> Serializer classes, etc. So it would be better to make this function call 
> optional to avoid printing unnecessary and confusing WARN entries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3463: change default receive buffer size...

2016-03-24 Thread hachikuji

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1140

KAFKA-3463: change default receive buffer size for consumer to 64K



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-3463

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1140.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1140


commit 3bd92aea373dad3c8cbdc0ee3f99e7d035f99b63
Author: Jason Gustafson 
Date:   2016-03-25T00:57:31Z

KAFKA-3463: change default receive buffer size for consumer to 64K




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3463) Change default consumer receive buffer size to 64K

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211231#comment-15211231
 ] 

ASF GitHub Bot commented on KAFKA-3463:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1140

KAFKA-3463: change default receive buffer size for consumer to 64K



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-3463

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1140.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1140


commit 3bd92aea373dad3c8cbdc0ee3f99e7d035f99b63
Author: Jason Gustafson 
Date:   2016-03-25T00:57:31Z

KAFKA-3463: change default receive buffer size for consumer to 64K




> Change default consumer receive buffer size to 64K
> --
>
> Key: KAFKA-3463
> URL: https://issues.apache.org/jira/browse/KAFKA-3463
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.0.0
>
>
> In KAFKA-3135, users have reported a strange pause when consuming data using 
> the new consumer. This can be easily reproduced with the console consumer, 
> but the root cause has so far proven elusive. Interestingly, the pause can 
> also be reproduced with the old consumer if you change the socket buffer size 
> to 32K to match the new consumer's default. Similarly, by increasing the new 
> consumer's connection receive buffer size to 64K to match the old consumer's 
> default, the problem seems to be mitigated (I am unable to reproduce it 
> locally with this setting, though others have reported that its impact is 
> merely reduced). Since there doesn't appear to be a good reason to have 
> lowered the default for the new consumer, we may as well revert to the old 
> consumer's default of 64K and accept the mitigation until an actual fix is 
> found. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3464) Connect security system tests

2016-03-24 Thread Ewen Cheslack-Postava (JIRA)

Ewen Cheslack-Postava created KAFKA-3464:


 Summary: Connect security system tests
 Key: KAFKA-3464
 URL: https://issues.apache.org/jira/browse/KAFKA-3464
 Project: Kafka
  Issue Type: Bug
  Components: copycat
Affects Versions: 0.9.0.1
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava


We need to validate that Connect can actually work with security enabled. 
System tests can easily cover this since they will be a small modification of 
existing connect tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3464: Add system tests for Connect with ...

2016-03-24 Thread ewencp

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/1141

KAFKA-3464: Add system tests for Connect with Kafka security enabled



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-3464-connect-security-system-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1141


commit c75a07b9e32a3c80dbeae308bf297fc256ffdf5f
Author: Ewen Cheslack-Postava 
Date:   2016-03-25T01:17:34Z

KAFKA-3464: Add system tests for Connect with Kafka security enabled




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3464) Connect security system tests

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211241#comment-15211241
 ] 

ASF GitHub Bot commented on KAFKA-3464:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/1141

KAFKA-3464: Add system tests for Connect with Kafka security enabled



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-3464-connect-security-system-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1141


commit c75a07b9e32a3c80dbeae308bf297fc256ffdf5f
Author: Ewen Cheslack-Postava 
Date:   2016-03-25T01:17:34Z

KAFKA-3464: Add system tests for Connect with Kafka security enabled




> Connect security system tests
> -
>
> Key: KAFKA-3464
> URL: https://issues.apache.org/jira/browse/KAFKA-3464
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Affects Versions: 0.9.0.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>
> We need to validate that Connect can actually work with security enabled. 
> System tests can easily cover this since they will be a small modification of 
> existing connect tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

KStream/KTable prioritization/initial offset

2016-03-24 Thread Greg Fodor

Really digging Kafka Streams so far, nice work all. I'm interested in
being able to materialize one or more KTables in full before the rest
of the topology begins processing messages. This seems fundamentally
useful since it allows you to get your database tables replicated up
off the change stream topics from Connect before the stream processing
workload starts.

In Samza we have bootstrap streams and stream prioritization to help
facilitate this. What seems desirable for Kafka Streams is:

- Per-source prioritization (by defaulting to >0, setting the stream
priority to 0 effectively bootstraps it.)
- Per-source initial offset settings (earliest or latest, default to latest)

To solve the KTable materialization problem, you'd set priority to 0
for its source and the source offset setting to earliest.

Right now it appears the only control you have for re-processing is
AUTO_OFFSET_RESET_CONFIG, but I believe this is a global setting for
the consumers, and hence, the entire job. Beyond that, I don't see any
way to prioritize stream consumption at all, so your KTables will be
getting materialized while the general stream processing work is
running concurrently.

I wanted to see if this case is actually supported already and I'm
missing something, or if not, if these options make sense. If this
seems reasonable and it's not too complicated, I could possibly try to
get together a patch. If so, any tips on implementing this would be
helpful as well. Thanks!

-Greg

Re: [VOTE] KIP-43: Kafka SASL enhancements

2016-03-24 Thread Rajini Sivaram

Gwen,

Is it still possible to include this in 0.10.0.0?

Thanks,

Rajini

On Wed, Mar 23, 2016 at 11:08 PM, Gwen Shapira  wrote:

> Sorry! Got distracted by the impending release!
>
> +1 on the current revision of the KIP.
>
> On Wed, Mar 23, 2016 at 3:33 PM, Harsha  wrote:
>
> > Any update on this. Gwen since the KIP is adjusted to address the
> > pluggable classes we should make a move on this.
> >
> > Rajini,
> >Can you restart the voting thread.
> >
> > Thanks,
> > Harsha
> >
> > On Wed, Mar 16, 2016, at 06:42 AM, Rajini Sivaram wrote:
> > > As discussed in the KIP meeting yesterday, the scope of KIP-43 has been
> > > reduced so that it can be integrated into 0.10.0.0. The updated KIP is
> > > here:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43%3A+Kafka+SASL+enhancements
> > > .
> > >
> > > Can we continue the vote on the updated KIP?
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > > On Thu, Mar 10, 2016 at 2:09 AM, Gwen Shapira 
> wrote:
> > >
> > > > Harsha,
> > > >
> > > > Since you are clearly in favor of the KIP, do you mind jumping into
> > > > the discussion thread and help me understand the decision behind the
> > > > configuration parameters only allowing a single Login and
> > > > CallbackHandler class? This seems too limiting to me, and while
> Rajini
> > > > is trying hard to convince me otherwise, I remain doubtful. Perhaps
> > > > (since we have similar experience with Hadoop), you can help me see
> > > > what I am missing.
> > > >
> > > > Gwen
> > > >
> > > > On Wed, Mar 9, 2016 at 12:02 PM, Harsha  wrote:
> > > > > +1 (binding)
> > > > >
> > > > > On Tue, Mar 8, 2016, at 02:37 AM, tao xiao wrote:
> > > > >> +1 (non-binding)
> > > > >>
> > > > >> On Tue, 8 Mar 2016 at 05:33 Andrew Schofield <
> > > > >> andrew_schofield_j...@outlook.com> wrote:
> > > > >>
> > > > >> > +1 (non-binding)
> > > > >> >
> > > > >> > 
> > > > >> > > From: ism...@juma.me.uk
> > > > >> > > Date: Mon, 7 Mar 2016 19:52:11 +
> > > > >> > > Subject: Re: [VOTE] KIP-43: Kafka SASL enhancements
> > > > >> > > To: dev@kafka.apache.org
> > > > >> > >
> > > > >> > > +1 (non-binding)
> > > > >> > >
> > > > >> > > On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
> > > > >> > > rajinisiva...@googlemail.com> wrote:
> > > > >> > >
> > > > >> > >> I would like to start the voting process for *KIP-43: Kafka
> > SASL
> > > > >> > >> enhancements*. This KIP extends the SASL implementation in
> > Kafka to
> > > > >> > support
> > > > >> > >> new SASL mechanisms to enable Kafka to be integrated with
> > different
> > > > >> > >> authentication servers.
> > > > >> > >>
> > > > >> > >> The KIP is available here for reference:
> > > > >> > >>
> > > > >> > >>
> > > > >> >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
> > > > >> > >>
> > > > >> > >> And here's is a link to the discussion on the mailing list:
> > > > >> > >>
> > > > >> > >>
> > > > >> >
> > > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> Thank you...
> > > > >> > >>
> > > > >> > >> Regards,
> > > > >> > >>
> > > > >> > >> Rajini
> > > > >> > >>
> > > > >> >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> >
>



-- 
Regards,

Rajini

RE: [VOTE] KIP-43: Kafka SASL enhancements

2016-03-24 Thread Andrew Schofield

+1 (non-binding) on the revised KIP


> Date: Thu, 24 Mar 2016 08:21:14 +
> Subject: Re: [VOTE] KIP-43: Kafka SASL enhancements
> From: rajinisiva...@googlemail.com
> To: dev@kafka.apache.org
>
> Gwen,
>
> Is it still possible to include this in 0.10.0.0?
>
> Thanks,
>
> Rajini
>
> On Wed, Mar 23, 2016 at 11:08 PM, Gwen Shapira  wrote:
>
>> Sorry! Got distracted by the impending release!
>>
>> +1 on the current revision of the KIP.
>>
>> On Wed, Mar 23, 2016 at 3:33 PM, Harsha  wrote:
>>
>>> Any update on this. Gwen since the KIP is adjusted to address the
>>> pluggable classes we should make a move on this.
>>>
>>> Rajini,
>>> Can you restart the voting thread.
>>>
>>> Thanks,
>>> Harsha
>>>
>>> On Wed, Mar 16, 2016, at 06:42 AM, Rajini Sivaram wrote:
 As discussed in the KIP meeting yesterday, the scope of KIP-43 has been
 reduced so that it can be integrated into 0.10.0.0. The updated KIP is
 here:

>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43%3A+Kafka+SASL+enhancements
 .

 Can we continue the vote on the updated KIP?

 Thank you,

 Rajini

 On Thu, Mar 10, 2016 at 2:09 AM, Gwen Shapira 
>> wrote:

> Harsha,
>
> Since you are clearly in favor of the KIP, do you mind jumping into
> the discussion thread and help me understand the decision behind the
> configuration parameters only allowing a single Login and
> CallbackHandler class? This seems too limiting to me, and while
>> Rajini
> is trying hard to convince me otherwise, I remain doubtful. Perhaps
> (since we have similar experience with Hadoop), you can help me see
> what I am missing.
>
> Gwen
>
> On Wed, Mar 9, 2016 at 12:02 PM, Harsha  wrote:
>> +1 (binding)
>>
>> On Tue, Mar 8, 2016, at 02:37 AM, tao xiao wrote:
>>> +1 (non-binding)
>>>
>>> On Tue, 8 Mar 2016 at 05:33 Andrew Schofield <
>>> andrew_schofield_j...@outlook.com> wrote:
>>>
 +1 (non-binding)

 
> From: ism...@juma.me.uk
> Date: Mon, 7 Mar 2016 19:52:11 +
> Subject: Re: [VOTE] KIP-43: Kafka SASL enhancements
> To: dev@kafka.apache.org
>
> +1 (non-binding)
>
> On Thu, Mar 3, 2016 at 10:37 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
>> I would like to start the voting process for *KIP-43: Kafka
>>> SASL
>> enhancements*. This KIP extends the SASL implementation in
>>> Kafka to
 support
>> new SASL mechanisms to enable Kafka to be integrated with
>>> different
>> authentication servers.
>>
>> The KIP is available here for reference:
>>
>>

>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
>>
>> And here's is a link to the discussion on the mailing list:
>>
>>

>
>>>
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
>>
>>
>> Thank you...
>>
>> Regards,
>>
>> Rajini
>>

>



 --
 Regards,

 Rajini
>>>
>>
>
>
>
> --
> Regards,
>
> Rajini

[jira] [Commented] (KAFKA-3445) ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit

2016-03-24 Thread Ewen Cheslack-Postava (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209863#comment-15209863
 ] 

Ewen Cheslack-Postava commented on KAFKA-3445:
--

[~Ryan P] Not nit picky at all, this is actually a pretty significant 
oversight! Patch looks trivial, but could you contribute via PR according to 
the contributor guidelines: 
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes ? 
This will make it easy for me to commit the patch.

> ConnectorConfig should validate TASKS_MAX_CONFIG's lower bound limit 
> -
>
> Key: KAFKA-3445
> URL: https://issues.apache.org/jira/browse/KAFKA-3445
> Project: Kafka
>  Issue Type: Improvement
>  Components: config
>Reporter: Ryan P
>Priority: Trivial
>  Labels: newbie
> Attachments: KAFKA-3445.patch
>
>
> I'll be the first to admit this is a bit nit picky any property marked with 
> Importance.HIGH should be guarded against nonsensical values. 
> With that said I would like to suggest that TASKS_MAX_CONFIG be validating 
> against a lower bound limit of 1. 
> I do understand this is unlikely to happen and the configuration is 
> nonsensical but there is no penalty for stopping someone from trying it out. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3461) Fix typos in Kafka web documentations

2016-03-24 Thread Dongjoon Hyun (JIRA)

Dongjoon Hyun created KAFKA-3461:


 Summary: Fix typos in Kafka web documentations
 Key: KAFKA-3461
 URL: https://issues.apache.org/jira/browse/KAFKA-3461
 Project: Kafka
  Issue Type: Bug
  Components: website
Reporter: Dongjoon Hyun
Priority: Trivial


This issue fixes the following typos.
* docs/api.html: compatability => compatibility
* docs/connect.html: simultaneoulsy => simultaneously
* docs/implementation.html: LATIEST_TIME => LATEST_TIME, nPartions => 
nPartitions
* docs/migration.html: Decomission => Decommission
* docs/ops.html: stoping => stopping, ConumserGroupCommand => 
ConsumerGroupCommand, youre => you're




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3461) Fix typos in Kafka web documentations

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211209#comment-15211209
 ] 

ASF GitHub Bot commented on KAFKA-3461:
---

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/kafka/pull/1138

KAFKA-3461: Fix typos in Kafka web documentations.

This PR fixes 8 typos in HTML files of `docs` module. I wrote explicitly 
here since Github sometimes does not highlight the corrections on long lines 
correctly.
- docs/api.html: compatability => compatibility
- docs/connect.html: simultaneoulsy => simultaneously
- docs/implementation.html: LATIEST_TIME => LATEST_TIME, nPartions => 
nPartitions
- docs/migration.html: Decomission => Decommission
- docs/ops.html: stoping => stopping, ConumserGroupCommand => 
ConsumerGroupCommand, youre => you're


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/kafka KAFKA-3461

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1138.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1138


commit 6f29291cb3b91d2098e3a65715fd42f826c4bcec
Author: Dongjoon Hyun 
Date:   2016-03-25T00:25:17Z

KAFKA-3461: Fix typos in Kafka web documentations.

This PR fixes 8 typos in HTML files of `docs` module.




> Fix typos in Kafka web documentations
> -
>
> Key: KAFKA-3461
> URL: https://issues.apache.org/jira/browse/KAFKA-3461
> Project: Kafka
>  Issue Type: Bug
>  Components: website
>Reporter: Dongjoon Hyun
>Priority: Trivial
>
> This issue fixes the following typos.
> * docs/api.html: compatability => compatibility
> * docs/connect.html: simultaneoulsy => simultaneously
> * docs/implementation.html: LATIEST_TIME => LATEST_TIME, nPartions => 
> nPartitions
> * docs/migration.html: Decomission => Decommission
> * docs/ops.html: stoping => stopping, ConumserGroupCommand => 
> ConsumerGroupCommand, youre => you're



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211215#comment-15211215
 ] 

ASF GitHub Bot commented on KAFKA-3462:
---

GitHub user Ishiihara opened a pull request:

https://github.com/apache/kafka/pull/1139

KAFKA-3462: Allow SinkTasks to disable consumer offset commit



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ishiihara/kafka disable-offset-commit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1139


commit 546877ddbb0acfa079b9322becbb2dbff21c56d7
Author: Liquan Pei 
Date:   2016-03-25T00:29:24Z

Allow SinkTasks to disable consumer offset commit




> Allow SinkTasks to disable consumer offset commit 
> --
>
> Key: KAFKA-3462
> URL: https://issues.apache.org/jira/browse/KAFKA-3462
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Affects Versions: 0.10.1.0
>Reporter: Liquan Pei
>Assignee: Liquan Pei
>Priority: Minor
> Fix For: 0.10.1.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
>  SinkTasks should be able to disable consumer offset commit if they manage 
> offsets in the sink data store rather than using Kafka consumer offsets.  For 
> example, an HDFS connector might record offsets in HDFS to provide exactly 
> once delivery. When the SinkTask is started or a rebalance occurs, the task
> would reload offsets from HDFS. In this case, disabling consumer offset 
> commit will save some CPU cycles and network IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3462: Allow SinkTasks to disable consume...

2016-03-24 Thread Ishiihara

GitHub user Ishiihara opened a pull request:

https://github.com/apache/kafka/pull/1139

KAFKA-3462: Allow SinkTasks to disable consumer offset commit



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ishiihara/kafka disable-offset-commit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1139


commit 546877ddbb0acfa079b9322becbb2dbff21c56d7
Author: Liquan Pei 
Date:   2016-03-25T00:29:24Z

Allow SinkTasks to disable consumer offset commit




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread Liquan Pei (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liquan Pei updated KAFKA-3462:
--
Description:  SinkTasks should be able to disable consumer offset commit if 
they manage offsets in the sink data store rather than using Kafka consumer 
offsets.  For example, an HDFS connector might record offsets in HDFS to 
provide exactly once delivery. When the SinkTask is started or a rebalance 
occurs, the task would reload offsets from HDFS. In this case, disabling 
consumer offset commit will save some CPU cycles and network IOs.  (was:  
SinkTasks should be able to disable consumer offset commit if they manage 
offsets in the sink data store rather than using Kafka consumer offsets.  For 
example, an HDFS connector might record offsets in HDFS to provide exactly once 
delivery. When the SinkTask is started or a rebalance occurs, the task
would reload offsets from HDFS. In this case, disabling consumer offset commit 
will save some CPU cycles and network IOs.)

> Allow SinkTasks to disable consumer offset commit 
> --
>
> Key: KAFKA-3462
> URL: https://issues.apache.org/jira/browse/KAFKA-3462
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Affects Versions: 0.10.1.0
>Reporter: Liquan Pei
>Assignee: Liquan Pei
>Priority: Minor
> Fix For: 0.10.1.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
>  SinkTasks should be able to disable consumer offset commit if they manage 
> offsets in the sink data store rather than using Kafka consumer offsets.  For 
> example, an HDFS connector might record offsets in HDFS to provide exactly 
> once delivery. When the SinkTask is started or a rebalance occurs, the task 
> would reload offsets from HDFS. In this case, disabling consumer offset 
> commit will save some CPU cycles and network IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3463) Change default consumer receive buffer size to 64K

2016-03-24 Thread Jason Gustafson (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3463:
---
Fix Version/s: 0.10.0.0

> Change default consumer receive buffer size to 64K
> --
>
> Key: KAFKA-3463
> URL: https://issues.apache.org/jira/browse/KAFKA-3463
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.0.0
>
>
> In KAFKA-3135, users have reported a strange pause when consuming data using 
> the new consumer. This can be easily reproduced with the console consumer, 
> but the root cause has so far proven elusive. Interestingly, the pause can 
> also be reproduced with the old consumer if you change the socket buffer size 
> to 32K to match the new consumer's default. Similarly, by increasing the new 
> consumer's connection receive buffer size to 64K to match the old consumer's 
> default, the problem seems to be mitigated (I am unable to reproduce it 
> locally with this setting, though others have reported that its impact is 
> merely reduced). Since there doesn't appear to be a good reason to have 
> lowered the default for the new consumer, we may as well revert to the old 
> consumer's default of 64K and accept the mitigation until an actual fix is 
> found. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3463) Change default consumer receive buffer size to 64K

2016-03-24 Thread Jason Gustafson (JIRA)

Jason Gustafson created KAFKA-3463:
--

 Summary: Change default consumer receive buffer size to 64K
 Key: KAFKA-3463
 URL: https://issues.apache.org/jira/browse/KAFKA-3463
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Jason Gustafson
Assignee: Jason Gustafson


In KAFKA-3135, users have reported a strange pause when consuming data using 
the new consumer. This can be easily reproduced with the console consumer, but 
the root cause has so far proven elusive. Interestingly, the pause can also be 
reproduced with the old consumer if you change the socket buffer size to 32K to 
match the new consumer's default. Similarly, by increasing the new consumer's 
connection receive buffer size to 64K to match the old consumer's default, the 
problem seems to be mitigated (I am unable to reproduce it locally with this 
setting, though others have reported that its impact is merely reduced). Since 
there doesn't appear to be a good reason to have lowered the default for the 
new consumer, we may as well revert to the old consumer's default of 64K and 
accept the mitigation until an actual fix is found. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3461: Fix typos in Kafka web documentati...

2016-03-24 Thread dongjoon-hyun

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/kafka/pull/1138

KAFKA-3461: Fix typos in Kafka web documentations.

This PR fixes 8 typos in HTML files of `docs` module. I wrote explicitly 
here since Github sometimes does not highlight the corrections on long lines 
correctly.
- docs/api.html: compatability => compatibility
- docs/connect.html: simultaneoulsy => simultaneously
- docs/implementation.html: LATIEST_TIME => LATEST_TIME, nPartions => 
nPartitions
- docs/migration.html: Decomission => Decommission
- docs/ops.html: stoping => stopping, ConumserGroupCommand => 
ConsumerGroupCommand, youre => you're


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/kafka KAFKA-3461

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1138.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1138


commit 6f29291cb3b91d2098e3a65715fd42f826c4bcec
Author: Dongjoon Hyun 
Date:   2016-03-25T00:25:17Z

KAFKA-3461: Fix typos in Kafka web documentations.

This PR fixes 8 typos in HTML files of `docs` module.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Work started] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread Liquan Pei (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3462 started by Liquan Pei.
-
> Allow SinkTasks to disable consumer offset commit 
> --
>
> Key: KAFKA-3462
> URL: https://issues.apache.org/jira/browse/KAFKA-3462
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Affects Versions: 0.10.1.0
>Reporter: Liquan Pei
>Assignee: Liquan Pei
>Priority: Minor
> Fix For: 0.10.1.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
>  SinkTasks should be able to disable consumer offset commit if they manage 
> offsets in the sink data store rather than using Kafka consumer offsets.  For 
> example, an HDFS connector might record offsets in HDFS to provide exactly 
> once delivery. When the SinkTask is started or a rebalance occurs, the task
> would reload offsets from HDFS. In this case, disabling consumer offset 
> commit will save some CPU cycles and network IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread Liquan Pei (JIRA)

Liquan Pei created KAFKA-3462:
-

 Summary: Allow SinkTasks to disable consumer offset commit 
 Key: KAFKA-3462
 URL: https://issues.apache.org/jira/browse/KAFKA-3462
 Project: Kafka
  Issue Type: Bug
  Components: copycat
Affects Versions: 0.10.1.0
Reporter: Liquan Pei
Assignee: Liquan Pei
Priority: Minor
 Fix For: 0.10.1.0


 SinkTasks should be able to disable consumer offset commit if they manage 
offsets in the sink data store rather than using Kafka consumer offsets.  For 
example, an HDFS connector might record offsets in HDFS to provide exactly once 
delivery. When the SinkTask is started or a rebalance occurs, the task
would reload offsets from HDFS. In this case, disabling consumer offset commit 
will save some CPU cycles and network IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread Liquan Pei (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liquan Pei updated KAFKA-3462:
--
Status: Patch Available  (was: In Progress)

> Allow SinkTasks to disable consumer offset commit 
> --
>
> Key: KAFKA-3462
> URL: https://issues.apache.org/jira/browse/KAFKA-3462
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Affects Versions: 0.10.1.0
>Reporter: Liquan Pei
>Assignee: Liquan Pei
>Priority: Minor
> Fix For: 0.10.1.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
>  SinkTasks should be able to disable consumer offset commit if they manage 
> offsets in the sink data store rather than using Kafka consumer offsets.  For 
> example, an HDFS connector might record offsets in HDFS to provide exactly 
> once delivery. When the SinkTask is started or a rebalance occurs, the task 
> would reload offsets from HDFS. In this case, disabling consumer offset 
> commit will save some CPU cycles and network IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3456) In-house KafkaMetric misreports metrics when periodically observed

2016-03-24 Thread Jay Kreps (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211302#comment-15211302
 ] 

Jay Kreps commented on KAFKA-3456:
--

I think the tradeoff is between getting a current estimate and using the same 
window length for all estimates--it's hard to do both efficiently without an 
exponential weighting scheme which has bigger drawbacks still. We need to get a 
current estimate because quotas need to shut off abusers right away and also 
because inherently the goal of monitoring and alerting is to detect bad things 
quickly. We did originally consider the approach you are proposing and rejected 
it after thinking more about what seemed important for the use case.

I think you are arguing that if you have alternating behavior where you have 30 
seconds of 1 req/sec and 30 seconds of 999 req/sec that the "right" answer is a 
flat estimate of 500 req/sec. I would argue that 500 req/sec is actually not a 
better answer, then having periods of 1, periods of 999, and periods of 
everything in between.

I think philosophically this is not the right way to think about things. I 
don't think in practice you can truly control the rate that the stat is checked 
and more importantly there isn't a "true" underlying rate that can be known. 
Rather the rate will always depend on the window and the goal is to give a good 
estimate of the "current" rate at the point in time you are asked. There is no 
"right" answer to this question--you can use more historical data to produce 
this estimate in which case it will update slower when the underlying behavior 
changes, or you can use less historical data in which case you will see more 
natural variance due to luck. Not using the most recent data isn't good though, 
since the most recent data is going to be most predictive of the current value.



> In-house KafkaMetric misreports metrics when periodically observed
> --
>
> Key: KAFKA-3456
> URL: https://issues.apache.org/jira/browse/KAFKA-3456
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, core, producer 
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0
>Reporter: The Data Lorax
>Assignee: Neha Narkhede
>Priority: Minor
>
> The metrics captured by Kafka through the in-house {{SampledStat}} suffer 
> from misreporting metrics if observed in a periodic manner.
> Consider a {{Rate}} metric that is using the default 2 samples and 30 second 
> sample window i.e. the {{Rate}} is capturing 60 seconds worth of data.  So, 
> to report this metric to some external system we might poll it every 60 
> seconds to observe the current value. Using a shorter period would, in the 
> case of a {{Rate}}, lead to smoothing of the plotted data, and worse, in the 
> case of a {{Count}}, would lead to double counting - so 60 seconds is the 
> only period at which we can poll the metrics if we are to report accurate 
> metrics.
> To demonstrate the issue consider the following somewhat extreme case:
> The {{Rate}}  is capturing data from a system which alternates between a 999 
> per sec rate and a 1 per sec rate every 30 seconds, with the different rates 
> aligned with the sample boundaries within the {{Rate}} instance i.e. after 60 
> seconds the first sample within the {{Rate}} instance will have a rate of 999 
> per sec, and the second 1 per sec. 
> If we were to ask the metric for its value at this 60 second boundary it 
> would correctly report 500 per sec. However, if we asked it again 1 
> millisecond later it would report 1 per sec, as the first sample window has 
> been aged out. Depending on how retarded into the 60 sec period of the metric 
> our periodic poll of the metric was, we would observe a constant rate 
> somewhere in the range of 1 to 500 per second, most likely around the 250 
> mark. 
> Other metrics based off of the {{SampledStat}} type suffer from the same 
> issue e.g. the {{Count}} metric, given a constant rate of 1 per second, will 
> report a constant count somewhere between 30 and 60, rather than the correct 
> 60.
> This can be seen in the following test code:
> {code:java}
> public class MetricsTest {
> private MetricConfig metricsConfig;
> @Before
> public void setUp() throws Exception {
> metricsConfig = new MetricConfig();
> }
> private long t(final int bucket) {
> return metricsConfig.timeWindowMs() * bucket;
> }
> @Test
> public void testHowRateDropsMetrics() throws Exception {
> Rate rate = new Rate();
> metricsConfig.samples(2);
> metricsConfig.timeWindow(30, TimeUnit.SECONDS);
> // First sample window from t0 -> (t1 -1), with rate 999 per second:
> for (long time = t(0); time != t(1); time += 1000) {
> rate.record(metricsConfig, 999,

Reg: Contribution to Kafka

2016-03-24 Thread BigData dev

Hi Kafka Contributors,
I am interested in contributing to kafka open source.
Can you please provide some suggestions in understanding the code of kafka.
Can you please provide any method you have followed to understand the code.



Regards,
BigDataDev

83 matches

Mail list logo