test of producer's delay and consumer's delay

[Kafka]

hello，I have done a series of tests on kafka 0.9.0，and one of the results 
confused me.

test enviroment:
 kafka cluster: 3 brokers，8core cpu ／ 8g mem ／1g netcard
 client：4core cpu／4g mem
 topic：6 partitions，2 replica
 
 total messages：1
 singal message size：1024byte
 fetch.min.bytes:1
 fetch.wait.max.ms:100ms

all send tests are under the enviroment of using scala sync interface，

when I set ack to 0，the producer’s delay is 0.3ms，the consumer’s delay is 7.7ms
when I set ack to 1，the producer's delay is 1.6ms, the consumer’s delay is 3.7ms
when I set ack to -1，the produce's delay is 3.5ms, the consumer’s delay is 4.2ms

but why consumer’s delay is decreased when I set ack from 0 to 1，its confused 
me。

Re: Kafka Streams KTable-KTable Join Error

[Guozhang Wang]

Thanks for validating!

Guozhang

On Fri, Jun 17, 2016 at 7:39 PM, Tim Renner 
wrote:

> Hi Guozhang,
>
> Apologies for the delay - had to convert to the new groupBy. The fix got
> rid of the error.
>
> Thanks!
>
> Tim
>



-- 
-- Guozhang

Re: Kafka Streams KTable-KTable Join Error

[Tim Renner]

Hi Guozhang,

Apologies for the delay - had to convert to the new groupBy. The fix got
rid of the error.

Thanks!

Tim

Re: Kafka Streams KTable-KTable Join Error

[Guozhang Wang]

Hello Tim,

I think I found the issue, could you apply the following patch and retry
your app?

https://github.com/apache/kafka/pull/1520

Guozhang


On Thu, Jun 16, 2016 at 11:11 PM, Guozhang Wang  wrote:

> Hello Tim,
>
> By looking through the source code I suspect it is a bug in Kafka Stream's 
> KStreamWindowReduce
> implementation. I'll do further investigation tomorrow and possibly file a
> JIRA with a patch.
>
>
> Guozhang
>
>
> On Thu, Jun 16, 2016 at 4:12 PM, Tim Renner 
> wrote:
>
>> Hi all,
>>
>> I'm trying to do a KTable-KTable join to compute an average within a
>> tumbling window.
>> Here's the KStreams code (I've put a fully working example in a gist:
>> https://gist.github.com/timothyrenner/a99c86b2d6ed2c22c8703e8c7760af3a)
>> KStreamBuilder builder = new KStreamBuilder();
>>
>> KStream longs = builder.stream(
>> Serdes.String(), Serdes.Long(), "longs");
>>
>> KTable longCounts =
>> longs.countByKey(TimeWindows.of("longCounts", 1L),
>> Serdes.String());
>>
>>
>> KTable longSums =
>> longs.reduceByKey((v1, v2) -> v1 + v2,
>>  TimeWindows.of("longSums", 1L),
>>  Serdes.String(),
>>  Serdes.Long());
>>
>> KTable longAvgs =
>> longSums.join(longCounts,
>>  (sum, count) ->
>>   sum.doubleValue()/count.doubleValue());
>>
>> longAvgs.toStream((wk, v) -> wk.key())
>> .to(Serdes.String(),
>> Serdes.Double(),
>> "long-avgs");
>>
>> KafkaStreams streams = new KafkaStreams(builder, config);
>> streams.start();
>>
>> When I run this, I get the following exception:
>>
>> java.util.NoSuchElementException
>> at
>>
>> org.apache.kafka.streams.state.internals.RocksDBWindowStore$RocksDBWindowStoreIterator.next(RocksDBWindowStore.java:95)
>> at
>>
>> org.apache.kafka.streams.state.internals.RocksDBWindowStore$RocksDBWindowStoreIterator.next(RocksDBWindowStore.java:64)
>> at
>>
>> org.apache.kafka.streams.state.internals.MeteredWindowStore$MeteredWindowStoreIterator.next(MeteredWindowStore.java:136)
>> at
>>
>> org.apache.kafka.streams.state.internals.MeteredWindowStore$MeteredWindowStoreIterator.next(MeteredWindowStore.java:117)
>> at
>>
>> org.apache.kafka.streams.kstream.internals.KStreamWindowReduce$KStreamAggregateValueGetter.get(KStreamWindowReduce.java:166)
>> at
>>
>> org.apache.kafka.streams.kstream.internals.KStreamWindowReduce$KStreamAggregateValueGetter.get(KStreamWindowReduce.java:147)
>> at
>>
>> org.apache.kafka.streams.kstream.internals.KTableKTableJoin$KTableKTableJoinProcessor.process(KTableKTableJoin.java:77)
>> at
>>
>> org.apache.kafka.streams.kstream.internals.KTableKTableJoin$KTableKTableJoinProcessor.process(KTableKTableJoin.java:48)
>> at
>>
>> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:68)
>> at
>>
>> org.apache.kafka.streams.processor.internals.StreamTask.forward(StreamTask.java:338)
>> at
>>
>> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:187)
>> at
>>
>> org.apache.kafka.streams.kstream.internals.KStreamWindowAggregate$KStreamWindowAggregateProcessor.process(KStreamWindowAggregate.java:136)
>> at
>>
>> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:68)
>> at
>>
>> org.apache.kafka.streams.processor.internals.StreamTask.forward(StreamTask.java:338)
>> at
>>
>> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:187)
>> at
>>
>> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:64)
>> at
>>
>> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:174)
>> at
>>
>> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:320)
>> at
>>
>> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:218)
>>
>> Looks like the join is throwing the exception. Any ideas?
>>
>> Thanks,
>> Tim
>>
>
>
>
> --
> -- Guozhang
>



-- 
-- Guozhang

Redhat 7 systemd scripts

[Tauzell, Dave]

Does anybody have good RHEL 7 systemd scripts for the broker, zookeeper, schema 
registry, etc ... ?


This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.

Re: [DISCUSS] Java 8 as a minimum requirement

[Ismael Juma]

Hi Harsha,

Comments below.

On Fri, Jun 17, 2016 at 7:48 PM, Harsha  wrote:

> Hi Ismael,
> "Are you saying that you are aware of many Kafka users still
> using Java 7
> > who would be ready to upgrade to the next Kafka feature release (whatever
> > that version number is) before they can upgrade to Java 8?"
> I know there quite few users who are still on java 7


This is good to know.


> and regarding the
> upgrade we can't say Yes or no.  Its upto the user discretion when they
> choose to upgrade and ofcourse if there are any critical fixes that
> might go into the release.  We shouldn't be restricting their upgrade
> path just because we removed Java 7 support.
>

My point is that both paths have their pros and cons and we need to weigh
them up. If some users are slow to upgrade the Java version (Java 7 has
been EOL'd for over a year), there's a good chance that they are slow to
upgrade Kafka too. And if that is the case (and it may not be), then
holding up improvements for the ones who actually do upgrade may be the
wrong call. To be clear, I am still in listening mode and I haven't made up
my mind on the subject.

Once we released 0.9.0 there aren't any 0.8.x releases. i.e we don't
> have LTS type release where we continually ship critical fixes over
> 0.8.x minor releases. So if a user notices a critical fix the only
> option today is to upgrade to next version where that fix is shipped.
>

We haven't done a great job at this in the past, but there is no decision
that once a new major release is out, we don't do patch releases for the
previous major release. In fact, we have been collecting critical fixes in
the 0.9.0 branch for a potential 0.9.0.2.

I understand there is no decision made yet but given the premise was to
> ship this in 0.10.x  , possibly 0.10.1 which I don't agree with. In
> general against shipping this in 0.10.x version. Removing Java 7 support
> when the release is minor in general not a good idea to users.
>

Sorry if I didn't communicate this properly. I simply meant the next
feature release. I used 0.10.1.0 as an example, but it could also be
0.11.0.0 if that turns out to be the next release. A discussion on that
will probably take place once the scope is clear. Personally, I think the
timing is more important the the version number, but it seems like some
people disagree.

Ismael

Consumer Question

[Chris Barlock]

If you have a consumer listening on a topic and that topic is deleted is 
the consumer made aware -- perhaps by some exception -- or does it 
continue listening, blissfully unaware that it will never hear anything 
more?

Thanks, 

Chris

Re: [DISCUSS] Java 8 as a minimum requirement

[Harsha]

Hi Ismael,
"Are you saying that you are aware of many Kafka users still
using Java 7
> who would be ready to upgrade to the next Kafka feature release (whatever
> that version number is) before they can upgrade to Java 8?"
I know there quite few users who are still on java 7 and regarding the
upgrade we can't say Yes or no.  Its upto the user discretion when they
choose to upgrade and ofcourse if there are any critical fixes that
might go into the release.  We shouldn't be restricting their upgrade
path just because we removed Java 7 support.

"The 0.10.1 versus 0.11.0.0 is something that can be discussed
separately
> as
> no decision has been made on what the next version will be (we did go
> straight from 0.9.0 to 0.10.0 whereas the 0.8.x series had multiple minor
> releases)"
Once we released 0.9.0 there aren't any 0.8.x releases. i.e we don't
have LTS type release where we continually ship critical fixes over
0.8.x minor releases. So if a user notices a critical fix the only
option today is to upgrade to next version where that fix is shipped.

"no decision has been made on what the next version will be (we did go
> straight from 0.9.0 to 0.10.0 whereas the 0.8.x series had multiple minor
> releases). 

I understand there is no decision made yet but given the premise was to
ship this in 0.10.x  , possibly 0.10.1 which I don't agree with. In
general against shipping this in 0.10.x version. Removing Java 7 support
when the release is minor in general not a good idea to users. 
-Harsha

"Also note that Kafka bug fixes go to 0.10.0.1, not 0.10.1 and
> 0.10.0.x would still be available for users using Java 7."


On Fri, Jun 17, 2016, at 12:18 AM, Ismael Juma wrote:
> Hi Harsha,
> 
> Are you saying that you are aware of many Kafka users still using Java 7
> who would be ready to upgrade to the next Kafka feature release (whatever
> that version number is) before they can upgrade to Java 8?
> 
> The 0.10.1 versus 0.11.0.0 is something that can be discussed separately
> as
> no decision has been made on what the next version will be (we did go
> straight from 0.9.0 to 0.10.0 whereas the 0.8.x series had multiple minor
> releases). Also note that Kafka bug fixes go to 0.10.0.1, not 0.10.1 and
> 0.10.0.x would still be available for users using Java 7.
> 
> Ismael
> 
> On Fri, Jun 17, 2016 at 12:48 AM, Harsha  wrote:
> 
> > -1 on removing suport 0.10.1.0 . This is minor release and removing
> > support JDK 1.7 which lot of users still depend on not a good idea and
> > definitely they are not getting enough heads up to migrate their other
> > services to JDK1.7.
> > We can consider this for 0.11.0 release time line again depends on the
> > dates .
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Jun 16, 2016, at 03:08 PM, Ismael Juma wrote:
> > > Hi Jan,
> > >
> > > That's interesting. Do you have some references you can share on this? It
> > > would be good to know which Java 8 versions have been tested and whether
> > > it
> > > is something that is being worked on.
> > >
> > > Ismael
> > >
> > > On Fri, Jun 17, 2016 at 12:02 AM,  wrote:
> > >
> > > >
> > > > Hi Ismael,
> > > >
> > > > Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a
> > lot of
> > > > JVM crashes running our 0.9 brokers on Java 8... Java 7 on the other
> > hand
> > > > is totally stable.
> > > >
> > > > Until these issues have been addressed, this would cause some serious
> > > > issues for us.
> > > >
> > > > Regards
> > > >
> > > > Jan
> >

Re: Message loss with kafka 0.8.2.2

[Tom Crayford]

Did you check if the controller is active in the cluster? If the controller
isn't active (there are known 0.8 bugs that can lead to this), then this
could cause this kind of data loss issue. I recommend upgrading to 0.9 ASAP.

Thanks

Tom Crayford
Heroku Kafka

On Friday, 17 June 2016, Gulia, Vikram  wrote:

> Hi Gerard, thanks for the reply. Few follow ups -
>
> 1. I can try setting acks = all but wouldn't it lead to performance hit (I
> am using sync produce thus response time will be more).
> 2. I will try unclean.leader.election.enable = false and update you if it
> helps.
> 3. Regarding your last point, I am confused. What I understood about kafka
> is that the producer client always retrieve the topic metadata and already
> knows who the leader for the topic is. And the producer client always
> sends the message to the leader only (the replicas replicate the message
> and send acknowledgements to the leader). Are you saying the producer
> client can send message to any broker who is not leader or to two or more
> brokers one of which may or may not be leader?
>
> Thank you,
> Vikram Gulia
>
>
>
>
> On 6/17/16, 12:29 AM, "Gerard Klijs"  > wrote:
>
> >You could try set the acks to -1, so you wait for the produce to be
> >succesfull, until most other brokers also received the message. Another
> >thing you could try is set the unclean.leader.election.enable to false
> >(this is a setting on the broker).
> >I think what's happening now is that the message in your example is send
> >to
> >two different brokers, because one of them is not sending the record to
> >the
> >actual leader. Since you have set your acks to one, you wont see any error
> >in the producer, cause it succeeded in sending it to the broker. You most
> >likely will see some error on the broker, because it is not the leader.
> >
> >On Fri, Jun 17, 2016 at 5:19 AM Gulia, Vikram  >
> >wrote:
> >
> >> Hi Users, I am facing message loss while using kafka v 0.8.2.2. Please
> >>see
> >> details below and help me if you can.
> >>
> >> Issue: 2 messages produced to same partition one by one ­ Kafka producer
> >> returns same offset back which means message produced earlier is lost.<
> >>
> >>
> http://stackoverflow.com/questions/37732088/2-messages-produced-to-same-p
> >>artition-one-by-one-message-1-overridden-by-next
> >> >
> >>
> >> Details:
> >> I have a unique problem which is happening like 50-100 times a day with
> >> message volume of more than 2 millions per day on the topic.I am using
> >> Kafka producer API 0.8.2.2 and I have 12 brokers (v 0.8.2.2) running in
> >> prod with replication of 4. I have a topic with 60 partitions and I am
> >> calculating partition for all my messages and providing the value in the
> >> ProducerRecord itself. Now, the issue -
> >>
> >> Application creates 'ProducerRecord' using -
> >>
> >> new ProducerRecord(topic, 30, null, message1);
> >> providing topic, value message1 and partition 30. Then application call
> >> the send method and future is returned -
> >>
> >> // null is for callback
> >> Future future = producer.send(producerRecord. null);
> >> Now, app prints the offset and partition value by calling get on Future
> >> and then getting values from RecordMetadata - this is what i get -
> >>
> >> Kafka Response : partition 30, offset 3416092
> >> Now, the app produce the next message - message2 to same partition -
> >>
> >> new ProducerRecord(topic, 30, null, message2);
> >> and kafka response -
> >>
> >> Kafka Response : partition 30, offset 3416092
> >> I received the same offset again, and if I pull message from the offset
> >>of
> >> partition 30 using simple consumer, it ends up being the message2 which
> >> essentially mean i lost the message1.
> >>
> >> Currently, the messages are produced using 10 threads each having its
> >>own
> >> instance of kafka producer (Earlier threads shared 1 Kafka producer but
> >>it
> >> was performing slow and we still had message loss).
> >> I am using all default properties for producer except a few mentioned
> >> below, the message (String payload) size can be a few kbs to a 500 kbs.
> >>I
> >> am using acks value of 1.
> >>
> >> value.serializer: org.apache.kafka.common.serialization.StringSerializer
> >> key.serializer: org.apache.kafka.common.serialization.StringSerializer
> >> bootstrap.servers: {SERVER VIP ENDPOINT}
> >> acks: 1
> >> batch.size: 204800
> >> linger.ms: 10
> >> send.buffer.bytes: 1048576
> >> max.request.size: 1000
> >>
> >> What am i doing wrong here? Is there something I can look into or any
> >> producer property or server property I can tweak to make sure i don't
> >>lose
> >> any messages. I need some help here soon as I am losing some critical
> >> messages in production which is not good at all because as there is no
> >> exception given by Kafka Producer its even hard to find out the message
> >> 

Re: [DISCUSS] Java 8 as a minimum requirement

[Tom Crayford]

+1

We can't promise security features whilst using a deprecated version of the
JDK and relying on it for underlying security functionality (e.g. SSL).
This impacts both clients and brokers. Java 7 has been deprecated for over
a year, and software that isn't up to date with that is at fault with
respect to comparability.

On Friday, 17 June 2016, Ofir Manor  wrote:

>  I also think that Kafka should drop java 7 and scala 2.10 already.
> However, I expect Kafka (or any other project) to do it in two steps:
> 1. announce NOW that both of them are deprecated in the 0.10 series and
> will be dropped in the next major release.
> 2. drop them a the next major release.
> (ideally, the announcement should have been part of the 0.10.0 release
> notes, but any early warning is better than none)
>
> Regarding clients - if clients are 0.10 and broker is 0.11, which new 0.11
> functionality is lost? Does THAT worth a deadline extension for clients
> only? Can support for some missing functionality be backported to 0.10.x
> clients if deemed critical? (don't have an answer, but this is the right
> question IMHO)
>
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io 
>
> On Thu, Jun 16, 2016 at 11:45 PM, Ismael Juma  > wrote:
>
> > Hi all,
> >
> > I would like to start a discussion on making Java 8 a minimum requirement
> > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> > is the first discussion on the topic so the idea is to understand how
> > people feel about it. If people feel it's too soon, then we can pick up
> the
> > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > positive, I will start a vote thread.
> >
> > Let's start with some dates. Java 7 hasn't received public updates since
> > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> scheduled
> > to be released in March 2017[3].
> >
> > The first argument for dropping support for Java 7 is that the last
> public
> > release by Oracle contains a large number of known security
> > vulnerabilities. The effectiveness of Kafka's security features is
> reduced
> > if the underlying runtime is not itself secure.
> >
> > The second argument for moving to Java 8 is that it adds a number of
> > compelling features:
> >
> > * Lambda expressions and method references (particularly useful for the
> > Kafka Streams DSL)
> > * Default methods (very useful for maintaining compatibility when adding
> > methods to interfaces)
> > * java.util.stream (helpful for making collection transformations more
> > concise)
> > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> > * Other nice things: SplittableRandom, Optional (and many others I have
> not
> > mentioned)
> >
> > The third argument is that it will simplify our testing matrix, we won't
> > have to test with Java 7 any longer (this is particularly useful for
> system
> > tests that take hours to run). It will also make it easier to support
> Scala
> > 2.12, which requires Java 8.
> >
> > The fourth argument is that many other open-source projects have taken
> the
> > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> > support Java 8 in the next version (although it will take a while before
> > most phones will use that version sadly). This reduces (but does not
> > eliminate) the chance that we would be the first project that would
> cause a
> > user to consider a Java upgrade.
> >
> > The main argument for not making the change is that a reasonable number
> of
> > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > More specifically, we care about the subset who would be able to upgrade
> to
> > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> would
> > be great if we could quantify this in some way.
> >
> > What do you think?
> >
> > Ismael
> >
> > [1] https://java.com/en/download/faq/java_7.xml
> > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > [3] http://openjdk.java.net/projects/jdk9/
> > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > [8] https://webtide.com/jetty-9-3-features/
> > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > [10]
> >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > [11] http://markmail.org/message/l7s276y3xkga2eqf
> >
>

Messages delayed in jUnit test (version 0.9.0.0)

[Rodrigo Ottero]

Hi.

I am trying to use an embedded Kafka server to allow me to create tests in
jUnit using a real Kafka implementation, instead of a stub or a mock.

I am using Kafka version 0.9.0.0.

The test works, but the consumer poll has to wait for 3 seconds to get the
message.

Here is the code I am running:

--
// given a Kafka producer, a kafka consumer, a topic and a message
final Producer producer = createKafkaProducer();
final KafkaConsumer consumer =
createKafkaConsumer(topicName);

// when the producer publishes the message to the topic
producer.send(new ProducerRecord(topicName,
message));

// then the consumer can read it from the topic
final ConsumerRecords records = consumer.poll(3000);
assertTrue("The topic should have only one message. The number of
messages found is: " + records.count(),
​​
records.count() == 1);
consumer.close();
--

For this to work, consumer.poll(...) must be 3 seconds to get the message,
otherwise the consumer will get an empty ConsumerRecords (
​
records.count() == 0).

I tried sleeping the execution for 10 seconds before reaching the consumer,
to see if it was some delay caused by the server startup, but it did not
change the poll's time.

However, if I use the kafka-console-consumer.bat present in Kafka
installation and point it to my embedded server, it reads it almost
immediately.

Here is the configuration used in the test's consumer:

--
final Properties props = new Properties();
props.put("bootstrap.servers", kafkaServerAndPort);
props.put("group.id", "anyGroupId");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "3");
props.put("auto.offset.reset", "earliest");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
final KafkaConsumer consumer = new
KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList(topicName));
--

Would anyone have some idea why it is taking 3 seconds for the poll to grab
the message?

Thanks in advance!

PS: here is part of the code used to startup the Kafka and Zookeeper
servers:

---
public EmbeddedKafkaAndZookeeperServers() throws Exception {
removeExistingLogFiles();
startZookeeperServer();
startKafkaServer();
}

private void startKafkaServer() throws IOException {
final KafkaConfig config =
getKafkaConfig(zookeeperServer.getConnectString());
kafkaServer = new KafkaServerStartable(config);
kafkaServer.startup();
}

private void startZookeeperServer() throws Exception {
final int shouldUseARandomPort = -1;
final File temporaryDirectory = new File(ZOOKEEPER_LOGS_DIRECTORY);
final boolean shouldStartImmediately = true;
zookeeperServer = new TestingServer(shouldUseARandomPort,
temporaryDirectory, shouldStartImmediately);
}

private static KafkaConfig getKafkaConfig(final String
zookeeperConnectString) throws IOException {
final Properties props = new Properties();
props.put("broker.id", BROKER_ID);
props.put("port", BROKER_PORT);
props.put("log.dir", KAFKA_LOGS_DIRECTORY);
props.put("zookeeper.connect", zookeeperConnectString);
props.put("host.name", "127.0.0.1");
props.put("auto.create.topics.enable", "true");
return new KafkaConfig(props);
}

---

Re: Groupby Operator

[Davood Rafiei]

Thank you for your thorough explanation Michael. It helped a lot.

Cheers
Davood

On Thu, Jun 16, 2016 at 5:01 PM, Michael Noll  wrote:

> Davood,
>
> you are reading the input topic into a KTable, which means that subsequent
> records for the same key (such as the key `1`, which appears twice in the
> input messages/records) will be considered as updates to any previous
> records for that key.  So I think what you actually want to do is read the
> input as a KStream instead of a KTable?
>
> The following code works for me, it looks like what you're trying to do.
> Note that I am reading the input data into a KStream, not a KTable.
>
> Input:
>   new KeyValue<>(1, "message1"),
>   new KeyValue<>(1, "message1"),
>   new KeyValue<>(2, "message2"),
>   new KeyValue<>(3, "message3"),
>   new KeyValue<>(4, "message4")
>
> Streams topology:
>
>   KStream input = builder.stream(Serdes.Integer(),
> Serdes.String(), inputTopic);
>   KTable counted = input
>   .map((key, value) -> KeyValue.pair(value, value))
>   .countByKey(Serdes.String(), "counted");
>   counted.to(Serdes.String(), Serdes.Long(), outputTopic);
>
> Output:
>   new KeyValue<>("message1", 1L),
>   new KeyValue<>("message1", 2L),
>   new KeyValue<>("message2", 1L),
>   new KeyValue<>("message3", 1L),
>   new KeyValue<>("message4", 1L)
>
> Does that help?
> Michael
>
>
>
>
> On Thu, Jun 16, 2016 at 4:20 PM, Davood Rafiei 
> wrote:
>
> > Hi,
> >
> >
> > I am trying to use groupby operator in simple example. However, I get
> > strange results.
> >
> > I have inputs  on "test" topic like: (Long, String)
> > 1Message_1
> > 1Message_1
> > 2Message_2
> > 3Message_3
> > 4Message_4
> >
> > I want to get counts of each value. So:
> > Message_1 2
> > Message_1 1
> > Message_2 1
> > Message_3 1
> > Message_4 1
> >
> > Because there is not any operator like groupby (fieldIndex), I assume
> that
> > groupby works always on keys.
> >
> > So, my program is:
> >
> >   KTable source = builder.table(longSerde, stringSerde,
> > "test");
> >   KTable counts =  source.groupBy(new
> KeyValueMapper > String, KeyValue>() {
> >
> > @Override
> > public KeyValue apply(Long key, String value) {
> > // TODO Auto-generated method stub
> >  return  KeyValue.pair(value, value);
> > }
> > },Serdes.String(), Serdes.String()).count("count");
> >   counts.print();;
> >
> > And I get this output as a result:
> >
> > Message_11
> > Message_10
> > Message_11
> > Message_10
> > Message_21
> > Message_20
> > Message_31
> > Message_30
> > Message_41
> > Message_40
> >
> > I couldn't  understand this behavior.
> >
> >
> > Cheers
> > Davood
> >
>

Concurrect Topic, Consumer leads to leader not available

[Lukas Hauser]

Hello,

I'm currently working with kafka 0.10.0.0 in Java 8 and facing the 
following issue:


When I create multiple topics with the AdminUtil one by one and 
immediately start an KafkaConsumer (subscribed to this topic) in a new 
thread, many (not all) KafkaConsumers do not receive any messages but 
this log output:


WARN  [org.apache.kafka.clients.NetworkClient]: Error while fetching 
metadata with correlation id 1 : {test-topic-1=LEADER_NOT_AVAILABLE}


If I firstly create all topics with the AdminUtil and thereafter start 
the different KafkaConsumers, everything works fine. However, I can not 
do that in my more complex application.

See http://pastebin.com/fHmJe2Yw for a more detailed code segment.


What am I doing wrong?


Thank you for your help,
Lukas

Re: Message loss with kafka 0.8.2.2

[Gulia, Vikram]

Hi Gerard, thanks for the reply. Few follow ups -

1. I can try setting acks = all but wouldn't it lead to performance hit (I
am using sync produce thus response time will be more).
2. I will try unclean.leader.election.enable = false and update you if it
helps.
3. Regarding your last point, I am confused. What I understood about kafka
is that the producer client always retrieve the topic metadata and already
knows who the leader for the topic is. And the producer client always
sends the message to the leader only (the replicas replicate the message
and send acknowledgements to the leader). Are you saying the producer
client can send message to any broker who is not leader or to two or more
brokers one of which may or may not be leader?

Thank you,
Vikram Gulia




On 6/17/16, 12:29 AM, "Gerard Klijs"  wrote:

>You could try set the acks to -1, so you wait for the produce to be
>succesfull, until most other brokers also received the message. Another
>thing you could try is set the unclean.leader.election.enable to false
>(this is a setting on the broker).
>I think what's happening now is that the message in your example is send
>to
>two different brokers, because one of them is not sending the record to
>the
>actual leader. Since you have set your acks to one, you wont see any error
>in the producer, cause it succeeded in sending it to the broker. You most
>likely will see some error on the broker, because it is not the leader.
>
>On Fri, Jun 17, 2016 at 5:19 AM Gulia, Vikram 
>wrote:
>
>> Hi Users, I am facing message loss while using kafka v 0.8.2.2. Please
>>see
>> details below and help me if you can.
>>
>> Issue: 2 messages produced to same partition one by one ­ Kafka producer
>> returns same offset back which means message produced earlier is lost.<
>> 
>>http://stackoverflow.com/questions/37732088/2-messages-produced-to-same-p
>>artition-one-by-one-message-1-overridden-by-next
>> >
>>
>> Details:
>> I have a unique problem which is happening like 50-100 times a day with
>> message volume of more than 2 millions per day on the topic.I am using
>> Kafka producer API 0.8.2.2 and I have 12 brokers (v 0.8.2.2) running in
>> prod with replication of 4. I have a topic with 60 partitions and I am
>> calculating partition for all my messages and providing the value in the
>> ProducerRecord itself. Now, the issue -
>>
>> Application creates 'ProducerRecord' using -
>>
>> new ProducerRecord(topic, 30, null, message1);
>> providing topic, value message1 and partition 30. Then application call
>> the send method and future is returned -
>>
>> // null is for callback
>> Future future = producer.send(producerRecord. null);
>> Now, app prints the offset and partition value by calling get on Future
>> and then getting values from RecordMetadata - this is what i get -
>>
>> Kafka Response : partition 30, offset 3416092
>> Now, the app produce the next message - message2 to same partition -
>>
>> new ProducerRecord(topic, 30, null, message2);
>> and kafka response -
>>
>> Kafka Response : partition 30, offset 3416092
>> I received the same offset again, and if I pull message from the offset
>>of
>> partition 30 using simple consumer, it ends up being the message2 which
>> essentially mean i lost the message1.
>>
>> Currently, the messages are produced using 10 threads each having its
>>own
>> instance of kafka producer (Earlier threads shared 1 Kafka producer but
>>it
>> was performing slow and we still had message loss).
>> I am using all default properties for producer except a few mentioned
>> below, the message (String payload) size can be a few kbs to a 500 kbs.
>>I
>> am using acks value of 1.
>>
>> value.serializer: org.apache.kafka.common.serialization.StringSerializer
>> key.serializer: org.apache.kafka.common.serialization.StringSerializer
>> bootstrap.servers: {SERVER VIP ENDPOINT}
>> acks: 1
>> batch.size: 204800
>> linger.ms: 10
>> send.buffer.bytes: 1048576
>> max.request.size: 1000
>>
>> What am i doing wrong here? Is there something I can look into or any
>> producer property or server property I can tweak to make sure i don't
>>lose
>> any messages. I need some help here soon as I am losing some critical
>> messages in production which is not good at all because as there is no
>> exception given by Kafka Producer its even hard to find out the message
>> lost unless downstream process reports it.
>>
>> Thank you,
>> Vikram Gulia
>>

RE: Intermittent runtime exception: broker already registered

[Thakrar, Jayesh]

My guess is that say your broker went down and you restarted it.
That time interval between shutdown/crash and the restart was shorter than the 
ZK node's ephemeral timeout value.

Once that time is over, your node disappears from Zookeeper, the broker is able 
to recreate the znode and hence the success.

In short, I would give about 3-30 seconds (you will need to check your 
settings) before doing a restart.

-Original Message-
From: Nomar Morado [mailto:nomar.mor...@gmail.com] 
Sent: Friday, June 17, 2016 6:15 AM
To: users@kafka.apache.org
Subject: Intermittent runtime exception: broker already registered

I am using Kafka 0.9.0.1 with ZK 3.5.0-alpha

I am seeing this error intermittently which goes away after several reboots.

Any ideas?

Sent from my iPad

Re: [DISCUSS] Java 8 as a minimum requirement

[Ismael Juma]

Hi Marcus,

Thanks for your feedback.

With regards to IBM WebSphere, the latest stable release (8.5.5) supports
Java 8 according to the documentation:

http://www-01.ibm.com/support/docview.wss?uid=swg27005002

Having said that, it is fair to discuss servers and clients separately. In
Kafka, you can't use newer clients with older brokers, but you can use
older clients with newer brokers. As such, the scenario we're talking about
is that of users who can upgrade their brokers and clients to the latest
Kafka version, but are stuck with an older version of WebSphere, right? Are
you aware of such users?

Ismael
On Fri, Jun 17, 2016 at 10:34 AM, Marcus Gründler <
marcus.gruend...@aixigo.de> wrote:

> -1
> Hi Ismael,
>
> Although I really like the Java 8 features and understand the advantages
> you
> mentioned about Java 8 migration, I would suggest to stay with Java 7 as
> a minimum requirement for a while.
>
> I think there are two aspects to consider - Kafka Server and Kafka
> clients. On
> the server part it would make sense to switch to Java 8 because you can run
> the broker independently from any enclosing runtime (no JEE server etc.)
>
> But if you change the requirement for Kafka clients, you would cut Kafka
> support for quite a lot of real world deployments that run for example on
> an IBM WebSphere JEE Server (*sigh*). Since up to today there is no
> WebSphere version that supports Java 8.
>
> And I think a split of Kafka server with Java8 and Kafka client JARs in
> Java7
> would be too complicated to maintain.
>
> So my conclusion is - stay with Java 7 for a while.
>
> Regards, Marcus
>
>
> > Am 16.06.2016 um 22:45 schrieb Ismael Juma :
> >
> > Hi all,
> >
> > I would like to start a discussion on making Java 8 a minimum requirement
> > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> > is the first discussion on the topic so the idea is to understand how
> > people feel about it. If people feel it's too soon, then we can pick up
> the
> > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > positive, I will start a vote thread.
> >
> > Let's start with some dates. Java 7 hasn't received public updates since
> > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> scheduled
> > to be released in March 2017[3].
> >
> > The first argument for dropping support for Java 7 is that the last
> public
> > release by Oracle contains a large number of known security
> > vulnerabilities. The effectiveness of Kafka's security features is
> reduced
> > if the underlying runtime is not itself secure.
> >
> > The second argument for moving to Java 8 is that it adds a number of
> > compelling features:
> >
> > * Lambda expressions and method references (particularly useful for the
> > Kafka Streams DSL)
> > * Default methods (very useful for maintaining compatibility when adding
> > methods to interfaces)
> > * java.util.stream (helpful for making collection transformations more
> > concise)
> > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> > * Other nice things: SplittableRandom, Optional (and many others I have
> not
> > mentioned)
> >
> > The third argument is that it will simplify our testing matrix, we won't
> > have to test with Java 7 any longer (this is particularly useful for
> system
> > tests that take hours to run). It will also make it easier to support
> Scala
> > 2.12, which requires Java 8.
> >
> > The fourth argument is that many other open-source projects have taken
> the
> > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> > support Java 8 in the next version (although it will take a while before
> > most phones will use that version sadly). This reduces (but does not
> > eliminate) the chance that we would be the first project that would
> cause a
> > user to consider a Java upgrade.
> >
> > The main argument for not making the change is that a reasonable number
> of
> > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > More specifically, we care about the subset who would be able to upgrade
> to
> > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> would
> > be great if we could quantify this in some way.
> >
> > What do you think?
> >
> > Ismael
> >
> > [1] https://java.com/en/download/faq/java_7.xml
> > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > [3] http://openjdk.java.net/projects/jdk9/
> > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > [8] https://webtide.com/jetty-9-3-features/
> > [9] 

Re: [DISCUSS] Java 8 as a minimum requirement

[Joe Stein]

Compatibility shouldn't be broken in a minor release. Minor versions are
for new features in a backwards-compatible manner. The Kafka bylaws do not
explicitly state this but I believe it is implied based on general practice
and so many other Apache projects explicitly calling this out, documenting
and communicating their semantic version strategy.

If JDK8 is so much desired then jump to 0.11 and only do bug fixes on the
0.10 release (which should be rigorous and not forceful to make folks
upgrade unnecessarily to get such improvements).

My 0.2824152382 cents.

Regards,

~ Joe Stein

On Fri, Jun 17, 2016 at 8:53 AM, Marina  wrote:

> +1 - wish it was already done with Kafka 0.9 version :)
>
>
>   From: Tommy Becker 
>  To: users@kafka.apache.org
>  Sent: Friday, June 17, 2016 7:55 AM
>  Subject: Re: [DISCUSS] Java 8 as a minimum requirement
>
> +1 We're on Java 8 already.
>
> On 06/16/2016 04:45 PM, Ismael Juma wrote:
>
> Hi all,
>
> I would like to start a discussion on making Java 8 a minimum requirement
> for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> is the first discussion on the topic so the idea is to understand how
> people feel about it. If people feel it's too soon, then we can pick up the
> conversation again after Kafka 0.10.1.0. If the feedback is mostly
> positive, I will start a vote thread.
>
> Let's start with some dates. Java 7 hasn't received public updates since
> April 2015[1], Java 8 was released in March 2014[2] and Java 9 is scheduled
> to be released in March 2017[3].
>
> The first argument for dropping support for Java 7 is that the last public
> release by Oracle contains a large number of known security
> vulnerabilities. The effectiveness of Kafka's security features is reduced
> if the underlying runtime is not itself secure.
>
> The second argument for moving to Java 8 is that it adds a number of
> compelling features:
>
> * Lambda expressions and method references (particularly useful for the
> Kafka Streams DSL)
> * Default methods (very useful for maintaining compatibility when adding
> methods to interfaces)
> * java.util.stream (helpful for making collection transformations more
> concise)
> * Lots of improvements to java.util.concurrent (CompletableFuture,
> DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> * Other nice things: SplittableRandom, Optional (and many others I have not
> mentioned)
>
> The third argument is that it will simplify our testing matrix, we won't
> have to test with Java 7 any longer (this is particularly useful for system
> tests that take hours to run). It will also make it easier to support Scala
> 2.12, which requires Java 8.
>
> The fourth argument is that many other open-source projects have taken the
> leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> support Java 8 in the next version (although it will take a while before
> most phones will use that version sadly). This reduces (but does not
> eliminate) the chance that we would be the first project that would cause a
> user to consider a Java upgrade.
>
> The main argument for not making the change is that a reasonable number of
> users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> More specifically, we care about the subset who would be able to upgrade to
> Kafka 0.10.1.0, but would not be able to upgrade the Java version. It would
> be great if we could quantify this in some way.
>
> What do you think?
>
> Ismael
>
> [1] https://java.com/en/download/faq/java_7.xml
> [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> [3] http://openjdk.java.net/projects/jdk9/
> [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> [7] https://issues.apache.org/jira/browse/HADOOP-11858
> [8] https://webtide.com/jetty-9-3-features/
> [9] http://markmail.org/message/l7s276y3xkga2eqf
> [10]
>
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> [11] http://markmail.org/message/l7s276y3xkga2eqf
>
>
>
> --
> Tommy Becker
> Senior Software Engineer
>
> Digitalsmiths
> A TiVo Company
>
> www.digitalsmiths.com
> tobec...@tivo.com
>
> 
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> 

Re: Kafka logs on a Docker volume

[OGrandeDiEnne]

On Fri, 17 Jun 2016, 14:41 Gerard Klijs,  wrote:

> What do you mean with a *docker volume*?


The container is started through docker-compose, using the VOLUMES keyword
to mount the folder where Kafka saves the log.
I'm running all of this on windows using the standard distribution of
docker for windows. Therefore I run docker-compose in windows but the
containers run in the docker-machine, a lightweight Linux VM.

I'm not using a data container, but I am trying to save the log files on
windows, as you said to keep them.

I am running other containers with the same configuration and I see their
files correctly stored on the windows filesystem.

So, why the other containers can access the files from windows and Kafka
can't?

I suspect the RandomAccessFile API is not working properly with the remote
(windows/docker-machine) mount.

Any other idea?

Thanks for the help Gerald!

Valerio


> On Fri, Jun 17, 2016 at 1:25 PM OGrandeDiEnne 
> wrote:
>
> > Hello people,
> >
> > I'm running one single kafka broker from within a docker container. The
> > folder where kafka writes the logs is mounted as *docker volume* on my
> > system.
> >
> > As soon as I try to create a topic I get this error
> >
> > [2016-06-15 10:22:53,602] ERROR [KafkaApi-0] Error when handling request
> >
> >
> {controller_id=0,controller_epoch=1,partition_states=[{topic=mytopic,partition=0,controller_epoch=1,leader=0,leader_epoch=0,isr=[0],zk_version=0,replicas=[0]}],live_leaders=[{id=0,host=kafkadocker,port=9092}]}
> > (kafka.server.KafkaApis)
> > *java.io.IOException: Invalid argument*
> > at sun.nio.ch.FileChannelImpl.map0(Native Method)
> > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:926)
> > *at kafka.log.OffsetIndex.(OffsetIndex.scala:75)*
> > at kafka.log.LogSegment.(LogSegment.scala:58)
> > at kafka.log.Log.loadSegments(Log.scala:233)
> > at kafka.log.Log.(Log.scala:101)
> > at kafka.log.LogManager.createLog(LogManager.scala:363)
> > at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:96)
> > at
> >
> >
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:176)
> > at
> >
> >
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:176)
> > at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> > at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:176)
> > at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:170)
> > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
> > at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
> > at kafka.cluster.Partition.makeLeader(Partition.scala:170)
> > at
> >
> >
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:699)
> > at
> >
> >
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:698)
> > at
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> > at
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> > at
> >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> > at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:698)
> > at
> >
> >
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:644)
> > at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:144)
> > at kafka.server.KafkaApis.handle(KafkaApis.scala:80)
> > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > The error is an IOException, so it looks like the broker has trouble
> trying
> > to access the log file.
> > Looks like Kafka assumes a feature of the underlying filesystem, which is
> > not present.
> >
> > I do not get any error if I keep the kafka log-files inside the docker
> > container.
> >
> > Have you seen the issue before ?
> >
> > Thanks.
> >
> > *Valerio*
> >
>

Re: [DISCUSS] Java 8 as a minimum requirement

[Philippe Derome]

Scala 2.12 requires JDK 8 to take advantage of JVM technology. Forcing
migration to JDK 8 explains to some extent the non rush to release 2.12. I
believe Adriaan Moors has a presentation detailing the 2.12 Scala release.
https://adriaanm.github.io/reveal.js/scala-next.html#/
On 17 Jun 2016 8:35 a.m., "Jeff Klukas"  wrote:

> On Thu, Jun 16, 2016 at 5:20 PM, Ismael Juma  wrote:
>
> > On Thu, Jun 16, 2016 at 11:13 PM, Stephen Boesch 
> > wrote:
> >
> > > @Jeff Klukas What is the concern about scala 2.11 vs 2.12?   2.11 runs
> on
> > > both java7 and java8
> > >
> >
> > Scala 2.10.5 and 2.10.6 also support Java 8 for what it's worth.
> >
>
> I was under the impression that Scala 2.12 would be the first version
> compatible with Java 8 bytecode, but looks like that was a misunderstanding
> on my part.
>
> +1
>

Re: [DISCUSS] Java 8 as a minimum requirement

[Marina]

+1 - wish it was already done with Kafka 0.9 version :)


  From: Tommy Becker 
 To: users@kafka.apache.org 
 Sent: Friday, June 17, 2016 7:55 AM
 Subject: Re: [DISCUSS] Java 8 as a minimum requirement
   
+1 We're on Java 8 already.

On 06/16/2016 04:45 PM, Ismael Juma wrote:

Hi all,

I would like to start a discussion on making Java 8 a minimum requirement
for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
is the first discussion on the topic so the idea is to understand how
people feel about it. If people feel it's too soon, then we can pick up the
conversation again after Kafka 0.10.1.0. If the feedback is mostly
positive, I will start a vote thread.

Let's start with some dates. Java 7 hasn't received public updates since
April 2015[1], Java 8 was released in March 2014[2] and Java 9 is scheduled
to be released in March 2017[3].

The first argument for dropping support for Java 7 is that the last public
release by Oracle contains a large number of known security
vulnerabilities. The effectiveness of Kafka's security features is reduced
if the underlying runtime is not itself secure.

The second argument for moving to Java 8 is that it adds a number of
compelling features:

* Lambda expressions and method references (particularly useful for the
Kafka Streams DSL)
* Default methods (very useful for maintaining compatibility when adding
methods to interfaces)
* java.util.stream (helpful for making collection transformations more
concise)
* Lots of improvements to java.util.concurrent (CompletableFuture,
DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
* Other nice things: SplittableRandom, Optional (and many others I have not
mentioned)

The third argument is that it will simplify our testing matrix, we won't
have to test with Java 7 any longer (this is particularly useful for system
tests that take hours to run). It will also make it easier to support Scala
2.12, which requires Java 8.

The fourth argument is that many other open-source projects have taken the
leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
support Java 8 in the next version (although it will take a while before
most phones will use that version sadly). This reduces (but does not
eliminate) the chance that we would be the first project that would cause a
user to consider a Java upgrade.

The main argument for not making the change is that a reasonable number of
users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
More specifically, we care about the subset who would be able to upgrade to
Kafka 0.10.1.0, but would not be able to upgrade the Java version. It would
be great if we could quantify this in some way.

What do you think?

Ismael

[1] https://java.com/en/download/faq/java_7.xml
[2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
[3] http://openjdk.java.net/projects/jdk9/
[4] https://github.com/apache/cassandra/blob/trunk/README.asc
[5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
[6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
[7] https://issues.apache.org/jira/browse/HADOOP-11858
[8] https://webtide.com/jetty-9-3-features/
[9] http://markmail.org/message/l7s276y3xkga2eqf
[10]
https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
[11] http://markmail.org/message/l7s276y3xkga2eqf



--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com
tobec...@tivo.com



This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

   

Re: Kafka logs on a Docker volume

[Ben Davison]

Try mounting it to /tmp/ then you will see if it's a file permission issue.

On Fri, Jun 17, 2016 at 1:27 PM, Gerard Klijs 
wrote:

> What do you mean with a *docker volume*? It's best to use a data container,
> and use the volumes in your broker container, this way you can destroy the
> broker container without affecting the data. The data container itself
> needs to be configured depending on the host. For example when the host is
> running SELinux, the volumes need some specific rights,
>
> http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/
>
>
> On Fri, Jun 17, 2016 at 1:25 PM OGrandeDiEnne 
> wrote:
>
> > Hello people,
> >
> > I'm running one single kafka broker from within a docker container. The
> > folder where kafka writes the logs is mounted as *docker volume* on my
> > system.
> >
> > As soon as I try to create a topic I get this error
> >
> > [2016-06-15 10:22:53,602] ERROR [KafkaApi-0] Error when handling request
> >
> >
> {controller_id=0,controller_epoch=1,partition_states=[{topic=mytopic,partition=0,controller_epoch=1,leader=0,leader_epoch=0,isr=[0],zk_version=0,replicas=[0]}],live_leaders=[{id=0,host=kafkadocker,port=9092}]}
> > (kafka.server.KafkaApis)
> > *java.io.IOException: Invalid argument*
> > at sun.nio.ch.FileChannelImpl.map0(Native Method)
> > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:926)
> > *at kafka.log.OffsetIndex.(OffsetIndex.scala:75)*
> > at kafka.log.LogSegment.(LogSegment.scala:58)
> > at kafka.log.Log.loadSegments(Log.scala:233)
> > at kafka.log.Log.(Log.scala:101)
> > at kafka.log.LogManager.createLog(LogManager.scala:363)
> > at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:96)
> > at
> >
> >
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:176)
> > at
> >
> >
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:176)
> > at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> > at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:176)
> > at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:170)
> > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
> > at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
> > at kafka.cluster.Partition.makeLeader(Partition.scala:170)
> > at
> >
> >
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:699)
> > at
> >
> >
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:698)
> > at
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> > at
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> > at
> >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> > at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:698)
> > at
> >
> >
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:644)
> > at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:144)
> > at kafka.server.KafkaApis.handle(KafkaApis.scala:80)
> > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > The error is an IOException, so it looks like the broker has trouble
> trying
> > to access the log file.
> > Looks like Kafka assumes a feature of the underlying filesystem, which is
> > not present.
> >
> > I do not get any error if I keep the kafka log-files inside the docker
> > container.
> >
> > Have you seen the issue before ?
> >
> > Thanks.
> >
> > *Valerio*
> >
>

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

Re: Kafka logs on a Docker volume

[Gerard Klijs]

What do you mean with a *docker volume*? It's best to use a data container,
and use the volumes in your broker container, this way you can destroy the
broker container without affecting the data. The data container itself
needs to be configured depending on the host. For example when the host is
running SELinux, the volumes need some specific rights,
http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/


On Fri, Jun 17, 2016 at 1:25 PM OGrandeDiEnne 
wrote:

> Hello people,
>
> I'm running one single kafka broker from within a docker container. The
> folder where kafka writes the logs is mounted as *docker volume* on my
> system.
>
> As soon as I try to create a topic I get this error
>
> [2016-06-15 10:22:53,602] ERROR [KafkaApi-0] Error when handling request
>
> {controller_id=0,controller_epoch=1,partition_states=[{topic=mytopic,partition=0,controller_epoch=1,leader=0,leader_epoch=0,isr=[0],zk_version=0,replicas=[0]}],live_leaders=[{id=0,host=kafkadocker,port=9092}]}
> (kafka.server.KafkaApis)
> *java.io.IOException: Invalid argument*
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:926)
> *at kafka.log.OffsetIndex.(OffsetIndex.scala:75)*
> at kafka.log.LogSegment.(LogSegment.scala:58)
> at kafka.log.Log.loadSegments(Log.scala:233)
> at kafka.log.Log.(Log.scala:101)
> at kafka.log.LogManager.createLog(LogManager.scala:363)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:96)
> at
>
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:176)
> at
>
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:176)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:176)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:170)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
> at kafka.cluster.Partition.makeLeader(Partition.scala:170)
> at
>
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:699)
> at
>
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:698)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:698)
> at
>
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:644)
> at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:144)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:80)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
>
>
> The error is an IOException, so it looks like the broker has trouble trying
> to access the log file.
> Looks like Kafka assumes a feature of the underlying filesystem, which is
> not present.
>
> I do not get any error if I keep the kafka log-files inside the docker
> container.
>
> Have you seen the issue before ?
>
> Thanks.
>
> *Valerio*
>

Re: [DISCUSS] Java 8 as a minimum requirement

[Jeff Klukas]

On Thu, Jun 16, 2016 at 5:20 PM, Ismael Juma  wrote:

> On Thu, Jun 16, 2016 at 11:13 PM, Stephen Boesch 
> wrote:
>
> > @Jeff Klukas What is the concern about scala 2.11 vs 2.12?   2.11 runs on
> > both java7 and java8
> >
>
> Scala 2.10.5 and 2.10.6 also support Java 8 for what it's worth.
>

I was under the impression that Scala 2.12 would be the first version
compatible with Java 8 bytecode, but looks like that was a misunderstanding
on my part.

+1

Re: [DISCUSS] Java 8 as a minimum requirement

[Philippe Derome]

That would be very doubtful imho. It is much more adventurous to go latest
Kafka release than to go JDK 8.
On 17 Jun 2016 3:19 a.m., "Ismael Juma"  wrote:

> Hi Harsha,
>
> Are you saying that you are aware of many Kafka users still using Java 7
> who would be ready to upgrade to the next Kafka feature release (whatever
> that version number is) before they can upgrade to Java 8?
>
> The 0.10.1 versus 0.11.0.0 is something that can be discussed separately as
> no decision has been made on what the next version will be (we did go
> straight from 0.9.0 to 0.10.0 whereas the 0.8.x series had multiple minor
> releases). Also note that Kafka bug fixes go to 0.10.0.1, not 0.10.1 and
> 0.10.0.x would still be available for users using Java 7.
>
> Ismael
>
> On Fri, Jun 17, 2016 at 12:48 AM, Harsha  wrote:
>
> > -1 on removing suport 0.10.1.0 . This is minor release and removing
> > support JDK 1.7 which lot of users still depend on not a good idea and
> > definitely they are not getting enough heads up to migrate their other
> > services to JDK1.7.
> > We can consider this for 0.11.0 release time line again depends on the
> > dates .
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Jun 16, 2016, at 03:08 PM, Ismael Juma wrote:
> > > Hi Jan,
> > >
> > > That's interesting. Do you have some references you can share on this?
> It
> > > would be good to know which Java 8 versions have been tested and
> whether
> > > it
> > > is something that is being worked on.
> > >
> > > Ismael
> > >
> > > On Fri, Jun 17, 2016 at 12:02 AM,  wrote:
> > >
> > > >
> > > > Hi Ismael,
> > > >
> > > > Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a
> > lot of
> > > > JVM crashes running our 0.9 brokers on Java 8... Java 7 on the other
> > hand
> > > > is totally stable.
> > > >
> > > > Until these issues have been addressed, this would cause some serious
> > > > issues for us.
> > > >
> > > > Regards
> > > >
> > > > Jan
> >
>

Re: [DISCUSS] Java 8 as a minimum requirement

[Tommy Becker]

+1 We're on Java 8 already.

On 06/16/2016 04:45 PM, Ismael Juma wrote:

Hi all,

I would like to start a discussion on making Java 8 a minimum requirement
for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
is the first discussion on the topic so the idea is to understand how
people feel about it. If people feel it's too soon, then we can pick up the
conversation again after Kafka 0.10.1.0. If the feedback is mostly
positive, I will start a vote thread.

Let's start with some dates. Java 7 hasn't received public updates since
April 2015[1], Java 8 was released in March 2014[2] and Java 9 is scheduled
to be released in March 2017[3].

The first argument for dropping support for Java 7 is that the last public
release by Oracle contains a large number of known security
vulnerabilities. The effectiveness of Kafka's security features is reduced
if the underlying runtime is not itself secure.

The second argument for moving to Java 8 is that it adds a number of
compelling features:

* Lambda expressions and method references (particularly useful for the
Kafka Streams DSL)
* Default methods (very useful for maintaining compatibility when adding
methods to interfaces)
* java.util.stream (helpful for making collection transformations more
concise)
* Lots of improvements to java.util.concurrent (CompletableFuture,
DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
* Other nice things: SplittableRandom, Optional (and many others I have not
mentioned)

The third argument is that it will simplify our testing matrix, we won't
have to test with Java 7 any longer (this is particularly useful for system
tests that take hours to run). It will also make it easier to support Scala
2.12, which requires Java 8.

The fourth argument is that many other open-source projects have taken the
leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
support Java 8 in the next version (although it will take a while before
most phones will use that version sadly). This reduces (but does not
eliminate) the chance that we would be the first project that would cause a
user to consider a Java upgrade.

The main argument for not making the change is that a reasonable number of
users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
More specifically, we care about the subset who would be able to upgrade to
Kafka 0.10.1.0, but would not be able to upgrade the Java version. It would
be great if we could quantify this in some way.

What do you think?

Ismael

[1] https://java.com/en/download/faq/java_7.xml
[2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
[3] http://openjdk.java.net/projects/jdk9/
[4] https://github.com/apache/cassandra/blob/trunk/README.asc
[5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
[6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
[7] https://issues.apache.org/jira/browse/HADOOP-11858
[8] https://webtide.com/jetty-9-3-features/
[9] http://markmail.org/message/l7s276y3xkga2eqf
[10]
https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
[11] http://markmail.org/message/l7s276y3xkga2eqf



--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com
tobec...@tivo.com



This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

Re: [DISCUSS] Java 8 as a minimum requirement

[OGrandeDiEnne]

I totally agree with Marcus.

No issue in upgrading the broker's code to 8.

For the client, there can be lots of users which are required to stay 7
because of dependencies.

So, stay with 7 at least for the client.

And... if you break compatibility to 7 please do that in a major release.

*Valerio*

On 17 June 2016 at 10:34, Marcus Gründler 
wrote:

> -1
> Hi Ismael,
>
> Although I really like the Java 8 features and understand the advantages
> you
> mentioned about Java 8 migration, I would suggest to stay with Java 7 as
> a minimum requirement for a while.
>
> I think there are two aspects to consider - Kafka Server and Kafka
> clients. On
> the server part it would make sense to switch to Java 8 because you can run
> the broker independently from any enclosing runtime (no JEE server etc.)
>
> But if you change the requirement for Kafka clients, you would cut Kafka
> support for quite a lot of real world deployments that run for example on
> an IBM WebSphere JEE Server (*sigh*). Since up to today there is no
> WebSphere version that supports Java 8.
>
> And I think a split of Kafka server with Java8 and Kafka client JARs in
> Java7
> would be too complicated to maintain.
>
> So my conclusion is - stay with Java 7 for a while.
>
> Regards, Marcus
>
>
> > Am 16.06.2016 um 22:45 schrieb Ismael Juma :
> >
> > Hi all,
> >
> > I would like to start a discussion on making Java 8 a minimum requirement
> > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> > is the first discussion on the topic so the idea is to understand how
> > people feel about it. If people feel it's too soon, then we can pick up
> the
> > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > positive, I will start a vote thread.
> >
> > Let's start with some dates. Java 7 hasn't received public updates since
> > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> scheduled
> > to be released in March 2017[3].
> >
> > The first argument for dropping support for Java 7 is that the last
> public
> > release by Oracle contains a large number of known security
> > vulnerabilities. The effectiveness of Kafka's security features is
> reduced
> > if the underlying runtime is not itself secure.
> >
> > The second argument for moving to Java 8 is that it adds a number of
> > compelling features:
> >
> > * Lambda expressions and method references (particularly useful for the
> > Kafka Streams DSL)
> > * Default methods (very useful for maintaining compatibility when adding
> > methods to interfaces)
> > * java.util.stream (helpful for making collection transformations more
> > concise)
> > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> > * Other nice things: SplittableRandom, Optional (and many others I have
> not
> > mentioned)
> >
> > The third argument is that it will simplify our testing matrix, we won't
> > have to test with Java 7 any longer (this is particularly useful for
> system
> > tests that take hours to run). It will also make it easier to support
> Scala
> > 2.12, which requires Java 8.
> >
> > The fourth argument is that many other open-source projects have taken
> the
> > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> > support Java 8 in the next version (although it will take a while before
> > most phones will use that version sadly). This reduces (but does not
> > eliminate) the chance that we would be the first project that would
> cause a
> > user to consider a Java upgrade.
> >
> > The main argument for not making the change is that a reasonable number
> of
> > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > More specifically, we care about the subset who would be able to upgrade
> to
> > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> would
> > be great if we could quantify this in some way.
> >
> > What do you think?
> >
> > Ismael
> >
> > [1] https://java.com/en/download/faq/java_7.xml
> > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > [3] http://openjdk.java.net/projects/jdk9/
> > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > [8] https://webtide.com/jetty-9-3-features/
> > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > [10]
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > [11] http://markmail.org/message/l7s276y3xkga2eqf
>

Kafka logs on a Docker volume

[OGrandeDiEnne]

Hello people,

I'm running one single kafka broker from within a docker container. The
folder where kafka writes the logs is mounted as *docker volume* on my
system.

As soon as I try to create a topic I get this error

[2016-06-15 10:22:53,602] ERROR [KafkaApi-0] Error when handling request
{controller_id=0,controller_epoch=1,partition_states=[{topic=mytopic,partition=0,controller_epoch=1,leader=0,leader_epoch=0,isr=[0],zk_version=0,replicas=[0]}],live_leaders=[{id=0,host=kafkadocker,port=9092}]}
(kafka.server.KafkaApis)
*java.io.IOException: Invalid argument*
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:926)
*at kafka.log.OffsetIndex.(OffsetIndex.scala:75)*
at kafka.log.LogSegment.(LogSegment.scala:58)
at kafka.log.Log.loadSegments(Log.scala:233)
at kafka.log.Log.(Log.scala:101)
at kafka.log.LogManager.createLog(LogManager.scala:363)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:96)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:176)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:176)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:176)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:170)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
at kafka.cluster.Partition.makeLeader(Partition.scala:170)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:699)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:698)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:698)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:644)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:144)
at kafka.server.KafkaApis.handle(KafkaApis.scala:80)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)


The error is an IOException, so it looks like the broker has trouble trying
to access the log file.
Looks like Kafka assumes a feature of the underlying filesystem, which is
not present.

I do not get any error if I keep the kafka log-files inside the docker
container.

Have you seen the issue before ?

Thanks.

*Valerio*

Intermittent runtime exception: broker already registered

[Nomar Morado]

I am using Kafka 0.9.0.1 with ZK 3.5.0-alpha

I am seeing this error intermittently which goes away after several reboots.

Any ideas?

Sent from my iPad

Fail fast producer/consumer when no connection to Kafka brokers cluster

[Spico Florin]

Hello!
  I would like to know what are the configurations/properties for the
producer/consumer in order fail fast when the connection to the entire
broker cluster is lost.
For example if we can set up a parameter that when the connection trial
reached a treshold then disconnect and throw an exception.

I look forward for your answers.
Regards,
Florin

Re: [DISCUSS] Java 8 as a minimum requirement

[Gerard Klijs]

+1 we already use java 8

On Fri, Jun 17, 2016 at 11:07 AM Jaikiran Pai 
wrote:

> +1 for Java 8. Our eco-system which uses Kafka and many other open
> source projects are now fully on Java 8 since a year or more.
>
> -Jaikiran
> On Friday 17 June 2016 02:15 AM, Ismael Juma wrote:
> > Hi all,
> >
> > I would like to start a discussion on making Java 8 a minimum requirement
> > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> > is the first discussion on the topic so the idea is to understand how
> > people feel about it. If people feel it's too soon, then we can pick up
> the
> > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > positive, I will start a vote thread.
> >
> > Let's start with some dates. Java 7 hasn't received public updates since
> > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> scheduled
> > to be released in March 2017[3].
> >
> > The first argument for dropping support for Java 7 is that the last
> public
> > release by Oracle contains a large number of known security
> > vulnerabilities. The effectiveness of Kafka's security features is
> reduced
> > if the underlying runtime is not itself secure.
> >
> > The second argument for moving to Java 8 is that it adds a number of
> > compelling features:
> >
> > * Lambda expressions and method references (particularly useful for the
> > Kafka Streams DSL)
> > * Default methods (very useful for maintaining compatibility when adding
> > methods to interfaces)
> > * java.util.stream (helpful for making collection transformations more
> > concise)
> > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> > * Other nice things: SplittableRandom, Optional (and many others I have
> not
> > mentioned)
> >
> > The third argument is that it will simplify our testing matrix, we won't
> > have to test with Java 7 any longer (this is particularly useful for
> system
> > tests that take hours to run). It will also make it easier to support
> Scala
> > 2.12, which requires Java 8.
> >
> > The fourth argument is that many other open-source projects have taken
> the
> > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> > support Java 8 in the next version (although it will take a while before
> > most phones will use that version sadly). This reduces (but does not
> > eliminate) the chance that we would be the first project that would
> cause a
> > user to consider a Java upgrade.
> >
> > The main argument for not making the change is that a reasonable number
> of
> > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > More specifically, we care about the subset who would be able to upgrade
> to
> > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> would
> > be great if we could quantify this in some way.
> >
> > What do you think?
> >
> > Ismael
> >
> > [1] https://java.com/en/download/faq/java_7.xml
> > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > [3] http://openjdk.java.net/projects/jdk9/
> > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > [8] https://webtide.com/jetty-9-3-features/
> > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > [10]
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > [11] http://markmail.org/message/l7s276y3xkga2eqf
> >
>
>

Re: [DISCUSS] Java 8 as a minimum requirement

[Jaikiran Pai]

+1 for Java 8. Our eco-system which uses Kafka and many other open 
source projects are now fully on Java 8 since a year or more.


-Jaikiran
On Friday 17 June 2016 02:15 AM, Ismael Juma wrote:

Hi all,

I would like to start a discussion on making Java 8 a minimum requirement
for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
is the first discussion on the topic so the idea is to understand how
people feel about it. If people feel it's too soon, then we can pick up the
conversation again after Kafka 0.10.1.0. If the feedback is mostly
positive, I will start a vote thread.

Let's start with some dates. Java 7 hasn't received public updates since
April 2015[1], Java 8 was released in March 2014[2] and Java 9 is scheduled
to be released in March 2017[3].

The first argument for dropping support for Java 7 is that the last public
release by Oracle contains a large number of known security
vulnerabilities. The effectiveness of Kafka's security features is reduced
if the underlying runtime is not itself secure.

The second argument for moving to Java 8 is that it adds a number of
compelling features:

* Lambda expressions and method references (particularly useful for the
Kafka Streams DSL)
* Default methods (very useful for maintaining compatibility when adding
methods to interfaces)
* java.util.stream (helpful for making collection transformations more
concise)
* Lots of improvements to java.util.concurrent (CompletableFuture,
DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
* Other nice things: SplittableRandom, Optional (and many others I have not
mentioned)

The third argument is that it will simplify our testing matrix, we won't
have to test with Java 7 any longer (this is particularly useful for system
tests that take hours to run). It will also make it easier to support Scala
2.12, which requires Java 8.

The fourth argument is that many other open-source projects have taken the
leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
support Java 8 in the next version (although it will take a while before
most phones will use that version sadly). This reduces (but does not
eliminate) the chance that we would be the first project that would cause a
user to consider a Java upgrade.

The main argument for not making the change is that a reasonable number of
users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
More specifically, we care about the subset who would be able to upgrade to
Kafka 0.10.1.0, but would not be able to upgrade the Java version. It would
be great if we could quantify this in some way.

What do you think?

Ismael

[1] https://java.com/en/download/faq/java_7.xml
[2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
[3] http://openjdk.java.net/projects/jdk9/
[4] https://github.com/apache/cassandra/blob/trunk/README.asc
[5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
[6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
[7] https://issues.apache.org/jira/browse/HADOOP-11858
[8] https://webtide.com/jetty-9-3-features/
[9] http://markmail.org/message/l7s276y3xkga2eqf
[10]
https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
[11] http://markmail.org/message/l7s276y3xkga2eqf

Re: [DISCUSS] Java 8 as a minimum requirement

[Manikumar Reddy]

I agree with Harsha and Marcus. Many of the kafka users are still on java 7
and
some of them definitely upgrade to newer versions.  We may need to support
for a while.
We can remove the support from next major version onwards.

Thanks,
Manikumar

On Fri, Jun 17, 2016 at 2:04 PM, Marcus Gründler  wrote:

> -1
> Hi Ismael,
>
> Although I really like the Java 8 features and understand the advantages
> you
> mentioned about Java 8 migration, I would suggest to stay with Java 7 as
> a minimum requirement for a while.
>
> I think there are two aspects to consider - Kafka Server and Kafka
> clients. On
> the server part it would make sense to switch to Java 8 because you can run
> the broker independently from any enclosing runtime (no JEE server etc.)
>
> But if you change the requirement for Kafka clients, you would cut Kafka
> support for quite a lot of real world deployments that run for example on
> an IBM WebSphere JEE Server (*sigh*). Since up to today there is no
> WebSphere version that supports Java 8.
>
> And I think a split of Kafka server with Java8 and Kafka client JARs in
> Java7
> would be too complicated to maintain.
>
> So my conclusion is - stay with Java 7 for a while.
>
> Regards, Marcus
>
>
> > Am 16.06.2016 um 22:45 schrieb Ismael Juma :
> >
> > Hi all,
> >
> > I would like to start a discussion on making Java 8 a minimum requirement
> > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> > is the first discussion on the topic so the idea is to understand how
> > people feel about it. If people feel it's too soon, then we can pick up
> the
> > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > positive, I will start a vote thread.
> >
> > Let's start with some dates. Java 7 hasn't received public updates since
> > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> scheduled
> > to be released in March 2017[3].
> >
> > The first argument for dropping support for Java 7 is that the last
> public
> > release by Oracle contains a large number of known security
> > vulnerabilities. The effectiveness of Kafka's security features is
> reduced
> > if the underlying runtime is not itself secure.
> >
> > The second argument for moving to Java 8 is that it adds a number of
> > compelling features:
> >
> > * Lambda expressions and method references (particularly useful for the
> > Kafka Streams DSL)
> > * Default methods (very useful for maintaining compatibility when adding
> > methods to interfaces)
> > * java.util.stream (helpful for making collection transformations more
> > concise)
> > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> > * Other nice things: SplittableRandom, Optional (and many others I have
> not
> > mentioned)
> >
> > The third argument is that it will simplify our testing matrix, we won't
> > have to test with Java 7 any longer (this is particularly useful for
> system
> > tests that take hours to run). It will also make it easier to support
> Scala
> > 2.12, which requires Java 8.
> >
> > The fourth argument is that many other open-source projects have taken
> the
> > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> > support Java 8 in the next version (although it will take a while before
> > most phones will use that version sadly). This reduces (but does not
> > eliminate) the chance that we would be the first project that would
> cause a
> > user to consider a Java upgrade.
> >
> > The main argument for not making the change is that a reasonable number
> of
> > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > More specifically, we care about the subset who would be able to upgrade
> to
> > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> would
> > be great if we could quantify this in some way.
> >
> > What do you think?
> >
> > Ismael
> >
> > [1] https://java.com/en/download/faq/java_7.xml
> > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > [3] http://openjdk.java.net/projects/jdk9/
> > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > [8] https://webtide.com/jetty-9-3-features/
> > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > [10]
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > [11] http://markmail.org/message/l7s276y3xkga2eqf
>
> --
>
> aixigo AG - einfach. besser. beraten
> Karl-Friedrich-Straße 68, 52072 Aachen, Germany
> fon: +49 (0)241 559709-43, fax: +49 (0)241 559709-99
> eMail: 

Re: [DISCUSS] Java 8 as a minimum requirement

[Stevo Slavić]

+1 on Java 8 already, Scala 2.11 is enough as well

On Fri, Jun 17, 2016 at 9:48 AM, Achanta Vamsi Subhash <
achanta.va...@flipkart.com> wrote:

> +1 to Java8 - we have already moved to Java8 last year.
>
> On Fri, Jun 17, 2016 at 12:48 PM, Ismael Juma  wrote:
>
> > Hi Harsha,
> >
> > Are you saying that you are aware of many Kafka users still using Java 7
> > who would be ready to upgrade to the next Kafka feature release (whatever
> > that version number is) before they can upgrade to Java 8?
> >
> > The 0.10.1 versus 0.11.0.0 is something that can be discussed separately
> as
> > no decision has been made on what the next version will be (we did go
> > straight from 0.9.0 to 0.10.0 whereas the 0.8.x series had multiple minor
> > releases). Also note that Kafka bug fixes go to 0.10.0.1, not 0.10.1 and
> > 0.10.0.x would still be available for users using Java 7.
> >
> > Ismael
> >
> > On Fri, Jun 17, 2016 at 12:48 AM, Harsha  wrote:
> >
> > > -1 on removing suport 0.10.1.0 . This is minor release and removing
> > > support JDK 1.7 which lot of users still depend on not a good idea and
> > > definitely they are not getting enough heads up to migrate their other
> > > services to JDK1.7.
> > > We can consider this for 0.11.0 release time line again depends on the
> > > dates .
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Thu, Jun 16, 2016, at 03:08 PM, Ismael Juma wrote:
> > > > Hi Jan,
> > > >
> > > > That's interesting. Do you have some references you can share on
> this?
> > It
> > > > would be good to know which Java 8 versions have been tested and
> > whether
> > > > it
> > > > is something that is being worked on.
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jun 17, 2016 at 12:02 AM,  wrote:
> > > >
> > > > >
> > > > > Hi Ismael,
> > > > >
> > > > > Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a
> > > lot of
> > > > > JVM crashes running our 0.9 brokers on Java 8... Java 7 on the
> other
> > > hand
> > > > > is totally stable.
> > > > >
> > > > > Until these issues have been addressed, this would cause some
> serious
> > > > > issues for us.
> > > > >
> > > > > Regards
> > > > >
> > > > > Jan
> > >
> >
>
>
>
> --
> Regards
> Vamsi Subhash
>

Re: [DISCUSS] Java 8 as a minimum requirement

[Marcus Gründler]

-1
Hi Ismael,

Although I really like the Java 8 features and understand the advantages you
mentioned about Java 8 migration, I would suggest to stay with Java 7 as
a minimum requirement for a while.

I think there are two aspects to consider - Kafka Server and Kafka clients. On 
the server part it would make sense to switch to Java 8 because you can run
the broker independently from any enclosing runtime (no JEE server etc.)

But if you change the requirement for Kafka clients, you would cut Kafka
support for quite a lot of real world deployments that run for example on
an IBM WebSphere JEE Server (*sigh*). Since up to today there is no 
WebSphere version that supports Java 8.

And I think a split of Kafka server with Java8 and Kafka client JARs in Java7 
would be too complicated to maintain.

So my conclusion is - stay with Java 7 for a while.

Regards, Marcus


> Am 16.06.2016 um 22:45 schrieb Ismael Juma :
> 
> Hi all,
> 
> I would like to start a discussion on making Java 8 a minimum requirement
> for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
> is the first discussion on the topic so the idea is to understand how
> people feel about it. If people feel it's too soon, then we can pick up the
> conversation again after Kafka 0.10.1.0. If the feedback is mostly
> positive, I will start a vote thread.
> 
> Let's start with some dates. Java 7 hasn't received public updates since
> April 2015[1], Java 8 was released in March 2014[2] and Java 9 is scheduled
> to be released in March 2017[3].
> 
> The first argument for dropping support for Java 7 is that the last public
> release by Oracle contains a large number of known security
> vulnerabilities. The effectiveness of Kafka's security features is reduced
> if the underlying runtime is not itself secure.
> 
> The second argument for moving to Java 8 is that it adds a number of
> compelling features:
> 
> * Lambda expressions and method references (particularly useful for the
> Kafka Streams DSL)
> * Default methods (very useful for maintaining compatibility when adding
> methods to interfaces)
> * java.util.stream (helpful for making collection transformations more
> concise)
> * Lots of improvements to java.util.concurrent (CompletableFuture,
> DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
> * Other nice things: SplittableRandom, Optional (and many others I have not
> mentioned)
> 
> The third argument is that it will simplify our testing matrix, we won't
> have to test with Java 7 any longer (this is particularly useful for system
> tests that take hours to run). It will also make it easier to support Scala
> 2.12, which requires Java 8.
> 
> The fourth argument is that many other open-source projects have taken the
> leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
> Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
> support Java 8 in the next version (although it will take a while before
> most phones will use that version sadly). This reduces (but does not
> eliminate) the chance that we would be the first project that would cause a
> user to consider a Java upgrade.
> 
> The main argument for not making the change is that a reasonable number of
> users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> More specifically, we care about the subset who would be able to upgrade to
> Kafka 0.10.1.0, but would not be able to upgrade the Java version. It would
> be great if we could quantify this in some way.
> 
> What do you think?
> 
> Ismael
> 
> [1] https://java.com/en/download/faq/java_7.xml
> [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> [3] http://openjdk.java.net/projects/jdk9/
> [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> [5] https://lucene.apache.org/#highlights-of-this-lucene-release-include
> [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> [7] https://issues.apache.org/jira/browse/HADOOP-11858
> [8] https://webtide.com/jetty-9-3-features/
> [9] http://markmail.org/message/l7s276y3xkga2eqf
> [10]
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> [11] http://markmail.org/message/l7s276y3xkga2eqf

-- 

aixigo AG - einfach. besser. beraten
Karl-Friedrich-Straße 68, 52072 Aachen, Germany
fon: +49 (0)241 559709-43, fax: +49 (0)241 559709-99
eMail: marcus.gruend...@aixigo.de, web: http://www.aixigo.de

Amtsgericht Aachen - HRB 8057
Vorstand: Erich Borsch, Christian Friedrich, Tobias Haustein
Vors. des Aufsichtsrates: Prof. Dr. Rüdiger von Nitzsch

Re: ConsoleProducer missing messages (random behavior)

[Radu Radutiu]

Hi,

Sorry for posting twice. It seems the issue I have encountered is indeed
https://issues.apache.org/jira/browse/KAFKA-3129 as indicated by Vahid
Hashemian in the other thread. If I change request.required.acks to 1 it
works fine.
As far as the problem going away if the broker is started a long time ago I
cannot reproduce any longer. The problem occurs 50% of the time if
request.required.acks=0 on my machine. If I use the --old-producer it works
fine.

Best regards,
Radu


On 16 June 2016 at 16:15, Dustin Cote  wrote:

> Hi Radu,
>
> Could you clarify what you mean by "I've noticed that if the kafka server
> was started a long time ago the console producer publishes all lines"?
> It's a known issue that if there are no brokers available, the producer
> will drop messages (https://issues.apache.org/jira/browse/KAFKA-1955) and
> if you've scripted this scenario, it's likely your broker hasn't come up
> completely before you start producing messages to it.
>
> Regards,
>
>
> On Thu, Jun 16, 2016 at 3:19 AM, Radu Radutiu  wrote:
>
> > Hi,
> >
> > I was following the Quickstart guide and I have noticed that
> > ConsoleProducer does not publish all messages (the number of messages
> > published differs from one run to another) and happens mostly on a fresh
> > started broker.
> > version: kafka_2.11-0.10.0.0
> > OS: Linux (Ubuntu 14.04, Centos 7.2)
> > JDK: java version "1.7.0_101"
> > OpenJDK Runtime Environment (IcedTea 2.6.6)
> (7u101-2.6.6-0ubuntu0.14.04.1),
> > openjdk version "1.8.0_91"
> > OpenJDK Runtime Environment (build 1.8.0_91-b14)
> >
> >
> > How to reproduce:
> > - start zookeeper:
> > ~/work/kafka_2.11-0.10.0.0$ bin/zookeeper-server-start.sh
> > config/zookeeper.properties &
> >
> > -start kafka:
> > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-server-start.sh
> > config/server.properties &
> >
> > -start console consumer (topic test1 is already created):
> > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh
> > --bootstrap-server localhost:9092 -topic test1 --zookeeper localhost:2181
> >
> > -in another terminal start console producer with the LICENSE file in
> kafka
> > directory as input:
> > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --topic test1
> > --broker-list localhost:9092>
> > The last line in the console consumer output is not the last line in the
> > LICENSE file for the first few runs of the console producer. If I use the
> > --old-producer parameter, all the lines in the LICENSE file are published
> > (and appear in the console consumer output). Different runs of console
> > producer with the same input file publish different number of lines
> > (sometimes all, sometimes only 182 lines out of 330). I've noticed that
> if
> > the kafka server was started a long time ago the console producer
> publishes
> > all lines.
> > I have checked the kafka binary log file (in my case
> > /tmp/kafka-logs/test1-0/.log ) and confirmed that the
> > messages are not published (the console consumer receives all the
> > messages).
> >
> > Is there an explanation for this behavior?
> >
> > Best regards,
> > Radu
> >
>
>
>
> --
> Dustin Cote
> confluent.io
>

Re: [DISCUSS] Java 8 as a minimum requirement

[Achanta Vamsi Subhash]

+1 to Java8 - we have already moved to Java8 last year.

On Fri, Jun 17, 2016 at 12:48 PM, Ismael Juma  wrote:

> Hi Harsha,
>
> Are you saying that you are aware of many Kafka users still using Java 7
> who would be ready to upgrade to the next Kafka feature release (whatever
> that version number is) before they can upgrade to Java 8?
>
> The 0.10.1 versus 0.11.0.0 is something that can be discussed separately as
> no decision has been made on what the next version will be (we did go
> straight from 0.9.0 to 0.10.0 whereas the 0.8.x series had multiple minor
> releases). Also note that Kafka bug fixes go to 0.10.0.1, not 0.10.1 and
> 0.10.0.x would still be available for users using Java 7.
>
> Ismael
>
> On Fri, Jun 17, 2016 at 12:48 AM, Harsha  wrote:
>
> > -1 on removing suport 0.10.1.0 . This is minor release and removing
> > support JDK 1.7 which lot of users still depend on not a good idea and
> > definitely they are not getting enough heads up to migrate their other
> > services to JDK1.7.
> > We can consider this for 0.11.0 release time line again depends on the
> > dates .
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Jun 16, 2016, at 03:08 PM, Ismael Juma wrote:
> > > Hi Jan,
> > >
> > > That's interesting. Do you have some references you can share on this?
> It
> > > would be good to know which Java 8 versions have been tested and
> whether
> > > it
> > > is something that is being worked on.
> > >
> > > Ismael
> > >
> > > On Fri, Jun 17, 2016 at 12:02 AM,  wrote:
> > >
> > > >
> > > > Hi Ismael,
> > > >
> > > > Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a
> > lot of
> > > > JVM crashes running our 0.9 brokers on Java 8... Java 7 on the other
> > hand
> > > > is totally stable.
> > > >
> > > > Until these issues have been addressed, this would cause some serious
> > > > issues for us.
> > > >
> > > > Regards
> > > >
> > > > Jan
> >
>



-- 
Regards
Vamsi Subhash

Re: [DISCUSS] Java 8 as a minimum requirement

[Ismael Juma]

Hi Harsha,

Are you saying that you are aware of many Kafka users still using Java 7
who would be ready to upgrade to the next Kafka feature release (whatever
that version number is) before they can upgrade to Java 8?

The 0.10.1 versus 0.11.0.0 is something that can be discussed separately as
no decision has been made on what the next version will be (we did go
straight from 0.9.0 to 0.10.0 whereas the 0.8.x series had multiple minor
releases). Also note that Kafka bug fixes go to 0.10.0.1, not 0.10.1 and
0.10.0.x would still be available for users using Java 7.

Ismael

On Fri, Jun 17, 2016 at 12:48 AM, Harsha  wrote:

> -1 on removing suport 0.10.1.0 . This is minor release and removing
> support JDK 1.7 which lot of users still depend on not a good idea and
> definitely they are not getting enough heads up to migrate their other
> services to JDK1.7.
> We can consider this for 0.11.0 release time line again depends on the
> dates .
>
> Thanks,
> Harsha
>
> On Thu, Jun 16, 2016, at 03:08 PM, Ismael Juma wrote:
> > Hi Jan,
> >
> > That's interesting. Do you have some references you can share on this? It
> > would be good to know which Java 8 versions have been tested and whether
> > it
> > is something that is being worked on.
> >
> > Ismael
> >
> > On Fri, Jun 17, 2016 at 12:02 AM,  wrote:
> >
> > >
> > > Hi Ismael,
> > >
> > > Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a
> lot of
> > > JVM crashes running our 0.9 brokers on Java 8... Java 7 on the other
> hand
> > > is totally stable.
> > >
> > > Until these issues have been addressed, this would cause some serious
> > > issues for us.
> > >
> > > Regards
> > >
> > > Jan
>

Re: Fail to build examples with gradle on Kafka using JDK 8

[Ismael Juma]

Great!

Ismael

On Fri, Jun 17, 2016 at 2:26 AM, Philippe Derome  wrote:

> Thanks a lot for that. I must had done something wrong.
>
> I reinstalled gradle on OSX 10.11 using MacPorts, then recloned kafka and
> then followed the normal steps and it worked just fine.
>
> On Thu, Jun 16, 2016 at 6:49 PM, Ismael Juma  wrote:
>
> > Hi Philippe,
> >
> > Building with Java 8 works for me. Can you please try `./gradlew clean
> > jarAll` instead?
> >
> > Ismael
> >
> > On Fri, Jun 17, 2016 at 12:23 AM, Philippe Derome 
> > wrote:
> >
> > > I will.
> > >
> > > thanks
> > > On 16 Jun 2016 1:30 p.m., "Eno Thereska" 
> wrote:
> > >
> > > > Hi Phil,
> > > >
> > > > Feel free to comment on that JIRA and re-open if necessary.
> > > >
> > > > Eno
> > > >
> > > >
> > > > > On 16 Jun 2016, at 17:02, Philippe Derome 
> > wrote:
> > > > >
> > > > > The issue had apparently existed and is apparently resolved, but so
> > far
> > > > it
> > > > > does not work for me:
> > https://issues.apache.org/jira/browse/KAFKA-2203
> > > .
> > > > >
> > > > > I issue same command as Stevo Slavic with JDK 1.8.0_91:
> > > > > gradle clean jarAll
> > > > >
> > > > > java -version
> > > > >
> > > > > java version "*1.8.0_91*"
> > > > >
> > > > > Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
> > > > >
> > > > > Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
> > > > >
> > > > >
> > > > > Much of it builds save for the examples, which incidentally is
> what I
> > > am
> > > > > mostly interested in.
> > > > >
> > > > >
> > > > > :core:jar UP-TO-DATE
> > > > >
> > > > > :examples:compileJava
> > > > >
> > > > > warning: [options] bootstrap class path not set in conjunction with
> > > > -source
> > > > > 1.7
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/philippederome/workspace/kafka/examples/src/main/java/kafka/examples/Consumer.java:19:
> > > > > error: package kafka.utils does not exist
> > > > >
> > > > > import kafka.utils.ShutdownableThread;
> > > > >
> > > > >  ^
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/philippederome/workspace/kafka/examples/src/main/java/kafka/examples/Consumer.java:28:
> > > > > error: cannot find symbol
> > > > >
> > > > > public class Consumer extends ShutdownableThread {
> > > > >
> > > > >  ^
> > > > >
> > > > >  symbol: class ShutdownableThread
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/philippederome/workspace/kafka/examples/src/main/java/kafka/examples/SimpleConsumerDemo.java:19:
> > > > > error: package kafka.api does not exist
> > > > >
> > > > > import kafka.api.FetchRequest;
> > > > >
> > > > > Phil Derome
> > > >
> > > >
> > >
> >
>

Re: Message loss with kafka 0.8.2.2

[Gerard Klijs]

You could try set the acks to -1, so you wait for the produce to be
succesfull, until most other brokers also received the message. Another
thing you could try is set the unclean.leader.election.enable to false
(this is a setting on the broker).
I think what's happening now is that the message in your example is send to
two different brokers, because one of them is not sending the record to the
actual leader. Since you have set your acks to one, you wont see any error
in the producer, cause it succeeded in sending it to the broker. You most
likely will see some error on the broker, because it is not the leader.

On Fri, Jun 17, 2016 at 5:19 AM Gulia, Vikram  wrote:

> Hi Users, I am facing message loss while using kafka v 0.8.2.2. Please see
> details below and help me if you can.
>
> Issue: 2 messages produced to same partition one by one – Kafka producer
> returns same offset back which means message produced earlier is lost.<
> http://stackoverflow.com/questions/37732088/2-messages-produced-to-same-partition-one-by-one-message-1-overridden-by-next
> >
>
> Details:
> I have a unique problem which is happening like 50-100 times a day with
> message volume of more than 2 millions per day on the topic.I am using
> Kafka producer API 0.8.2.2 and I have 12 brokers (v 0.8.2.2) running in
> prod with replication of 4. I have a topic with 60 partitions and I am
> calculating partition for all my messages and providing the value in the
> ProducerRecord itself. Now, the issue -
>
> Application creates 'ProducerRecord' using -
>
> new ProducerRecord(topic, 30, null, message1);
> providing topic, value message1 and partition 30. Then application call
> the send method and future is returned -
>
> // null is for callback
> Future future = producer.send(producerRecord. null);
> Now, app prints the offset and partition value by calling get on Future
> and then getting values from RecordMetadata - this is what i get -
>
> Kafka Response : partition 30, offset 3416092
> Now, the app produce the next message - message2 to same partition -
>
> new ProducerRecord(topic, 30, null, message2);
> and kafka response -
>
> Kafka Response : partition 30, offset 3416092
> I received the same offset again, and if I pull message from the offset of
> partition 30 using simple consumer, it ends up being the message2 which
> essentially mean i lost the message1.
>
> Currently, the messages are produced using 10 threads each having its own
> instance of kafka producer (Earlier threads shared 1 Kafka producer but it
> was performing slow and we still had message loss).
> I am using all default properties for producer except a few mentioned
> below, the message (String payload) size can be a few kbs to a 500 kbs. I
> am using acks value of 1.
>
> value.serializer: org.apache.kafka.common.serialization.StringSerializer
> key.serializer: org.apache.kafka.common.serialization.StringSerializer
> bootstrap.servers: {SERVER VIP ENDPOINT}
> acks: 1
> batch.size: 204800
> linger.ms: 10
> send.buffer.bytes: 1048576
> max.request.size: 1000
>
> What am i doing wrong here? Is there something I can look into or any
> producer property or server property I can tweak to make sure i don't lose
> any messages. I need some help here soon as I am losing some critical
> messages in production which is not good at all because as there is no
> exception given by Kafka Producer its even hard to find out the message
> lost unless downstream process reports it.
>
> Thank you,
> Vikram Gulia
>

Re: WordCount example errors

[Michael Noll]

Jeyhun,

just to confirm for you: Kafka Streams only works with Kafka 0.10 brokers
[1].

Best,
Michael

[1]
http://docs.confluent.io/3.0.0/streams/faq.html#can-i-use-kafka-streams-with-kafka-clusters-running-0-9-0-8-or-0-7



On Tue, Jun 14, 2016 at 3:03 PM, Jeyhun Karimov 
wrote:

> Thanks for reply. I am using the recent version of kafka and streams
> library. I forked them from github. @Peter, yes probably this is due to
> versions.
> The problem was that, I already installed kafka on OSX with brew and when I
> tried to run, probably it linked to that library instead of the one I
> forked from github. I deleted the installed library, now it works both from
> IDE and terminal.
>
> Thanks
>
> On Tue, Jun 14, 2016 at 2:47 PM Peter Davis  wrote:
>
> > This looks like the error that occurs when you use the 0.10 client with
> an
> > 0.9 broker.  The broker needs to be upgraded first.  Jeyhun, what
> versions
> > are you running?
> >
> > (I sincerely hope this error message will be improved next time there are
> > wire protocol changes!)
> >
> > -Peter
> >
> >
> > > On Jun 14, 2016, at 4:19 AM, Eno Thereska 
> > wrote:
> > >
> > > HI Jeyhun,
> > >
> > > What version of Kafka are you using?
> > >
> > > I haven't run this using Eclipse, but could you try building and
> running
> > from the command line (instead of from within Eclipse) as described in
> that
> > quickstart document? Does that work?
> > >
> > > Thanks
> > > Eno
> > >
> > >> On 13 Jun 2016, at 20:27, Jeyhun Karimov 
> wrote:
> > >>
> > >> Hi Community,
> > >>
> > >> I am new to kafka-streams and trying to run WordCount example.
> > >> I get this error whatever I do:
> > >>
> > >> Exception in thread "StreamThread-1"
> > >> org.apache.kafka.common.protocol.types.SchemaException: Error reading
> > field
> > >> 'topic_metadata': Error reading array of size 1209204, only 46 bytes
> > >> available
> > >>
> > >> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> > >>
> > >> at org.apache.kafka.clients.NetworkClient.parseResponse(
> > >> NetworkClient.java:380)
> > >>
> > >> at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(
> > >> NetworkClient.java:449)
> > >>
> > >> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
> > >>
> > >> at
> > >>
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(
> > >> ConsumerNetworkClient.java:360)
> > >>
> > >> at
> > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(
> > >> ConsumerNetworkClient.java:224)
> > >>
> > >> at
> > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(
> > >> ConsumerNetworkClient.java:192)
> > >>
> > >> at
> > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(
> > >> ConsumerNetworkClient.java:163)
> > >>
> > >> at
> > >>
> >
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(
> > >> AbstractCoordinator.java:179)
> > >>
> > >> at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(
> > >> KafkaConsumer.java:973)
> > >>
> > >> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(
> > >> KafkaConsumer.java:937)
> > >>
> > >> at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
> > >> StreamThread.java:295)
> > >>
> > >> at org.apache.kafka.streams.processor.internals.StreamThread.run(
> > >> StreamThread.java:218)
> > >>
> > >>
> > >> I am running exactly the same code WordCountDemo from github. I
> > downloaded
> > >> all libraries, build with gradle for eclipse, imported them to eclipse
> > and
> > >> run from there.
> > >> I initialized the zookeper and kafka-server and created kafka brooker
> as
> > >> described in http://docs.confluent.io/3.0.0/streams/quickstart.html.
> > >>
> > >> I am using Eclipse Scala IDE 3.0 .
> > >>
> > >>
> > >> Cheers
> > >> Jeyhun
> > >> --
> > >> -Cheers
> > >>
> > >> Jeyhun
> > >
> >
> --
> -Cheers
>
> Jeyhun
>

Re: Kafka Streams KTable-KTable Join Error

[Guozhang Wang]

Hello Tim,

By looking through the source code I suspect it is a bug in Kafka
Stream's KStreamWindowReduce
implementation. I'll do further investigation tomorrow and possibly file a
JIRA with a patch.


Guozhang


On Thu, Jun 16, 2016 at 4:12 PM, Tim Renner 
wrote:

> Hi all,
>
> I'm trying to do a KTable-KTable join to compute an average within a
> tumbling window.
> Here's the KStreams code (I've put a fully working example in a gist:
> https://gist.github.com/timothyrenner/a99c86b2d6ed2c22c8703e8c7760af3a)
> KStreamBuilder builder = new KStreamBuilder();
>
> KStream longs = builder.stream(
> Serdes.String(), Serdes.Long(), "longs");
>
> KTable longCounts =
> longs.countByKey(TimeWindows.of("longCounts", 1L),
> Serdes.String());
>
>
> KTable longSums =
> longs.reduceByKey((v1, v2) -> v1 + v2,
>  TimeWindows.of("longSums", 1L),
>  Serdes.String(),
>  Serdes.Long());
>
> KTable longAvgs =
> longSums.join(longCounts,
>  (sum, count) ->
>   sum.doubleValue()/count.doubleValue());
>
> longAvgs.toStream((wk, v) -> wk.key())
> .to(Serdes.String(),
> Serdes.Double(),
> "long-avgs");
>
> KafkaStreams streams = new KafkaStreams(builder, config);
> streams.start();
>
> When I run this, I get the following exception:
>
> java.util.NoSuchElementException
> at
>
> org.apache.kafka.streams.state.internals.RocksDBWindowStore$RocksDBWindowStoreIterator.next(RocksDBWindowStore.java:95)
> at
>
> org.apache.kafka.streams.state.internals.RocksDBWindowStore$RocksDBWindowStoreIterator.next(RocksDBWindowStore.java:64)
> at
>
> org.apache.kafka.streams.state.internals.MeteredWindowStore$MeteredWindowStoreIterator.next(MeteredWindowStore.java:136)
> at
>
> org.apache.kafka.streams.state.internals.MeteredWindowStore$MeteredWindowStoreIterator.next(MeteredWindowStore.java:117)
> at
>
> org.apache.kafka.streams.kstream.internals.KStreamWindowReduce$KStreamAggregateValueGetter.get(KStreamWindowReduce.java:166)
> at
>
> org.apache.kafka.streams.kstream.internals.KStreamWindowReduce$KStreamAggregateValueGetter.get(KStreamWindowReduce.java:147)
> at
>
> org.apache.kafka.streams.kstream.internals.KTableKTableJoin$KTableKTableJoinProcessor.process(KTableKTableJoin.java:77)
> at
>
> org.apache.kafka.streams.kstream.internals.KTableKTableJoin$KTableKTableJoinProcessor.process(KTableKTableJoin.java:48)
> at
>
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:68)
> at
>
> org.apache.kafka.streams.processor.internals.StreamTask.forward(StreamTask.java:338)
> at
>
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:187)
> at
>
> org.apache.kafka.streams.kstream.internals.KStreamWindowAggregate$KStreamWindowAggregateProcessor.process(KStreamWindowAggregate.java:136)
> at
>
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:68)
> at
>
> org.apache.kafka.streams.processor.internals.StreamTask.forward(StreamTask.java:338)
> at
>
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:187)
> at
>
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:64)
> at
>
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:174)
> at
>
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:320)
> at
>
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:218)
>
> Looks like the join is throwing the exception. Any ideas?
>
> Thanks,
> Tim
>



-- 
-- Guozhang