Re: build error

[Grant Henke]

I too would like to understand the command you used to build and what
directory.

That aside it is likely best to test using the release candidate artifacts
that are described in Jun's email here:
http://search-hadoop.com/m/uyzND1bBATE1M3hVn/candidate=+VOTE+0+9+0+0+Candiate+2

This gives you all built artifacts and a maven repo
 with the client
artifacts

.

Thanks,
Grant

On Tue, Nov 17, 2015 at 12:17 AM, Guozhang Wang  wrote:

> Did you just use "./gradlew build" in root directory?
>
> Guozhang
>
> On Mon, Nov 16, 2015 at 6:41 PM, hsy...@gmail.com 
> wrote:
>
> > The actual thing I want to do is I want to build and install in my local
> > maven repository so I can include new api in my dependencies. When the
> > release is officially out, I can have both my code ready with the
> official
> > maven dependency
> >
> > Thanks,
> > Siyuan
> >
> > On Monday, November 16, 2015, Grant Henke  wrote:
> >
> > > Hi Siyuan,
> > >
> > > My guess is that you are trying to build from a subdirectory. I have a
> > > minor patch available to fix this that has not been pulled in yet here:
> > > https://github.com/apache/kafka/pull/509
> > >
> > > In the mean time, if you need to build a subproject you can execute a
> > > command like the following:
> > > gradle clients:build
> > >
> > > Thanks,
> > > Grant
> > >
> > > On Mon, Nov 16, 2015 at 6:33 PM, Guozhang Wang  > > > wrote:
> > >
> > > > Siyuan,
> > > >
> > > > Which command did you use to build?
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Nov 16, 2015 at 4:01 PM, hsy...@gmail.com  <
> > > hsy...@gmail.com >
> > > > wrote:
> > > >
> > > > > I got a build error on both trunk and 0.9.0 branch
> > > > >
> > > > > > docs/producer_config.html (No such file or directory)
> > > > >
> > > > > Do I miss anything before build
> > > > >
> > > > > Thanks,
> > > > > Siyuan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > >
> > > --
> > > Grant Henke
> > > Software Engineer | Cloudera
> > > gr...@cloudera.com  | twitter.com/gchenke |
> > > linkedin.com/in/granthenke
> > >
> >
>
>
>
> --
> -- Guozhang
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

issue with javaapi consumer

[Kudumula, Surender]

Hi all
Iam currently using kafka 0.8.2.1 consumer which has kafka java api consumer 
but I had to upgrade to kafka 0.8.2.3 and when I add the jars for kafka 0.8.2.3 
my consumer code doesn't compile.
consumerConnector = ConsumerConfig
 
.createJavaConsumerConnector(consumerConfig);

Do I have to rewrite the whole consumer to use the new jar. Whats the best way 
forward please?

Regards

Surender Kudumula

Is a Kafka 0.9 broker supposed to connect to itself?

[Damian Guy]

I would think not
I'm bringing up a new 0.9 cluster and i'm getting the below Exception (and
the same thing on all nodes) - the IP address is the IP for the host the
broker is running on. I think DNS is a bit stuffed on these machines and
maybe that is the cause, but... any ideas?

[2015-11-17 04:01:30,248] WARN Unexpected error from /10.137.231.233;
closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive
(size = 1195725856 larger than 104857600)
at
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:91)
at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
at
org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:160)
at
org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:141)
at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
at kafka.network.Processor.run(SocketServer.scala:413)
at java.lang.Thread.run(Thread.java:745)

Re: build error

[hsy...@gmail.com]

I got main class not found error. So I installed gradle 2.5 and run gradle
build (not the wrapper)

On Mon, Nov 16, 2015 at 10:17 PM, Guozhang Wang  wrote:

> Did you just use "./gradlew build" in root directory?
>
> Guozhang
>
> On Mon, Nov 16, 2015 at 6:41 PM, hsy...@gmail.com 
> wrote:
>
> > The actual thing I want to do is I want to build and install in my local
> > maven repository so I can include new api in my dependencies. When the
> > release is officially out, I can have both my code ready with the
> official
> > maven dependency
> >
> > Thanks,
> > Siyuan
> >
> > On Monday, November 16, 2015, Grant Henke  wrote:
> >
> > > Hi Siyuan,
> > >
> > > My guess is that you are trying to build from a subdirectory. I have a
> > > minor patch available to fix this that has not been pulled in yet here:
> > > https://github.com/apache/kafka/pull/509
> > >
> > > In the mean time, if you need to build a subproject you can execute a
> > > command like the following:
> > > gradle clients:build
> > >
> > > Thanks,
> > > Grant
> > >
> > > On Mon, Nov 16, 2015 at 6:33 PM, Guozhang Wang  > > > wrote:
> > >
> > > > Siyuan,
> > > >
> > > > Which command did you use to build?
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Nov 16, 2015 at 4:01 PM, hsy...@gmail.com  <
> > > hsy...@gmail.com >
> > > > wrote:
> > > >
> > > > > I got a build error on both trunk and 0.9.0 branch
> > > > >
> > > > > > docs/producer_config.html (No such file or directory)
> > > > >
> > > > > Do I miss anything before build
> > > > >
> > > > > Thanks,
> > > > > Siyuan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > >
> > > --
> > > Grant Henke
> > > Software Engineer | Cloudera
> > > gr...@cloudera.com  | twitter.com/gchenke |
> > > linkedin.com/in/granthenke
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: build error

[Grant Henke]

Glad you resolved it. Note that we are currently on gradle 2.8 and likely
2.9 soon feel free to update whenever makes sense for you.

Ewen recently updated the developer setup page here
 to show
how to use the wrapper effectively in the Kafka repository. Since we do not
checkin the wrapper jar, you need to execute the default gradle task by
simply executing `gradle` before  running and gradle wrapper tasks.

See here

for the related email chain.

On Tue, Nov 17, 2015 at 10:27 AM, hsy...@gmail.com  wrote:

> I got main class not found error. So I installed gradle 2.5 and run gradle
> build (not the wrapper)
>
> On Mon, Nov 16, 2015 at 10:17 PM, Guozhang Wang 
> wrote:
>
> > Did you just use "./gradlew build" in root directory?
> >
> > Guozhang
> >
> > On Mon, Nov 16, 2015 at 6:41 PM, hsy...@gmail.com 
> > wrote:
> >
> > > The actual thing I want to do is I want to build and install in my
> local
> > > maven repository so I can include new api in my dependencies. When the
> > > release is officially out, I can have both my code ready with the
> > official
> > > maven dependency
> > >
> > > Thanks,
> > > Siyuan
> > >
> > > On Monday, November 16, 2015, Grant Henke  wrote:
> > >
> > > > Hi Siyuan,
> > > >
> > > > My guess is that you are trying to build from a subdirectory. I have
> a
> > > > minor patch available to fix this that has not been pulled in yet
> here:
> > > > https://github.com/apache/kafka/pull/509
> > > >
> > > > In the mean time, if you need to build a subproject you can execute a
> > > > command like the following:
> > > > gradle clients:build
> > > >
> > > > Thanks,
> > > > Grant
> > > >
> > > > On Mon, Nov 16, 2015 at 6:33 PM, Guozhang Wang  > > > > wrote:
> > > >
> > > > > Siyuan,
> > > > >
> > > > > Which command did you use to build?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Nov 16, 2015 at 4:01 PM, hsy...@gmail.com  <
> > > > hsy...@gmail.com >
> > > > > wrote:
> > > > >
> > > > > > I got a build error on both trunk and 0.9.0 branch
> > > > > >
> > > > > > > docs/producer_config.html (No such file or directory)
> > > > > >
> > > > > > Do I miss anything before build
> > > > > >
> > > > > > Thanks,
> > > > > > Siyuan
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Grant Henke
> > > > Software Engineer | Cloudera
> > > > gr...@cloudera.com  | twitter.com/gchenke |
> > > > linkedin.com/in/granthenke
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: build error

[hsy...@gmail.com]

And I couldn't find wrapper jar files under the gradle folder
https://github.com/apache/kafka/tree/0.9.0/gradle



On Mon, Nov 16, 2015 at 10:17 PM, Guozhang Wang  wrote:

> Did you just use "./gradlew build" in root directory?
>
> Guozhang
>
> On Mon, Nov 16, 2015 at 6:41 PM, hsy...@gmail.com 
> wrote:
>
> > The actual thing I want to do is I want to build and install in my local
> > maven repository so I can include new api in my dependencies. When the
> > release is officially out, I can have both my code ready with the
> official
> > maven dependency
> >
> > Thanks,
> > Siyuan
> >
> > On Monday, November 16, 2015, Grant Henke  wrote:
> >
> > > Hi Siyuan,
> > >
> > > My guess is that you are trying to build from a subdirectory. I have a
> > > minor patch available to fix this that has not been pulled in yet here:
> > > https://github.com/apache/kafka/pull/509
> > >
> > > In the mean time, if you need to build a subproject you can execute a
> > > command like the following:
> > > gradle clients:build
> > >
> > > Thanks,
> > > Grant
> > >
> > > On Mon, Nov 16, 2015 at 6:33 PM, Guozhang Wang  > > > wrote:
> > >
> > > > Siyuan,
> > > >
> > > > Which command did you use to build?
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Nov 16, 2015 at 4:01 PM, hsy...@gmail.com  <
> > > hsy...@gmail.com >
> > > > wrote:
> > > >
> > > > > I got a build error on both trunk and 0.9.0 branch
> > > > >
> > > > > > docs/producer_config.html (No such file or directory)
> > > > >
> > > > > Do I miss anything before build
> > > > >
> > > > > Thanks,
> > > > > Siyuan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > >
> > > --
> > > Grant Henke
> > > Software Engineer | Cloudera
> > > gr...@cloudera.com  | twitter.com/gchenke |
> > > linkedin.com/in/granthenke
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: issue with javaapi consumer

[Grant Henke]

Are you sure you mean 0.8.2.3? The latest version available is 0.8.2.2, and
it is completely compatible.
http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.kafka%22%20a%3A%22kafka_2.10%22

If you still have an issue, what error are you seeing?

Thanks,
Grant

On Tue, Nov 17, 2015 at 5:47 AM, Kudumula, Surender <
surender.kudum...@hpe.com> wrote:

> Hi all
> Iam currently using kafka 0.8.2.1 consumer which has kafka java api
> consumer but I had to upgrade to kafka 0.8.2.3 and when I add the jars for
> kafka 0.8.2.3 my consumer code doesn't compile.
> consumerConnector = ConsumerConfig
>
>  .createJavaConsumerConnector(consumerConfig);
>
> Do I have to rewrite the whole consumer to use the new jar. Whats the best
> way forward please?
>
> Regards
>
> Surender Kudumula
>
>
>


-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: Questions about new consumer API

[hsy...@gmail.com]

Thanks Guozhang,

Maybe I should give a few words about what I'm going to achieve with new API

Currently, I'm building a new kafka connector for Apache Apex(
http://apex.incubator.apache.org/) using 0.9.0 API
Apex support dynamic partition, so in the old version, We manage all the
consumer partitions in either 1:1 strategy (each consumer process consumes
only from one kafka partition) or 1:n strategy (each consumer process could
consume from multiple kafka partitions, using round-robin to distribute)
And we also have separate thread to monitor topic metadata change(leader
broker change, new partition added, using internal API like ZkUtil etc)
and do dynamic partition based on that(for example auto-reconnect to new
leader broker, create new partition to consume from new kafka partition at
runtime).  You can see High-level consumer doesn't work(It can only balance
between existing consumers unless you manually add new one)  I'm thinking
if the new consumer could be used to save some work we did before.

I'm still confused with assign() and subscribe().  My understanding is if
you use assign() only, the consumer becomes more like a simple consumer
except if the leader broker changes it automatically reconnect to the new
leader broker, is it correct?   If you use subscribe() method only then all
the partitions will be distributed to running consumer process with same "
group.id" using "partition.assignment.strategy". Is it true?

So I assume assign() and subscribe()(and group.id
partition.assignment.strategy settings) can not be used together?

Also in the old API we found one thread per broker is the most efficient
way to consume data, for example, if one process consumes from p1, p2, p3
and p1,p2 are sitting on one broker b1, p3 is sitting on another one b2,
the best thing is create 2 threads each thread use simple consumer API and
only consume from one broker.  I'm thinking how do I use the new API to do
this.

Thanks,
Siyuan

On Mon, Nov 16, 2015 at 4:43 PM, Guozhang Wang  wrote:

> Hi Siyuan,
>
> 1) new consumer is single-threaded, it does not maintain any internal
> threads as the old high-level consumer.
>
> 2) each consumer will only maintain one TCP connection with each broker.
> The only extra socket is the one with its coordinator. That is, if there is
> three brokers S1, S2, S3, and S1 is the coordinator for this consumer, it
> will maintain 4 sockets in total, 2 for S1 (one for fetching, one for
> coordinating) and 1 for S2 and S3 (only for fetching).
>
> 3) Currently the connection is not closed by consumer, although the
> underlying network client / selector will close idle ones after some
> timeout. So in worst case it will only maintain N+1 sockets in total for N
> Kafka brokers at one time.
>
> Guozhang
>
> On Mon, Nov 16, 2015 at 4:22 PM, hsy...@gmail.com 
> wrote:
>
> > The new consumer API looks good. If I understand it correctly you can use
> > it like simple consumer or high-level consumer. But I have couple
> questions
> > about it's internal implementation
> >
> > First of all does the consumer have any internal fetcher threads like
> > high-level consumer?
> >
> > When you assign multiple TopicPartitions to a consumer, how many TCP
> > connections it establish to the brokers. Is it same as number of leader
> > brokers that host those partitions or just number of TopicPartitions. If
> > there is any leader broker change does it establish new connections/using
> > existing connections to fetch the data? Can it continue consuming? Also
> is
> > the connection kept until the consumer is closed?
> >
> > Thanks!
> >
> > Best,
> > Siyuan
> >
>
>
>
> --
> -- Guozhang
>

Re: Questions about new consumer API

[Jason Gustafson]

Hi Siyuan,

Your understanding about assign/subscribe is correct. We think of topic
subscription as enabling automatic assignment as opposed to doing manual
assignment through assign(). We don't currently them to be mixed.

Can you elaborate on your findings with respect to using one thread per
broker? In what sense was it more efficient? Doing the same thing might be
tricky with the new consumer, but I think you could do it using
partitionsFor() to find the current partition leaders and assign() to set
the assignment in each thread.

-Jason

On Tue, Nov 17, 2015 at 10:25 AM, hsy...@gmail.com  wrote:

> Thanks Guozhang,
>
> Maybe I should give a few words about what I'm going to achieve with new
> API
>
> Currently, I'm building a new kafka connector for Apache Apex(
> http://apex.incubator.apache.org/) using 0.9.0 API
> Apex support dynamic partition, so in the old version, We manage all the
> consumer partitions in either 1:1 strategy (each consumer process consumes
> only from one kafka partition) or 1:n strategy (each consumer process could
> consume from multiple kafka partitions, using round-robin to distribute)
> And we also have separate thread to monitor topic metadata change(leader
> broker change, new partition added, using internal API like ZkUtil etc)
> and do dynamic partition based on that(for example auto-reconnect to new
> leader broker, create new partition to consume from new kafka partition at
> runtime).  You can see High-level consumer doesn't work(It can only balance
> between existing consumers unless you manually add new one)  I'm thinking
> if the new consumer could be used to save some work we did before.
>
> I'm still confused with assign() and subscribe().  My understanding is if
> you use assign() only, the consumer becomes more like a simple consumer
> except if the leader broker changes it automatically reconnect to the new
> leader broker, is it correct?   If you use subscribe() method only then all
> the partitions will be distributed to running consumer process with same "
> group.id" using "partition.assignment.strategy". Is it true?
>
> So I assume assign() and subscribe()(and group.id
> partition.assignment.strategy settings) can not be used together?
>
> Also in the old API we found one thread per broker is the most efficient
> way to consume data, for example, if one process consumes from p1, p2, p3
> and p1,p2 are sitting on one broker b1, p3 is sitting on another one b2,
> the best thing is create 2 threads each thread use simple consumer API and
> only consume from one broker.  I'm thinking how do I use the new API to do
> this.
>
> Thanks,
> Siyuan
>
> On Mon, Nov 16, 2015 at 4:43 PM, Guozhang Wang  wrote:
>
> > Hi Siyuan,
> >
> > 1) new consumer is single-threaded, it does not maintain any internal
> > threads as the old high-level consumer.
> >
> > 2) each consumer will only maintain one TCP connection with each broker.
> > The only extra socket is the one with its coordinator. That is, if there
> is
> > three brokers S1, S2, S3, and S1 is the coordinator for this consumer, it
> > will maintain 4 sockets in total, 2 for S1 (one for fetching, one for
> > coordinating) and 1 for S2 and S3 (only for fetching).
> >
> > 3) Currently the connection is not closed by consumer, although the
> > underlying network client / selector will close idle ones after some
> > timeout. So in worst case it will only maintain N+1 sockets in total for
> N
> > Kafka brokers at one time.
> >
> > Guozhang
> >
> > On Mon, Nov 16, 2015 at 4:22 PM, hsy...@gmail.com 
> > wrote:
> >
> > > The new consumer API looks good. If I understand it correctly you can
> use
> > > it like simple consumer or high-level consumer. But I have couple
> > questions
> > > about it's internal implementation
> > >
> > > First of all does the consumer have any internal fetcher threads like
> > > high-level consumer?
> > >
> > > When you assign multiple TopicPartitions to a consumer, how many TCP
> > > connections it establish to the brokers. Is it same as number of leader
> > > brokers that host those partitions or just number of TopicPartitions.
> If
> > > there is any leader broker change does it establish new
> connections/using
> > > existing connections to fetch the data? Can it continue consuming? Also
> > is
> > > the connection kept until the consumer is closed?
> > >
> > > Thanks!
> > >
> > > Best,
> > > Siyuan
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: Custom metadata in message header

[Lance Laursen]

Hey Lesong,

Unfortunately not. Any metadata you would like to include must be placed in
the message payload. This means you must consume each message and process
it in order to determine whether it is relevant to your current task.

You can achieve some data locality by using keyed messages and an
associated partitioner.class (ie: semantic partitioning). This will result
in messages with a common key ending up on the same partition, thereby
reducing the workload of your consumer.

On Tue, Nov 17, 2015 at 12:55 PM, Lesong Jia  wrote:

> Hey all,
>
> I am new to Kafka. Is there a way to provide a custom metadata in the
> message header?
>
> Thanks,
> Lesong
>

Re: Questions about new consumer API

[hsy...@gmail.com]

By efficiency, I mean maximize throughput while minimize resources on both
broker sides and consumer sides.

One example is if you have over 200 partitions on 10 brokers and you can
start 5 consumer processes to consume data, if each one is single-thread
and you do round-robin to distribute the load then each one will try to
fetch from over 40 partitions one by one through 10 connections
possibly(overall is 50),  but if it's smart enough to group partitions by
brokers, each process can have 2 separate threads(consuming from 2
different brokers concurrently). That seems a more optimal solution than
another, right?

On Tue, Nov 17, 2015 at 2:54 PM, Jason Gustafson  wrote:

> Hi Siyuan,
>
> Your understanding about assign/subscribe is correct. We think of topic
> subscription as enabling automatic assignment as opposed to doing manual
> assignment through assign(). We don't currently them to be mixed.
>
> Can you elaborate on your findings with respect to using one thread per
> broker? In what sense was it more efficient? Doing the same thing might be
> tricky with the new consumer, but I think you could do it using
> partitionsFor() to find the current partition leaders and assign() to set
> the assignment in each thread.
>
> -Jason
>
> On Tue, Nov 17, 2015 at 10:25 AM, hsy...@gmail.com 
> wrote:
>
> > Thanks Guozhang,
> >
> > Maybe I should give a few words about what I'm going to achieve with new
> > API
> >
> > Currently, I'm building a new kafka connector for Apache Apex(
> > http://apex.incubator.apache.org/) using 0.9.0 API
> > Apex support dynamic partition, so in the old version, We manage all the
> > consumer partitions in either 1:1 strategy (each consumer process
> consumes
> > only from one kafka partition) or 1:n strategy (each consumer process
> could
> > consume from multiple kafka partitions, using round-robin to distribute)
> > And we also have separate thread to monitor topic metadata change(leader
> > broker change, new partition added, using internal API like ZkUtil etc)
> > and do dynamic partition based on that(for example auto-reconnect to new
> > leader broker, create new partition to consume from new kafka partition
> at
> > runtime).  You can see High-level consumer doesn't work(It can only
> balance
> > between existing consumers unless you manually add new one)  I'm thinking
> > if the new consumer could be used to save some work we did before.
> >
> > I'm still confused with assign() and subscribe().  My understanding is if
> > you use assign() only, the consumer becomes more like a simple consumer
> > except if the leader broker changes it automatically reconnect to the new
> > leader broker, is it correct?   If you use subscribe() method only then
> all
> > the partitions will be distributed to running consumer process with same
> "
> > group.id" using "partition.assignment.strategy". Is it true?
> >
> > So I assume assign() and subscribe()(and group.id
> > partition.assignment.strategy settings) can not be used together?
> >
> > Also in the old API we found one thread per broker is the most efficient
> > way to consume data, for example, if one process consumes from p1, p2, p3
> > and p1,p2 are sitting on one broker b1, p3 is sitting on another one b2,
> > the best thing is create 2 threads each thread use simple consumer API
> and
> > only consume from one broker.  I'm thinking how do I use the new API to
> do
> > this.
> >
> > Thanks,
> > Siyuan
> >
> > On Mon, Nov 16, 2015 at 4:43 PM, Guozhang Wang 
> wrote:
> >
> > > Hi Siyuan,
> > >
> > > 1) new consumer is single-threaded, it does not maintain any internal
> > > threads as the old high-level consumer.
> > >
> > > 2) each consumer will only maintain one TCP connection with each
> broker.
> > > The only extra socket is the one with its coordinator. That is, if
> there
> > is
> > > three brokers S1, S2, S3, and S1 is the coordinator for this consumer,
> it
> > > will maintain 4 sockets in total, 2 for S1 (one for fetching, one for
> > > coordinating) and 1 for S2 and S3 (only for fetching).
> > >
> > > 3) Currently the connection is not closed by consumer, although the
> > > underlying network client / selector will close idle ones after some
> > > timeout. So in worst case it will only maintain N+1 sockets in total
> for
> > N
> > > Kafka brokers at one time.
> > >
> > > Guozhang
> > >
> > > On Mon, Nov 16, 2015 at 4:22 PM, hsy...@gmail.com 
> > > wrote:
> > >
> > > > The new consumer API looks good. If I understand it correctly you can
> > use
> > > > it like simple consumer or high-level consumer. But I have couple
> > > questions
> > > > about it's internal implementation
> > > >
> > > > First of all does the consumer have any internal fetcher threads like
> > > > high-level consumer?
> > > >
> > > > When you assign multiple TopicPartitions to a consumer, how many TCP
> > > > connections it establish to the brokers. Is it same as number of
> 

RE: where do I get the Kafka classes

[Adaryl Wakefield]

Learning build tools is going to take a bit. For the time being, I'd really 
just like to get the jars and add them manually. I'm not sure anybody has 
showed me where to get the jars yet.
B.

> From: ghe...@cloudera.com
> Date: Tue, 17 Nov 2015 08:25:11 -0600
> Subject: Re: where do I get the Kafka classes
> To: users@kafka.apache.org
> 
> Hi Adaryl,
> 
> Kafka and its clients doesn't define how you must build your project. You
> are free to pull together all the jars needed and include them however you
> like. Though that may be difficult and error prone depending on the method
> you choose. Thats where build tools like maven, gradle, etc come in. They
> make sure your build is "easy", repeatable, and clear.
> 
> Here are some answers on the web about why using a build tool is a good
> idea:
> 
>-
>
> http://stackoverflow.com/questions/1382432/whats-with-all-the-java-build-tools
>-
>
> http://stackoverflow.com/questions/10764576/why-do-we-need-maven-or-ant-if-we-already-have-eclipse
> 
> Here are some quick-start guides to get you started with maven or gradle:
> 
>- https://maven.apache.org/guides/getting-started/
>- https://docs.gradle.org/current/userguide/tutorials.html#N100D0
> 
> Stackoverflow is often a good place for general programming or java
> questions (Just search first to be sure it hasn't been asked before). For
> Kafka questions you can also search the mailing list archives here:
> http://search-hadoop.com/kafka
> 
> Hope that helps get you going. If you have any more Kafka related
> questions, don't hesitate to ask.
> 
> Thanks,
> Grant
> 
> On Tue, Nov 17, 2015 at 12:51 AM, Adaryl "Bob" Wakefield, MBA <
> adaryl.wakefi...@hotmail.com> wrote:
> 
> > Is it not possible to just manually include the packages in my Eclipse
> > project? Do you have to use a build tool?
> >
> >
> >
> > Adaryl "Bob" Wakefield, MBA
> > Principal
> > Mass Street Analytics, LLC
> > 913.938.6685
> > www.linkedin.com/in/bobwakefieldmba
> > Twitter: @BobLovesData
> > -Original Message- From: Ewen Cheslack-Postava
> > Sent: Tuesday, November 17, 2015 12:32 AM
> > To: users@kafka.apache.org
> > Subject: Re: where do I get the Kafka classes
> >
> >
> > Hi Adaryl,
> >
> > First, it looks like you might be trying to use the old producer interface.
> > That interface is going to be deprecated in favor of the new producer
> > (under org.apache.kafka.clients.producer). I'd highly recommend using the
> > new producer interface instead.
> >
> > Second, perhaps this repository of examples will be a helpful starting
> > point: https://github.com/confluentinc/examples It's just a few basic
> > examples, but also includes the necessary Maven build scripts. For example,
> > the couple of lines after the highlighted one here:
> > https://github.com/confluentinc/examples/blob/master/producer/pom.xml#L32
> > will include the necessary jar that includes the new producer.
> >
> > -Ewen
> >
> > On Mon, Nov 16, 2015 at 10:16 PM, Adaryl "Bob" Wakefield, MBA <
> > adaryl.wakefi...@hotmail.com> wrote:
> >
> > I'm somewhat new to java development and am studying how to write
> >> producers. The sample code I'm looking at has the following import
> >> statements:
> >>
> >> import kafka.javaapi.producer.Producer;
> >> import kafka.producer.KeyedMessage;
> >> import kafka.producer.ProducerConfig;
> >>
> >> The thing is, he doesn't use any packages that contain these classes.
> >> You're supposed to use Gradle to compile the code but I'm not a whiz with
> >> Gradle yet. I'm guessing that Gradle is somehow importing the necessary
> >> classes at compile time. If I didn't want to use Gradle, how would I go
> >> about just getting Kafka packages with the classes I need? I can't seem to
> >> find them by googling.
> >>
> >>
> >> Adaryl "Bob" Wakefield, MBA
> >> Principal
> >> Mass Street Analytics, LLC
> >> 913.938.6685
> >> www.linkedin.com/in/bobwakefieldmba
> >> Twitter: @BobLovesData
> >>
> >>
> >
> >
> > --
> > Thanks,
> > Ewen
> >
> 
> 
> 
> -- 
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
  

questions on number of partitions

[Chen Song]

We have a cluster of 32 nodes cluster, with a few topics. When we set 100
partitions for each topic, the overall CPU usage is 30%, and when we
increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. The #
of producers are not changed.

I understand that more partitions will drive up broker CPU usage. However,
this seems to be a limiting factor to parallelism (mostly for consumer
perspective) because all I test is just 400 partitions.

Not sure if this is being addressed in an way. Any thoughts on this?

-- 
Chen Song

Re: Optimal number of partitions for topic

[Chen Song]

Hi Carles

Just check in, did you do the benchmarks eventually on your end? Curious to
know the data points.

Chen

On Thu, May 21, 2015 at 2:47 PM, Carles Sistare <
carles.sist...@googlemail.com> wrote:

> Thanks a lot guys for your answers,
> We’ll be doing some benchmarks comparing different amount of partitions
> for the same load. We’ll share the results.
>
> Cheers
>
> > On 21 May 2015, at 06:04, Saladi Naidu 
> wrote:
> >
> > In general partitions are to improve throughput by parallelism. From
> your explanation below yes partitions are written to different physical
> locations but still append only. With write ahead buffering and append only
> writes, having partitions still will increase throughput.
> > Below is an excellent article by Jun Rao about partitions and its impact
> > How to choose the number of topics/partitions in a Kafka cluster?
> >
> > |   |
> > |   |  |   |   |   |   |   |
> > | How to choose the number of topics/partitions in a Kafka...This is a
> common question asked by many Kafka users. The goal of this post is to
> explain a few important determining factors and provide a few simple
> formulas. More... |
> > |  |
> > | View on blog.confluent.io | Preview by Yahoo |
> > |  |
> > |   |
> >
> >   Naidu Saladi
> >
> >  From: Daniel Compton 
> > To: users@kafka.apache.org
> > Sent: Wednesday, May 20, 2015 8:21 PM
> > Subject: Re: Optimal number of partitions for topic
> >
> > One of the beautiful things about Kafka is that it uses the disk and OS
> > disk caching really efficiently.
> >
> > Because Kafka writes messages to a contiguous log, it needs very little
> > seek time to move the write head to the next point. Similarly for
> reading,
> > if the consumers are mostly up to date with the topic then the disk head
> > will be close to the reading point, or the data will already be in disk
> > cache and can be read from memory. Even cooler, because disk access is so
> > predictable, the OS can easily prefetch data because it knows what you're
> > going to ask for in 50ms.
> >
> > In summary, data locality rocks.
> >
> > What I haven't worked out is how the data locality benefits interact with
> > having multiple partitions. I would assume that this would make things
> > slower because you will be writing to multiple physical locations on
> disk,
> > though this may be ameliorated by only fsyncing every n seconds.
> >
> > I'd be keen to hear how this impacts it, or even better to see some
> > benchmarks.
> >
> > --
> > Daniel.
> >
> >
> >
> >
> > On Thu, 21 May 2015 at 12:56 pm Manoj Khangaonkar  >
> > wrote:
> >
> >> With knowing the actual implementation details, I would get guess more
> >> partitions implies more parallelism, more concurrency, more threads,
> more
> >> files to write to - all of which will contribute to more CPU load.
> >>
> >> Partitions allow you to scale by partitioning the topic across multiple
> >> brokers. Partition is also a unit of replication ( 1 leader + replicas
> ).
> >> And for consumption of messages, the order is maintained within a
> >> partitions.
> >>
> >> But if you put 100 partitions per topic on 1 single broker, I wonder if
> it
> >> is going to be an overhead.
> >>
> >>
> >>
> >> On Wed, May 20, 2015 at 1:02 AM, Carles Sistare 
> wrote:
> >>
> >>> Hi,
> >>> We are implementing a Kafka cluster with 9 brokers into EC2 instances,
> >> and
> >>> we are trying to find out the optimal number of partitions for our
> >> topics,
> >>> finding out the maximal number in order not to update the partition
> >> number
> >>> anymore.
> >>> What we understood is that the number of partitions shouldn’t affect
> the
> >>> CPU load of the brokers, but when we add 512 partitions instead of 128,
> >> for
> >>> instance, the CPU load exploses.
> >>> We have three topics with 10 messages/sec each, a replication
> factor
> >>> of 3 and two consumer groups for each partition.
> >>>
> >>> Could somebody explain, why the increase of the number of partitions
> has
> >> a
> >>> so dramatic impact to the CPU load?
> >>>
> >>>
> >>> Here under i paste the config file of kafka:
> >>>
> >>> broker.id=3
> >>>
> >>> default.replication.factor=3
> >>>
> >>>
> >>> # The port the socket server listens on
> >>> port=9092
> >>>
> >>> # The number of threads handling network requests
> >>> num.network.threads=2
> >>>
> >>> # The number of threads doing disk I/O
> >>> num.io.threads=8
> >>>
> >>> # The send buffer (SO_SNDBUF) used by the socket server
> >>> socket.send.buffer.bytes=1048576
> >>>
> >>> # The receive buffer (SO_RCVBUF) used by the socket server
> >>> socket.receive.buffer.bytes=1048576
> >>>
> >>> # The maximum size of a request that the socket server will accept
> >>> (protection against OOM)
> >>> socket.request.max.bytes=104857600
> >>>
> >>>
> >>>
> >>> # A comma seperated list of directories under which to store log files
> >>> 

Re: questions on number of partitions

[Chen Song]

BTW, we are running Kafka 0.8.2.2.

On Tue, Nov 17, 2015 at 3:48 PM, Chen Song  wrote:

> We have a cluster of 32 nodes cluster, with a few topics. When we set 100
> partitions for each topic, the overall CPU usage is 30%, and when we
> increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. The #
> of producers are not changed.
>
> I understand that more partitions will drive up broker CPU usage. However,
> this seems to be a limiting factor to parallelism (mostly for consumer
> perspective) because all I test is just 400 partitions.
>
> Not sure if this is being addressed in an way. Any thoughts on this?
>
> --
> Chen Song
>
>


-- 
Chen Song

Custom metadata in message header

[Lesong Jia]

Hey all,

I am new to Kafka. Is there a way to provide a custom metadata in the
message header?

Thanks,
Lesong

Re: Is a Kafka 0.9 broker supposed to connect to itself?

[Jun Rao]

There is inter-broker communication. It seems that the broker got a request
more than the default allowed size (~10MB). How many topic/partitions do
you have on this cluster? Do you have clients running on the broker host?

Thanks,

Jun

On Tue, Nov 17, 2015 at 4:10 AM, Damian Guy  wrote:

> I would think not
> I'm bringing up a new 0.9 cluster and i'm getting the below Exception (and
> the same thing on all nodes) - the IP address is the IP for the host the
> broker is running on. I think DNS is a bit stuffed on these machines and
> maybe that is the cause, but... any ideas?
>
> [2015-11-17 04:01:30,248] WARN Unexpected error from /10.137.231.233;
> closing connection (org.apache.kafka.common.network.Selector)
> org.apache.kafka.common.network.InvalidReceiveException: Invalid receive
> (size = 1195725856 larger than 104857600)
> at
>
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:91)
> at
>
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:160)
> at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:141)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
> at kafka.network.Processor.run(SocketServer.scala:413)
> at java.lang.Thread.run(Thread.java:745)
>

Re: Questions about new consumer API

[Jason Gustafson]

Thanks for the explanation. Certainly you'd use less connections with this
approach, but it might be worthwhile to do some performance analysis to see
whether there is much difference in throughput (I'd be interested in seeing
these results myself). Another approach that might be interesting would be
to implement your own partition assignor which took into account the
leaders of each partition. Then you could just use subscribe() and let
Kafka manage the group for you. This is similar to how we were thinking of
implementing consumer rack-awareness.

-Jason

On Tue, Nov 17, 2015 at 4:04 PM, hsy...@gmail.com  wrote:

> By efficiency, I mean maximize throughput while minimize resources on both
> broker sides and consumer sides.
>
> One example is if you have over 200 partitions on 10 brokers and you can
> start 5 consumer processes to consume data, if each one is single-thread
> and you do round-robin to distribute the load then each one will try to
> fetch from over 40 partitions one by one through 10 connections
> possibly(overall is 50),  but if it's smart enough to group partitions by
> brokers, each process can have 2 separate threads(consuming from 2
> different brokers concurrently). That seems a more optimal solution than
> another, right?
>
> On Tue, Nov 17, 2015 at 2:54 PM, Jason Gustafson 
> wrote:
>
> > Hi Siyuan,
> >
> > Your understanding about assign/subscribe is correct. We think of topic
> > subscription as enabling automatic assignment as opposed to doing manual
> > assignment through assign(). We don't currently them to be mixed.
> >
> > Can you elaborate on your findings with respect to using one thread per
> > broker? In what sense was it more efficient? Doing the same thing might
> be
> > tricky with the new consumer, but I think you could do it using
> > partitionsFor() to find the current partition leaders and assign() to set
> > the assignment in each thread.
> >
> > -Jason
> >
> > On Tue, Nov 17, 2015 at 10:25 AM, hsy...@gmail.com 
> > wrote:
> >
> > > Thanks Guozhang,
> > >
> > > Maybe I should give a few words about what I'm going to achieve with
> new
> > > API
> > >
> > > Currently, I'm building a new kafka connector for Apache Apex(
> > > http://apex.incubator.apache.org/) using 0.9.0 API
> > > Apex support dynamic partition, so in the old version, We manage all
> the
> > > consumer partitions in either 1:1 strategy (each consumer process
> > consumes
> > > only from one kafka partition) or 1:n strategy (each consumer process
> > could
> > > consume from multiple kafka partitions, using round-robin to
> distribute)
> > > And we also have separate thread to monitor topic metadata
> change(leader
> > > broker change, new partition added, using internal API like ZkUtil etc)
> > > and do dynamic partition based on that(for example auto-reconnect to
> new
> > > leader broker, create new partition to consume from new kafka partition
> > at
> > > runtime).  You can see High-level consumer doesn't work(It can only
> > balance
> > > between existing consumers unless you manually add new one)  I'm
> thinking
> > > if the new consumer could be used to save some work we did before.
> > >
> > > I'm still confused with assign() and subscribe().  My understanding is
> if
> > > you use assign() only, the consumer becomes more like a simple consumer
> > > except if the leader broker changes it automatically reconnect to the
> new
> > > leader broker, is it correct?   If you use subscribe() method only then
> > all
> > > the partitions will be distributed to running consumer process with
> same
> > "
> > > group.id" using "partition.assignment.strategy". Is it true?
> > >
> > > So I assume assign() and subscribe()(and group.id
> > > partition.assignment.strategy settings) can not be used together?
> > >
> > > Also in the old API we found one thread per broker is the most
> efficient
> > > way to consume data, for example, if one process consumes from p1, p2,
> p3
> > > and p1,p2 are sitting on one broker b1, p3 is sitting on another one
> b2,
> > > the best thing is create 2 threads each thread use simple consumer API
> > and
> > > only consume from one broker.  I'm thinking how do I use the new API to
> > do
> > > this.
> > >
> > > Thanks,
> > > Siyuan
> > >
> > > On Mon, Nov 16, 2015 at 4:43 PM, Guozhang Wang 
> > wrote:
> > >
> > > > Hi Siyuan,
> > > >
> > > > 1) new consumer is single-threaded, it does not maintain any internal
> > > > threads as the old high-level consumer.
> > > >
> > > > 2) each consumer will only maintain one TCP connection with each
> > broker.
> > > > The only extra socket is the one with its coordinator. That is, if
> > there
> > > is
> > > > three brokers S1, S2, S3, and S1 is the coordinator for this
> consumer,
> > it
> > > > will maintain 4 sockets in total, 2 for S1 (one for fetching, one for
> > > > coordinating) and 1 for S2 and S3 (only for fetching).
> > > >
> > > > 3) Currently the 

Re: [VOTE] 0.9.0.0 Candiate 2

[Jun Rao]

We hit a blocker (KAFKA-2841) in RC2. It's being addressed now. Will roll
out RC3 once it's fixed.

Thanks,

Jun

On Fri, Nov 13, 2015 at 6:28 PM, Jun Rao  wrote:

> This is the second candidate for release of Apache Kafka 0.9.0.0. This a
> major release that includes (1) authentication (through SSL and SASL) and
> authorization, (2) a new java consumer, (3) a Kafka connect framework for
> data ingestion and egression, and (4) quotas. Since this is a major
> release, we will give people a bit more time for trying this out.
>
> Release Notes for the 0.9.0.0 release
>
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, Nov. 19, 7pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate2/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate2/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate2/javadoc/
>
> * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=fad4826f4a58308a29a96ef706bb52b1a9a29e13
>
> * Documentation
> http://kafka.apache.org/090/documentation.html
>
> /***
>
> Thanks,
>
> Jun
>

[VOTE] 0.9.0.0 Candiate 3

[Jun Rao]

This is the third candidate for release of Apache Kafka 0.9.0.0. This a
major release that includes (1) authentication (through SSL and SASL) and
authorization, (2) a new java consumer, (3) a Kafka connect framework for
data ingestion and egression, and (4) quotas. Since this is a major
release, we will give people a bit more time for trying this out.

Release Notes for the 0.9.0.0 release
https://people.apache.org/~junrao/kafka-0.9.0.0-candidate3/RELEASE_NOTES.html

*** Please download, test and vote by Friday, Nov. 20, 10pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS in addition to the md5, sha1
and sha2 (SHA256) checksum.

* Release artifacts to be voted upon (source and binary):
https://people.apache.org/~junrao/kafka-0.9.0.0-candidate3/

* Maven artifacts to be voted upon prior to release:
https://repository.apache.org/content/groups/staging/

* scala-doc
https://people.apache.org/~junrao/kafka-0.9.0.0-candidate3/scaladoc/

* java-doc
https://people.apache.org/~junrao/kafka-0.9.0.0-candidate3/javadoc/

* The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=b8872868168f7b1172c08879f4b699db1ccb5ab7

* Documentation
http://kafka.apache.org/090/documentation.html

/***

Thanks,

Jun

Subscription Request

[saurabh pratap singh]

Hi

I have recently started using kafka and I want to subscribe to mailing list
.


Thanks
Saurabh

Re: Subscription Request

[Prabhjot Bharaj]

Hi Saurabh,

Sure thing. You can follow the steps here:
http://kafka.apache.org/contact.html

Regards,
Prabhjot
On Nov 18, 2015 11:32 AM, "saurabh pratap singh" 
wrote:

> Hi
>
> I have recently started using kafka and I want to subscribe to mailing list
> .
>
>
> Thanks
> Saurabh
>

Re: where do I get the Kafka classes

[Ofir Manor]

BTW - you could download the jars directly from maven central if you want,
for example:
  http://search.maven.org/#search|ga|1|a%3A%22kafka-clients%22

http://search.maven.org/#search|ga|1|a%3A%22spark-streaming-kafka-assembly_2.11%22%20AND%20v%3A%221.5.2%22

Ofir

On Tue, Nov 17, 2015 at 9:36 PM, Adaryl Wakefield <
adaryl.wakefi...@hotmail.com> wrote:

> Learning build tools is going to take a bit. For the time being, I'd
> really just like to get the jars and add them manually. I'm not sure
> anybody has showed me where to get the jars yet.
> B.
>
> > From: ghe...@cloudera.com
> > Date: Tue, 17 Nov 2015 08:25:11 -0600
> > Subject: Re: where do I get the Kafka classes
> > To: users@kafka.apache.org
> >
> > Hi Adaryl,
> >
> > Kafka and its clients doesn't define how you must build your project. You
> > are free to pull together all the jars needed and include them however
> you
> > like. Though that may be difficult and error prone depending on the
> method
> > you choose. Thats where build tools like maven, gradle, etc come in. They
> > make sure your build is "easy", repeatable, and clear.
> >
> > Here are some answers on the web about why using a build tool is a good
> > idea:
> >
> >-
> >
> http://stackoverflow.com/questions/1382432/whats-with-all-the-java-build-tools
> >-
> >
> http://stackoverflow.com/questions/10764576/why-do-we-need-maven-or-ant-if-we-already-have-eclipse
> >
> > Here are some quick-start guides to get you started with maven or gradle:
> >
> >- https://maven.apache.org/guides/getting-started/
> >- https://docs.gradle.org/current/userguide/tutorials.html#N100D0
> >
> > Stackoverflow is often a good place for general programming or java
> > questions (Just search first to be sure it hasn't been asked before). For
> > Kafka questions you can also search the mailing list archives here:
> > http://search-hadoop.com/kafka
> >
> > Hope that helps get you going. If you have any more Kafka related
> > questions, don't hesitate to ask.
> >
> > Thanks,
> > Grant
> >
> > On Tue, Nov 17, 2015 at 12:51 AM, Adaryl "Bob" Wakefield, MBA <
> > adaryl.wakefi...@hotmail.com> wrote:
> >
> > > Is it not possible to just manually include the packages in my Eclipse
> > > project? Do you have to use a build tool?
> > >
> > >
> > >
> > > Adaryl "Bob" Wakefield, MBA
> > > Principal
> > > Mass Street Analytics, LLC
> > > 913.938.6685
> > > www.linkedin.com/in/bobwakefieldmba
> > > Twitter: @BobLovesData
> > > -Original Message- From: Ewen Cheslack-Postava
> > > Sent: Tuesday, November 17, 2015 12:32 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: where do I get the Kafka classes
> > >
> > >
> > > Hi Adaryl,
> > >
> > > First, it looks like you might be trying to use the old producer
> interface.
> > > That interface is going to be deprecated in favor of the new producer
> > > (under org.apache.kafka.clients.producer). I'd highly recommend using
> the
> > > new producer interface instead.
> > >
> > > Second, perhaps this repository of examples will be a helpful starting
> > > point: https://github.com/confluentinc/examples It's just a few basic
> > > examples, but also includes the necessary Maven build scripts. For
> example,
> > > the couple of lines after the highlighted one here:
> > >
> https://github.com/confluentinc/examples/blob/master/producer/pom.xml#L32
> > > will include the necessary jar that includes the new producer.
> > >
> > > -Ewen
> > >
> > > On Mon, Nov 16, 2015 at 10:16 PM, Adaryl "Bob" Wakefield, MBA <
> > > adaryl.wakefi...@hotmail.com> wrote:
> > >
> > > I'm somewhat new to java development and am studying how to write
> > >> producers. The sample code I'm looking at has the following import
> > >> statements:
> > >>
> > >> import kafka.javaapi.producer.Producer;
> > >> import kafka.producer.KeyedMessage;
> > >> import kafka.producer.ProducerConfig;
> > >>
> > >> The thing is, he doesn't use any packages that contain these classes.
> > >> You're supposed to use Gradle to compile the code but I'm not a whiz
> with
> > >> Gradle yet. I'm guessing that Gradle is somehow importing the
> necessary
> > >> classes at compile time. If I didn't want to use Gradle, how would I
> go
> > >> about just getting Kafka packages with the classes I need? I can't
> seem to
> > >> find them by googling.
> > >>
> > >>
> > >> Adaryl "Bob" Wakefield, MBA
> > >> Principal
> > >> Mass Street Analytics, LLC
> > >> 913.938.6685
> > >> www.linkedin.com/in/bobwakefieldmba
> > >> Twitter: @BobLovesData
> > >>
> > >>
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
>