Re: Memory consumption of Kafka-examples Kafka-streams around 1.5 GB

[Michael Noll]

Phil,

I think what you're seeing w.r.t. memory consumption is that the packaging
step (`mvn package`) for the Kafka Streams examples under
https://github.com/confluentinc/examples also runs the full test suite, and
this test suite launches embedded Kafka brokers, ZooKeeper, and Confluent
schema registry.

As I mentioned to you via our offline email conversation, the easiest
workaround might be to disable the tests when packaging:

$ mvn -DskipTests=true clean package

I also updated the kafka-streams/README.md to include this tip.

Hope this helps,
Michael





On Thu, Jun 16, 2016 at 12:37 AM, Philippe Derome 
wrote:

> Guozhang,
>
> No two in particular, at first it was simply the last two that the target
> would choose: SumLambdaIntegrationTest and WordCountLambdaIntegrationTest .
> I tried another couple to exclude and it was fine as well. There's one
> Scala test that is included in the run and I run it as well as 7/9 of the
> Java ones.
>
> I am not convinced by what you say because after 2-3 tests, I'd think ZK
> and Kafka should be loaded in and I think anecdotally there's about 700MB
> loaded, so there's quite a bit loaded when running the next few tests, at
> least as far as I can see.
>
> The main thing for me is to be reassured that the order of magnitude of
> memory consumption looks right to you. I find it a bit on the high side but
> I won't argue that. So, all in all, I am satisfied with your answer.
>
> On Wed, Jun 15, 2016 at 6:25 PM, Guozhang Wang  wrote:
>
> > Hello Phillippe,
> >
> > I used to run the "SimpleBenchmark" on my laptop with 4GB also, and it
> > usually used close to, but less than 1GB.
> >
> >
> >
> https://www.codatlas.com/github.com/apache/kafka/HEAD/streams/src/test/java/org/apache/kafka/streams/perf/SimpleBenchmark.java
> >
> > Note that I need to bootstrap a real ZK instance and a Kafka instance in
> > order to run that benchmark, and I think those two instances are actually
> > taking the major memory usage than Kafka Streams instance itself.
> >
> > There may be some extra memory overhead from maven framework but I would
> be
> > surprised if that is taking large amount.
> >
> > Which two test cases specifically are causing OOMs on your laptop?
> >
> > Guozhang
> >
> >
> > On Tue, Jun 14, 2016 at 4:50 PM, Philippe Derome 
> > wrote:
> >
> > > I am running "mvn test" as per tip from
> > > https://github.com/confluentinc/examples/tree/master/kafka-streams
> > > README.MD.
> > > This uses embedded Kafka components from test alone (no ZK, Kafka,
> > > schema-registry running).
> > >
> > > I monitor on OSX El Capitan (10.11.5) memory usage and it grows on Java
> > > processes from nothing to about 1.3GB when it fails to execute last 2
> > tests
> > > with Java out of memory exceptions. Selecting 2 tests to avoid makes
> the
> > > test pass but with my 4GB system, I cannot pass them all.
> > >
> > > Is that relatively large memory consumption to be expected on these
> test
> > > cases?
> > >
> > > I'd like to run stand-alone from jar and I'll be able to do so by
> > excluding
> > > test cases.
> > >
> > > Fyi, on mailing list I see only 1 issue related to Streams and memory
> if
> > > that's any relevant (KAFKA-3738).
> > >
> > > Phil
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
Best regards,
Michael Noll



*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
*

Re: Memory consumption of Kafka-examples Kafka-streams around 1.5 GB

[Philippe Derome]

Guozhang,

No two in particular, at first it was simply the last two that the target
would choose: SumLambdaIntegrationTest and WordCountLambdaIntegrationTest .
I tried another couple to exclude and it was fine as well. There's one
Scala test that is included in the run and I run it as well as 7/9 of the
Java ones.

I am not convinced by what you say because after 2-3 tests, I'd think ZK
and Kafka should be loaded in and I think anecdotally there's about 700MB
loaded, so there's quite a bit loaded when running the next few tests, at
least as far as I can see.

The main thing for me is to be reassured that the order of magnitude of
memory consumption looks right to you. I find it a bit on the high side but
I won't argue that. So, all in all, I am satisfied with your answer.

On Wed, Jun 15, 2016 at 6:25 PM, Guozhang Wang  wrote:

> Hello Phillippe,
>
> I used to run the "SimpleBenchmark" on my laptop with 4GB also, and it
> usually used close to, but less than 1GB.
>
>
> https://www.codatlas.com/github.com/apache/kafka/HEAD/streams/src/test/java/org/apache/kafka/streams/perf/SimpleBenchmark.java
>
> Note that I need to bootstrap a real ZK instance and a Kafka instance in
> order to run that benchmark, and I think those two instances are actually
> taking the major memory usage than Kafka Streams instance itself.
>
> There may be some extra memory overhead from maven framework but I would be
> surprised if that is taking large amount.
>
> Which two test cases specifically are causing OOMs on your laptop?
>
> Guozhang
>
>
> On Tue, Jun 14, 2016 at 4:50 PM, Philippe Derome 
> wrote:
>
> > I am running "mvn test" as per tip from
> > https://github.com/confluentinc/examples/tree/master/kafka-streams
> > README.MD.
> > This uses embedded Kafka components from test alone (no ZK, Kafka,
> > schema-registry running).
> >
> > I monitor on OSX El Capitan (10.11.5) memory usage and it grows on Java
> > processes from nothing to about 1.3GB when it fails to execute last 2
> tests
> > with Java out of memory exceptions. Selecting 2 tests to avoid makes the
> > test pass but with my 4GB system, I cannot pass them all.
> >
> > Is that relatively large memory consumption to be expected on these test
> > cases?
> >
> > I'd like to run stand-alone from jar and I'll be able to do so by
> excluding
> > test cases.
> >
> > Fyi, on mailing list I see only 1 issue related to Streams and memory if
> > that's any relevant (KAFKA-3738).
> >
> > Phil
> >
>
>
>
> --
> -- Guozhang
>

Re: Memory consumption of Kafka-examples Kafka-streams around 1.5 GB

[Guozhang Wang]

Hello Phillippe,

I used to run the "SimpleBenchmark" on my laptop with 4GB also, and it
usually used close to, but less than 1GB.

https://www.codatlas.com/github.com/apache/kafka/HEAD/streams/src/test/java/org/apache/kafka/streams/perf/SimpleBenchmark.java

Note that I need to bootstrap a real ZK instance and a Kafka instance in
order to run that benchmark, and I think those two instances are actually
taking the major memory usage than Kafka Streams instance itself.

There may be some extra memory overhead from maven framework but I would be
surprised if that is taking large amount.

Which two test cases specifically are causing OOMs on your laptop?

Guozhang


On Tue, Jun 14, 2016 at 4:50 PM, Philippe Derome  wrote:

> I am running "mvn test" as per tip from
> https://github.com/confluentinc/examples/tree/master/kafka-streams
> README.MD.
> This uses embedded Kafka components from test alone (no ZK, Kafka,
> schema-registry running).
>
> I monitor on OSX El Capitan (10.11.5) memory usage and it grows on Java
> processes from nothing to about 1.3GB when it fails to execute last 2 tests
> with Java out of memory exceptions. Selecting 2 tests to avoid makes the
> test pass but with my 4GB system, I cannot pass them all.
>
> Is that relatively large memory consumption to be expected on these test
> cases?
>
> I'd like to run stand-alone from jar and I'll be able to do so by excluding
> test cases.
>
> Fyi, on mailing list I see only 1 issue related to Streams and memory if
> that's any relevant (KAFKA-3738).
>
> Phil
>



-- 
-- Guozhang

Memory consumption of Kafka-examples Kafka-streams around 1.5 GB

[Philippe Derome]

I am running "mvn test" as per tip from
https://github.com/confluentinc/examples/tree/master/kafka-streams README.MD.
This uses embedded Kafka components from test alone (no ZK, Kafka,
schema-registry running).

I monitor on OSX El Capitan (10.11.5) memory usage and it grows on Java
processes from nothing to about 1.3GB when it fails to execute last 2 tests
with Java out of memory exceptions. Selecting 2 tests to avoid makes the
test pass but with my 4GB system, I cannot pass them all.

Is that relatively large memory consumption to be expected on these test
cases?

I'd like to run stand-alone from jar and I'll be able to do so by excluding
test cases.

Fyi, on mailing list I see only 1 issue related to Streams and memory if
that's any relevant (KAFKA-3738).

Phil