Phil,
I think what you're seeing w.r.t. memory consumption is that the packaging
step (`mvn package`) for the Kafka Streams examples under
https://github.com/confluentinc/examples also runs the full test suite, and
this test suite launches embedded Kafka brokers, ZooKeeper, and Confluent
schema registry.
As I mentioned to you via our offline email conversation, the easiest
workaround might be to disable the tests when packaging:
$ mvn -DskipTests=true clean package
I also updated the kafka-streams/README.md to include this tip.
Hope this helps,
Michael
On Thu, Jun 16, 2016 at 12:37 AM, Philippe Derome <phder...@gmail.com>
wrote:
> Guozhang,
>
> No two in particular, at first it was simply the last two that the target
> would choose: SumLambdaIntegrationTest and WordCountLambdaIntegrationTest .
> I tried another couple to exclude and it was fine as well. There's one
> Scala test that is included in the run and I run it as well as 7/9 of the
> Java ones.
>
> I am not convinced by what you say because after 2-3 tests, I'd think ZK
> and Kafka should be loaded in and I think anecdotally there's about 700MB
> loaded, so there's quite a bit loaded when running the next few tests, at
> least as far as I can see.
>
> The main thing for me is to be reassured that the order of magnitude of
> memory consumption looks right to you. I find it a bit on the high side but
> I won't argue that. So, all in all, I am satisfied with your answer.
>
> On Wed, Jun 15, 2016 at 6:25 PM, Guozhang Wang <wangg...@gmail.com> wrote:
>
> > Hello Phillippe,
> >
> > I used to run the "SimpleBenchmark" on my laptop with 4GB also, and it
> > usually used close to, but less than 1GB.
> >
> >
> >
> https://www.codatlas.com/github.com/apache/kafka/HEAD/streams/src/test/java/org/apache/kafka/streams/perf/SimpleBenchmark.java
> >
> > Note that I need to bootstrap a real ZK instance and a Kafka instance in
> > order to run that benchmark, and I think those two instances are actually
> > taking the major memory usage than Kafka Streams instance itself.
> >
> > There may be some extra memory overhead from maven framework but I would
> be
> > surprised if that is taking large amount.
> >
> > Which two test cases specifically are causing OOMs on your laptop?
> >
> > Guozhang
> >
> >
> > On Tue, Jun 14, 2016 at 4:50 PM, Philippe Derome <phder...@gmail.com>
> > wrote:
> >
> > > I am running "mvn test" as per tip from
> > > https://github.com/confluentinc/examples/tree/master/kafka-streams
> > > README.MD.
> > > This uses embedded Kafka components from test alone (no ZK, Kafka,
> > > schema-registry running).
> > >
> > > I monitor on OSX El Capitan (10.11.5) memory usage and it grows on Java
> > > processes from nothing to about 1.3GB when it fails to execute last 2
> > tests
> > > with Java out of memory exceptions. Selecting 2 tests to avoid makes
> the
> > > test pass but with my 4GB system, I cannot pass them all.
> > >
> > > Is that relatively large memory consumption to be expected on these
> test
> > > cases?
> > >
> > > I'd like to run stand-alone from jar and I'll be able to do so by
> > excluding
> > > test cases.
> > >
> > > Fyi, on mailing list I see only 1 issue related to Streams and memory
> if
> > > that's any relevant (KAFKA-3738).
> > >
> > > Phil
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
Best regards,
Michael Noll
*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*
