Thanks for the reply. I also came to that conclusion eventually. However,
could you explain to me more how I'd store each batch in a file?
It is my understanding so far that the Aggregator is called whenever the
input emits a new exchange. If I were to store the messages in a file,
how'd I know inside the aggregator that the current batch is complete?
After all, I can hardly continue writing messages to the file while aws-s3
is pushing the same to s3
On 14 Oct 2014 09:08, "Willem Jiang" <willem.ji...@gmail.com> wrote:
> I just found the code.
>
> class JsonBodyAppender {
> def append(existing: String, next: String) = existing + "\n" + next
> }
>
> There are too many String there, maybe you can just append the message
> into a file.
>
> --
> Willem Jiang
>
> Red Hat, Inc.
> Web: http://www.redhat.com
> Blog: http://willemjiang.blogspot.com (English)
> http://jnn.iteye.com (Chinese)
> Twitter: willemjiang
> Weibo: 姜宁willem
>
>
>
> On October 14, 2014 at 2:32:00 PM, Willem Jiang (willem.ji...@gmail.com)
> wrote:
> > Can you show us how did you do the aggregation? Just some code snippet
> is OK.
> >
> > BTW, you can always using write the stream into to file to avoid load
> the whole message
> > into memory.
> >
> > --
> > Willem Jiang
> >
> > Red Hat, Inc.
> > Web: http://www.redhat.com
> > Blog: http://willemjiang.blogspot.com (English)
> > http://jnn.iteye.com (Chinese)
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> >
> >
> > On October 14, 2014 at 5:23:21 AM, Andreas C. Osowski (
> andreas.chr.osow...@gmail.com)
> > wrote:
> > > Hey there.
> > > I'm running into "OOM: Java heap space" exceptions trying to aggregate
> > > twitter messages before pushing them to s3. (Xmx/Xms = 1.5g; there's
> > > definitely sufficient free memory available when the exceptions happen)
> > > I've also tried to use camel-leveldb as the AggregationRepository but
> > > without luck. Smaller batch sizes (i.e. 100) work just fine... 5000
> > > aggregated messages should also take up just 60mb-ish (assuming an
> average
> > > of 3000chars per message).
> > >
> > > The relevant code can be found here:
> > > https://gist.github.com/th0br0/a1484ea1ad18b8b20c25
> > >
> > > Could somebody point me to what I'm doing wrong?
> > >
> > > Thanks a lot!
> > >
> >
> >
>
>
