Mit den besten Gruessen/Best regards,
Dr. Thomas Beer
Architect Big Data Solutions
Interior Electronics Solutions
I IES BE
Rechnungsadresse / Invoice address:
Continental Automotive GmbH
Blumenstraße 18
93055 Regensburg, Germany
Mobile: +49 151 20306831
Fax: +49 941 790-8730
E-Mail: thomas.b...@
Eran, thanks a lot. "idleTimeout" solved my problem.
Best,
Thomas
Von:IT CTO
An: user@flume.apache.org,
Datum: 21.09.2015 12:48
Betreff: Re: HDFS sink - rollover
My guess would be that you don't have a steady flow of messages and you
get to the interval
er count varies (it is not always done after 4 events, as
> written before).
>
> Best,
> Thomas
>
>
>
>
> Von:thomas.b...@continental-corporation.com
> An:user@flume.apache.org,
> Datum: 21.09.2015 12:05
> Betreff:HDFS sink - rollover
Addon: The rollover count varies (it is not always done after 4 events, as
written before).
Best,
Thomas
Von:thomas.b...@continental-corporation.com
An: user@flume.apache.org,
Datum: 21.09.2015 12:05
Betreff:HDFS sink - rollover
Hi,
I'm using the Kafka-Flume sourc
Hi,
I'm using the Kafka-Flume source and the Flume-HDFS sink for writing
SequenceFiles. I would like to rollover a SequenceFile after a specific
count of events/messages was written, e.g. after 50 messages (see
rollCount parameter below) a new file should be written.
My configuration seems to b
Yes, you are right. Flume uses uncompressed size to judge the case of
rolling. The appropriate place to calculate size is in-memory. Normally,
compression ratio of snappy might be 5x-10x, more better if there have too
many duplicated data. Thus, it fits your setting, do you agree?
-Regards
Denny Y
On Sun, Aug 26, 2012 at 6:47 AM, Denny Ye wrote:
> hi Mohit,
> Why you confirm it doesn't work at time? I think it reaches to size
> limitation of your setting 'hdfs.rollSize'. Each snappy file almost 5
> hundreds megabytes every 6 or 7 minutes. It fits the compression radio of
> snappy for
hi Mohit,
Why you confirm it doesn't work at time? I think it reaches to size
limitation of your setting 'hdfs.rollSize'. Each snappy file almost 5
hundreds megabytes every 6 or 7 minutes. It fits the compression radio of
snappy format
I rearraged your file order. It's well from my point
I have rollover defined either to roll every 5G or 1+ hr but doesn't seem
to be working. Could you please suggest if I got the conf incorrectly
configured?
foo.sinks.hdfsSink.hdfs.filePrefix = web
foo.sinks.hdfsSink.hdfs.rollInterval = 4000
foo.sinks.hdfsSink.hdfs.rollCount = 0
foo.sinks.hdfsSi
