[
https://issues.apache.org/jira/browse/FLUME-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Snehal Waghmare updated FLUME-3137:
-----------------------------------
Comment: was deleted
(was: Temporarily we are moving files continuously from the processed folder to
as to prevent the bulk pushing of the data.
Also, updated the JAVA_OPTS configuration with following configuration:
"-Xms2000m -Xmx16000m -Xss128k -XX:MaxDirectMemorySize=256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"
Kindly help me solve the above issue.)
> GC overhead limit exceeded in KafkaSink
> ---------------------------------------
>
> Key: FLUME-3137
> URL: https://issues.apache.org/jira/browse/FLUME-3137
> Project: Flume
> Issue Type: Bug
> Reporter: Snehal Waghmare
>
> We are trying to push number of log files are located at folder
> "/path-to-logs/processed_logs/" through flume to kafka.
> Now when flume agent is started it tries to send a few number of files but
> after that gives the below error:
> And finally, it is unable to tail the remaining files. This inhibits the
> further flow of data.
> [ERROR -
> org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:334)]
> Failed writing positionFile
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:95)
> at
> org.apache.flume.source.taildir.TaildirSource.toPosInfoJson(TaildirSource.java:349)
> at
> org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:330)
> at
> org.apache.flume.source.taildir.TaildirSource.access$600(TaildirSource.java:59)
> at
> org.apache.flume.source.taildir.TaildirSource$PositionWriterRunnable.run(TaildirSource.java:320)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> [ERROR -
> org.apache.flume.source.taildir.TaildirSource.process(TaildirSource.java:236)]
> Unable to tail files
> java.lang.OutOfMemoryError: GC overhead limit exceeded
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
