The default memory settings for flume are extremely low. Try giving it more
Java memory.
On Jun 18, 2014, at 12:57 PM, David Gates <david.ga...@gmail.com> wrote:
> When running a test to see if I can get Morphline Interceptor working to
> convert some timestamps in logfiles, I am getting a java GC overhead limit
> exceeded error.
>
> Command:
>
> flume-ng agent --conf ./conf -f testflume.conf
> -Dflume.root.logger=DBUG,console -n agent
>
> testflume.conf:
>
> agent.channels.memory-channel.type = memory
> agent.sources.spool-source.type = spooldir
> agent.sources.spool-source.spoolDir = /home/impala/spool/
> agent.sources.spool-source.channels = memory-channel
> agent.sources.spool-source.interceptors = morphlineinterceptor
> agent.sources.spool-source.interceptors.morphlineinterceptor.type =
> org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
> agent.sources.spool-source.interceptors.morphlineinterceptor.morphlineFile
> = /root/morphline.conf
> agent.sources.spool-source.interceptors.morphlineinterceptor.morphlineId =
> morphline1
> agent.sinks.hdfs-sink.channel = memory-channel
> agent.sinks.hdfs-sink.type = hdfs
> agent.sinks.hdfs-sink.hdfs.path = /user/impala/
> agent.sinks.hdfs-sink.hdfs.fileType = DataStream
> agent.channels = memory-channel
> agent.sources = spool-source
> agent.sinks = hdfs-sink
>
> morphline.conf:
>
> morphlines : [
> {
> id : morphline1
> importCommands : ["com.cloudera.**"]
>
> commands : [
> {
> readCSV {
> separator: ";"
> trim: true
> columns:
> [Header1,Header2,Header3,ConnectionType,SessionID,ReleaseCause,StartTime,AnswerTime,ReleaseTime,MinutesWest,ReleaseCauseProto,ReleaseCauseNum,FirstReleaseDialogue,TrunkIDOrig,VOIPProtoOrig,SourceNumOrig,SourceHostOrig,DestNumOrig,DestHostOrig,OrigCallID,OrigRemotePayloadIP,OrigRemotePayloadUDP,OrigLocalPayloadIP,OrigLocalPayloadUDP,OrigCodecList,OrigIngressPackets,OrigEgressPackets,OrigIngressOctets,OrigEgressOctets,OrigIngressPacketLoss,OrigIngressDelay,OrigIngressJitter,TrunkIDTerm,VOIPProtoTerm,SourceNumTerm,SourceHostTerm,DestNumTerm,DestHostTerm,TermCallID,TermRemotePayloadIP,TermRemotePayloadUDP,TermLocalPayloadIP,TermLocalPayloadUDP,TermCodecList,TermIngressPackets,TermEgressPackets,TermIngressOctets,TermEgressOctets,TermIngpressPacketLoss,TermIngressDelay,TermIngressJitter,FinalRouteIndication,RoutingDigits,CallDurSec,PostDialDelaySec,RingTimeSec,DurMiniSec,ConfID,RPIDANI,RouteEntryIndex,RouteTable,LNPDip,IngressLRN,EgressLRN,CNAMDip,DNCDip,OrigTrunkAlias,TermTrunkAlias,ERSDip,OLIDigits]
> }
> }
> ]
> }
> ]
>
>
> I had originally also had a convertTimeStamp command in there but removed
> it to troubleshoot.
>
> The error I get when running is as follows:
>
> 14/05/21 08:37:23 INFO api.MorphlineContext: Importing commands
> 14/05/21 08:37:31 ERROR node.PollingPropertiesFileConfigurationProvider:
> Unhandled error
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
>
> Ive tried googling but I can't find anything specific to flume/morphline
> with GC limit exceeded, any help/ideas would be appreciated.
