[
https://issues.apache.org/jira/browse/FLUME-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613310#comment-15613310
]
Jeff Field commented on FLUME-2812:
-----------------------------------
We're encountering this while a source (Kafka) is trying to deliver data to the
channel (Memory) much much faster than the sink (HDFS) can write it out. The
(repeatable) chain of events is: channel fills up, HDFS keeps writing, channel
keeps throwing errors because it is full, eventually HDFS sinks all die with
this error. We don't encounter it on 1.6 because in 1.6 we're using Kafka
Channel -> Sink directly. We were trying to do a similar config for 1.7 but a
separate issue is preventing that (we're getting Kafka consumer
timeout/rebalance problems with the source and channel in 1.7 that we didn't
get in 1.6).
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.Error: Maximum permit count exceeded
> ------------------------------------------------------------------------------------------------------------------
>
> Key: FLUME-2812
> URL: https://issues.apache.org/jira/browse/FLUME-2812
> Project: Flume
> Issue Type: Bug
> Components: Channel, Sinks+Sources
> Affects Versions: v1.6.0
> Environment: **OS INFO**
> CentOS release 6.6 (Final)
> Kernel \r on an \m
> **JAVA INFO**
> java version "1.8.0_40"
> Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
> Reporter: Rollin Crittendon
> Priority: Critical
>
> We are finding that around after an hour or so of heavy processing of Flume
> data in an agent we are getting the following exception. This is after
> processing about 5-7 k lines/second during that time.
> The configuration of this agent is using a Kafka source, the one that comes
> with 1.6.0.
> It is also using a Memory channel, and a Thrift sink.
> =======
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.Error: Maximum permit count exceeded
> at
> java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at java.util.concurrent.Semaphore.release(Semaphore.java:609)
> at
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> =======
> The above error is from standard error when running the Flume agent. The
> effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread
> disappears from the agent, this can be seen on a JMX console.
> For us, this means that the Flume agent needs to get restarted. It is an
> error that is terminal in that instance of the Java process due to the thread
> disappearing as a result.
> It sounds like something in JDK 7+ got stricter?!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
