[ 
https://issues.apache.org/jira/browse/FLUME-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957868#comment-14957868
 ] 

Rollin Crittendon commented on FLUME-2812:
------------------------------------------

This is what we have, with machine names, ports, some things like that 
redacted, for example, <ZOOKEEPER-SETTINGS>.

=============================
bridge.channels = bridge-channel
bridge.sources = spool-source kafka-source
bridge.sinks = to-conduit-sink

bridge.channels.bridge-channel.type = memory
bridge.channels.bridge-channel.capacity = 100000
bridge.channels.bridge-channel.transactionCapacity = 40000
bridge.channels.bridge-channel.byteCapacityBufferPercentage = 20
bridge.channels.bridge-channel.byteCapacity = 0

bridge.sources.spool-source.channels = bridge-channel
bridge.sources.spool-source.type = spooldir
bridge.sources.spool-source.spoolDir = <SPOOL-DIRECTORY-LOCATION> 
bridge.sources.spool-source.fileHeader = false
bridge.sources.spool-source.deletePolicy = immediate
bridge.sources.spool-source.batchSize = 100
bridge.sources.spool-source.deserializer.maxLineLength = 100000

bridge.sources.kafka-source.channels = bridge-channel
bridge.sources.kafka-source.type = org.apache.flume.source.kafka.KafkaSource
bridge.sources.kafka-source.zookeeperConnect = <ZOOKEEPER-SETTINGS>
bridge.sources.kafka-source.groupId = <GROUP-ID>
bridge.sources.kafka-source.topic = <TOPIC>
bridge.sources.kafka-source.batchSize = 100

bridge.sinks.to-conduit-sink.channel = bridge-channel
bridge.sinks.to-conduit-sink.type = thrift
bridge.sinks.to-conduit-sink.hostname = <OTHER-AGENT-MACHINE-NAME> 
bridge.sinks.to-conduit-sink.port = <OTHER-AGENT-MACHINE-PORT>
bridge.sinks.to-conduit-sink.batch-size = 1000

> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
> java.lang.Error: Maximum permit count exceeded
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLUME-2812
>                 URL: https://issues.apache.org/jira/browse/FLUME-2812
>             Project: Flume
>          Issue Type: Bug
>          Components: Channel, Sinks+Sources
>    Affects Versions: v1.6.0
>         Environment: **OS INFO**
> CentOS release 6.6 (Final)
> Kernel \r on an \m
> **JAVA INFO**
> java version "1.8.0_40"
> Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
>            Reporter: Rollin Crittendon
>            Priority: Critical
>
> We are finding that around after an hour or so of heavy processing of Flume 
> data in an agent we are getting the following exception.  This is after 
> processing about 5-7 k lines/second during that time.
> The configuration of this agent is using a Kafka source, the one that comes 
> with 1.6.0. 
> It is also using a Memory channel, and a Thrift sink.
> =======
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
> java.lang.Error: Maximum permit count exceeded
>       at 
> java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
>       at java.util.concurrent.Semaphore.release(Semaphore.java:609)
>       at 
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
>       at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>       at 
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
>       at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>       at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>       at java.lang.Thread.run(Thread.java:745)
> =======
> The above error is from standard error when running the Flume agent.  The 
> effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread 
> disappears from the agent, this can be seen on a JMX console.
> For us, this means that the Flume agent needs to get restarted.  It is an 
> error that is terminal in that instance of the Java process due to the thread 
> disappearing as a result.
> It sounds like something in JDK 7+ got stricter?!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to