Hello AnandKumar,
The issue is again repeating. Yesterday the files were deleting from spool
but the data was not available in the destination Hbase tables, but now
this morning what I have noticed is the log is same.
Earlier, as you have suggested, "verify the firewall settings. It blocks
the connection it seems."
Can you be more specific, I mean how can verify the firewall settings. Any
suggestion?
Thanks in advance.
8:45:01.040 AMERRORorg.apa
che.flume.SinkRunner
Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host:
sdldalplhdw02.suddenlink.cequel3.com, port: 40002 }: RPC connection
error
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:182)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:121)
at
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:638)
at
org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:89)
at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
at
org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:211)
at
org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:272)
at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:349)
... 3 more
Caused by: java.io.IOException: Error connecting to
sdldalplhdw02.suddenlink.cequel3.com/10.48.210.244:40002
at
org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:292)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:155)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:168)
... 10 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:148)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:104)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Regards,
Nik.
On Fri, Aug 7, 2015 at 8:28 AM, Nikhil Gs <gsnikhil1432...@gmail.com> wrote:
> Hello Anandkumar,
>
> Thank you for your time and also for the reply.
>
> Now, flume acts differently. I don't see any log errors, in fact my log
> now looks as below;
>
> 2015-08-06 22:51:14,426 INFO
> org.apache.flume.client.avro.ReliableSpoolingFileEv
> entReader:
> Preparing to delete file /home/a_nikhil.gopishetti/pnm/GVLLCMTK03-675
>
> 79944.pf
> 2015-08-06 23:12:15,026 INFO
> org.apache.flume.client.avro.ReliableSpoolingFileEv
> entReader:
> Preparing to delete file /home/a_nikhil.gopishetti/pnm/KNTNCMTK01.SUD
>
> DENLINK.NET-7682.pf
> 2015-08-06 23:12:22,030 INFO
> org.apache.flume.client.avro.ReliableSpoolingFileEv
> entReader:
> Preparing to delete file /home/a_nikhil.gopishetti/pnm/KNTNCMTK01.SUD
>
> DENLINK.NET-7682.pf
> 2015-08-06 23:13:47,570 INFO
> org.apache.flume.client.avro.ReliableSpoolingFileEv
> entReader:
> Preparing to delete file /home/a_nikhil.gopishetti/pnm/NBRNCMTK01-340
>
> 17296.pf.filepart
> 2015-08-06 23:13:57,076 INFO
> org.apache.flume.client.avro.ReliableSpoolingFileEv
> entReader:
> Preparing to delete file /home/a_nikhil.gopishetti/pnm/NBRNCMTK01-675
>
> 84072.pf
> 2015-08-06 23:14:03,581 INFO
> org.apache.flume.client.avro.ReliableSpoolingFileEv
> entReader:
> Preparing to delete file /home/a_nikhil.gopishetti/pnm/NBRNCMTK01-675
>
> 79944.pf
> 2015-08-06 23:23:23,348 INFO
> org.apache.flume.client.avro.ReliableSpoolingFileEv
> entReader:
> Preparing to delete file /home/a_nikhil.gopishetti/pnm/PMLECMTK01.SUD
>
> DENLINK.NET-8426.pf.filepart
>
>
> So the files are deleting. So when I go back to my destination i.e. Hbase
> tables, I don't see any data. Earlier, a couple of days back, I could
> clearly see that my data was deleting from the spool directory and it was
> getting into my Hbase table. But now I don't see any data in my destination
> tables. I don't know why. Any suggestions.
>
> Thanks in advance for your time and reply.
>
> Regards,
> Nik.
>
> On Thu, Aug 6, 2015 at 11:43 PM, Anandkumar Lakshmanan <an...@orzota.com>
> wrote:
>
>> Hi Nik,
>>
>> Please verify the firewall settings. It blocks the connection it seems.
>>
>> Thanks
>> Anand.
>>
>>
>>
>> On 08/07/2015 02:07 AM, Nikhil Gs wrote:
>>
>> Hello Team,
>>
>> Facing the below error very alternatively even though worked with
>> different port numbers. I have pasted my flume config file along with the
>> error.
>>
>> Thanks in advance.
>>
>>
>> Below is my flume configuration
>>
>> #################################################
>>
>> # Please paste flume.conf here. Example:
>>
>> # Sources, channels, and sinks are defined per
>> # agent name, in this case 'pnmtest2'.
>> pnmtest2.sources = SPOOL
>> pnmtest2.channels = MemChanneltest2
>> pnmtest2.sinks = AVRO
>>
>> # For each source, channel, and sink, set
>> # standard properties.
>> pnmtest2.sources.SPOOL.type = spooldir
>> pnmtest2.sources.SPOOL.spoolDir = /home/a.nikhill/pnm
>> pnmtest2.sources.SPOOL.ignorePattern = \.*tmp$
>> pnmtest2.sources.SPOOL.channels = MemChanneltest2
>> pnmtest2.sources.SPOOL.fileHeader = true
>> pnmtest2.sources.SPOOL.deletePolicy = immediate
>> pnmtest2.sources.SPOOL.consumeOrder = oldest
>> pnmtest2.sources.SPOOL.batchSize = 100
>>
>> pnmtest2.sources.SPOOL.interceptors = time
>> pnmtest2.sources.SPOOL.interceptors.time.type =
>> org.apache.flume.interceptor.TimestampInterceptor$Builder
>> pnmtest2.sources.SPOOL.deserializer =
>> com.sudnline.flume.WholeFileDeserializer$Builder
>>
>> pnmtest2.sinks.AVRO.type = avro
>> pnmtest2.sinks.AVRO.channel = MemChanneltest2
>> pnmtest2.sinks.AVRO.hostname = sdldalplhdw02.sudnline.cequel3.com
>> <http://sdldalplhdw02.suddenlink.cequel3.com/>
>> pnmtest2.sinks.AVRO.port = 40002
>> pnmtest2.sinks.AVRO.batch-size = 100
>> pnmtest2.sinks.AVRO.connect-timeout = 40000
>>
>>
>> # pnmtest2.sinks.HDFS.type = hdfs
>> # pnmtest2.sinks.HDFS.channel = MemChannel2
>> # pnmtest2.sinks.HDFS.hdfs.path = /user/flume/poll/%Y/%m/%d/%H/
>> # pnmtest2.sinks.HDFS.hdfs.fileType = DataStream
>> # pnmtest2.sinks.HDFS.hdfs.writeFormat = Text
>> # pnmtest2.sinks.HDFS.hdfs.batchSize = 100
>> # pnmtest2.sinks.HDFS.hdfs.rollSize = 0
>> # pnmtest2.sinks.HDFS.hdfs.rollCount = 1000
>> # pnmtest2.sinks.HDFS.hdfs.rollInterval = 600
>>
>> # Other properties are specific to each type of
>> # source, channel, or sink. In this case, we
>> # specify the capacity of the memory channel.
>>
>> #pnmtest2.channels.MemChanneltest1.capacity = 10000
>> #pnmtest2.channels.MemChanneltest1.type = memory
>>
>> pnmtest2.channels.MemChanneltest2.capacity = 1000000
>> pnmtest2.channels.MemChanneltest2.type = memory
>>
>>
>> Below is my error.
>>
>>> ERROR org.apache.flume.SinkRunner
>>>>
>>>> Unable to deliver event. Exception follows.
>>>> org.apache.flume.EventDeliveryException: Failed to send events
>>>> at
>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
>>>> at
>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host:
>>>> sdldalplhdw02.suddenlink.cequel3.com, port: 40002 }: RPC connection error
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:182)
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:121)
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:638)
>>>> at
>>>> org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:89)
>>>> at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
>>>> at
>>>> org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:211)
>>>> at
>>>> org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:272)
>>>> at
>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:349)
>>>> ... 3 more
>>>> Caused by: java.io.IOException: Error connecting to
>>>> sdldalplhdw02.suddenlink.cequel3.com/10.48.210.244:40002
>>>> at
>>>> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:292)
>>>> at
>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
>>>> at
>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:155)
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:168)
>>>> ... 10 more
>>>> Caused by: java.net.ConnectException: Connection refused
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:148)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:104)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>
>>>>
>>>>
>>>>
>> Regards,
>> Nik.
>>
>>
>>
>
